We recently upgraded from ESX 3.5 to vSphere at work, and man – is it awesome. The infrastructure manager (vSphere Client) supports a whole boatload of new options for better monitoring and managing your VMs and clusters – including a patch management system for keeping your VMs and hosts up to date. If you’d like the full details of all the new features, check out the VMware page here.

The Console OS

One new aspect of vSphere is it separates out the hypervisor from the service console to a greater degree, actually creating a VM for the console. Great from an architectural point of view – however, one gotcha is where it actually STORES this VM. During the upgrade (and I’m assuming new install process), it asks where you’d like to store this Console OS VM. It suggests you store it on local storage, but you can just as easily store it in a Datastore on your SAN. Which is what I (and others judging from the web) chose as an option.

An issue arises in that there is no easy way (that I know of, after scouring the web for ages) to move the console VM in the future. We recently needed to redo all the LUNs on our SANs, which meant we needed them empty – which is when we ran into this issue. The service console was on the SAN with no good way to move it.

The Solution

First off – proceed at your own risk. [UPDATE] When asking VMware support about this solution, they said it works, but is not officially supported. [END UPDATE] So far we’ve had no issues, but that’s not to say in 2 months our whole ESX cluster won’t detonate, showering death and destruction down on us. That being said, I think we’re pretty safe.

This question has been asked before, and everyone on the VMware forums said you needed to reinstall your ESX server to move the console VM – this is the official stance by VMware as well. I originally decided to do a little digging though, as it just seemed like there must be some way of doing it. It was just a file that needed to be moved, and the underlying OS is Linux, so I guessed it was all being done through scripts. I was right.

If you take a look at /etc/vmware/esx.conf on your ESX host in question, you’ll see all your configuration options. One of which is

/boot/cosvmdk

This points to the path and filename of the service console. This value is later used by the initialization script “/etc/vmware/init/init.d/66.vsd-mount” to mount the service console. We can change this value to anything we want, inclung a new location.

1. Identify the correct service console VM for the ESX server in question. This isn’t an issue if you’ve got 1 ESX server, but if you’ve got a cluster, it’s not always clear the one in question. The console VM is stored with the name “esxconsole-<uuid>”. You can find the unique identifier/cos vmdk filename for your server within the /etc/vmware/esx.conf file.
2. Identify the Datastore where you want to keep the service console. Take VMware’s advice and keep it on local storage – that way, if your SAN dies or you need to do maintenance, you aren’t in a pickle. Look in /vmfs/volumes and write down the ID of the storage you want to use.
3. Put the ESX host in question into service mode – you’ll need to reboot it to perform the move, and you’ll need local access as it won’t reboot to a point where you have ssh.
4. Make a backup copy of /etc/vmware/esx.conf in case you make a boo-boo.
5. Edit /etc/vmware/esx.conf and change the path for the “/boot/cosvmdk” option to point to the new Datastore you recorded in #2. Save the conf file.
6. Reboot the server. It will go smoothly until it hits the point where it attempts to mount the COS – at this point, it will choke as it can’t find esxconsole in the new place you told it to look. At this point, you’ll get a shell prompt.
7. Do a recursive copy of the esxconsole-<uuid> directory from its old location to the new location. You should have all your /vmfs/volumes mounted since this takes place in the initialization sequence before the COS is mounted. [UPDATE] Jim points out that while local and FC storage will be available, it looks like iSCSI mounts take place later in the boot process, so they will not. Just a heads up if you’re planning on moving your COS to/from iSCSI [END UPDATE]
8. Reboot. It should boot back up completely at this point. Ensure life is good, and then delete the old copy of the esxconsole-<uuid> directory.
9. Ensure ESX automatically updated any other values to the new path (like /adv/Misc/CosCorefile)

Please don’t hesitate to comment with your experiences or if you know a better way to handle this (or if this solution could cause issues). Good luck!