Virtual Centre

Yes, I know, it should be Virtual Center (or as its now known vCenter) not Centre, but in English centre is spelt that way and that's the language we're using here, not a localised dialect. Any complaints, speak to the Queen, its her language.

Install

For a production installation you should have a ready to use SQL database server (MS or Oracle), for small installations you can use MS SQL Express (which is installed as part of the vCentre install when no external database is specified).

This is a basic install that will get you going, see Installing ESX 4.1 and vCenter Server 4.1 best practices for more detailed info, or the Architecture Suggestions section below for some hints.

SQL DSN requirements

• VC < v4.1 - Requires a 32-bit DSN even if vCentre is being run on a 64-bit machine
• VC v4.1 - Requires a 64-bit DSN
• vUM - * Requires a 32-bit DSN

To install vCentre...

1. Start the installer
2. Accept EULA, and then click Next
3. Enter vCentre license key (if you have one), and then click Next
4. Setup a database to use, and then click Next
   • Use MS SQL 2005 Express option for small scale or non-production deployments only
   • Select an existing DSN (to set-up a new DSN, see #Create DSN)
5. Enter the appropriate authentication information, and then click Next
6. Select the appropriate Linked Mode (this proc assumes Standalone), and then click Next
7. Accept the default TCP ports (its recommended that you don't share the server with other services which, for example) might necessitate changing the vCentre ports), and then click Next
8. Click Install

Normally you'd now install Update Manager

Create DSN

Use this procedure to create a new SQL DSN on the server on which you are installing vCentre, if your database is Oracle rather than Microsoft then you'll need to install an Oracle db driver 1st and select that rather than the SQL Native Client in this procedure. A blank/empty database should be create on the database server prior to carrying out this procedure (see Create Database).

If you need to specifically create a 32-bit DSN. Start the ODBC admin tool from %systemdrive%\Windows\SysWoW64\Odbcad32.exe instead of the Administrative Tools menu.

1. Go to Data Sources (ODBC) in the Administrative Tools menu in order to start the ODBC admin tool
2. Go to the System DSN tab and click on Add...
3. Select the SQL Native Client driver and click Finish
4. Enter the required options described below, and then select Next...
   • Name: This is the name that you'll know the DSN by, and use in the vCentre setup, eg VC
   • Description: Optional description for DSN, eg vCentre database
   • Server: Database server hostname, eg localhost or db1.company.com
5. Enter the appropriate authentication options for your database (ask your DBA if in doubt!), and then click Next
   • Use With SQL Server authentication... and enter the vpxuser credentials if you pre-created them when creating the database
6. Check the Change the default database to option and select the appropriate database (that's been created on the database server), if required, then click Next
7. Leave default options as is on final screen, and click Finish
8. Click the Test Data Source... button to test connectivity to the database, then OK to finish
   • If the test fails, recheck your config, and also test connectivity to the database server (ping, telnet porttest etc)

Upgrade

Upgrades are performed using the same package as used for a fresh installation. Obviously VI management will be unavailable during the upgrade and the database should be backed up prior to starting work. Its quite common for the VC to lose contact with the ESX's it manages following an upgrade. Reconnecting requires some manual effort (right-click options through the VI Client, restart the ESX management agent, nothing too involved, just time consuming if you've lots of ESX's), and may require the root user/pass for the ESX's.

1. Start the installer
2. The installer will detect that its an upgrade and ask for confirmation to continue, click Yes
3. Once the wizard starts, click Next
4. Accept EULA, and then click Next
5. Confirm the database selection is correct, and then click Next
6. Enter the appropriate database authentication information (if required), and then click Next
7. You may receive warnings about any of the following (or other), this is normal, take note and proceed
   • Update Manager extension won't be supported - upgrade following vCentre upgrade
   • SQL server recovery model
   • SQL server to use TCP/IP and named pipes - config post upgrade via MS SQL Server Surface Area Config tool, Database Engines | Remote Connections
8. Unless you really want to blank your database and lose all configs, performance logs etc, leave on the default Do not overwrite... option and click Next
9. Unless you know otherwise, leave vCentre service account info as is and click Next
10. Leave TCP ports as is and click Next
11. Click Install
    • The upgrade will proceed and might take some time (5 - 15 mins)
    • Once the vCentre is upgraded its common for the vCentre agent that resides on the ESX's to also be upgraded, this occurs automatically once the ESX's reconnect after the vCentre upgrade.

Build Numbers

VC version	Server	Appliance
2.5 Update 1	84767
2.5 Update 2	104217
2.5 Update 3	119598
2.5 Update 4	147633
4.0	162902
4.0 Update 1	208156
4.1	258902
4.1 Update 1	345043
4.1 Update 2	493063
4.1 Update 3	816786
5.0	456005	472350
5.0 Update 1	639890
5.0 Update 1a	755629	757163
5.0 Update 1b	804276	804279
5.0 Update 2	913577	920217
5.1	799735
5.1a	880471	880472
5.1b	947939

Architecture Suggestions

The following are some suggestions on how set-up your VI management infrastructure, take all of the below with a large amount of consideration for your specific requirements. For example, even if you've a small number of ESX's, if you're always performing lots of tasks you may want to consider beefing up your management infrastructure.

• Install on Virtual machines (rather than physical)
  • VI management is resilient to hardware failures/maintenance through HA and vMotion
  • Besides, you can hardly expect service owners to be comfortable with virtualisation if you're not confident enough to locate your own systems on the platform!
• Install both vCentre and SQL on the same VM
  • Installing on separate machines is of little benefit, and if they're always together the VC service becomes impervious to network and other problems (it readily fails if it has trouble access the SQL database).
• Join the VM to the domain
  • Makes user management so much easier (and its more secure that way)
• Install vCentre to run using a Local System account
  • If a domain service account, you can't start vCentre without the domain, which might be a big problem if you're DC's are virtual and your starting up following a power-down
  • If a local admin/service account, you can't browse/add domain users/groups to your vCentre permissions
• Virtual hardware
  • 2 - 4 vCPU's (2 is fine for small installations)
  • 4 - 6 GB RAM (2 for OS, SQL, and vCentre respectively, 4GB total fine for small installations)

General Configuration

Sysprep Setup

The following Sysprep support directories were created during Virtual Center installation, these need to be populated in order for the VC to be able to sysprep VM's as they are deployed using Guest Customisation:

• Windows 2003 vCentre
  • %ALLUSERSPROFILE%\Application Data\Vmware\VMware VirtualCenter\sysprep\
  • C:\ProgramData\VMware\VMware VirtualCenter\sysprep\
• Windows 2008 vCentre
  • %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\sysprep\
  • C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep\

...\1.1\
...\2k\
...\xp\
...\svr2003\
...\xp-64\
...\svr2003-64\

Windows 2008 (and Vista) don’t need sysprep files as the function is built into the OS.

Download the ‘System Preparation Tool’ or ‘Deployment Tools’ package that contains ‘sysprep’ from the Microsoft download centre. Make sure that you download the correct version for the guest operating system and ‘Service Pack’ that you want to customize.

1.1: In the case of the generic version 1.1 simply run the downloaded executable to extract the files then copy the contents of the Tools directory into the appropriate directory (see below).

Windows 2000: In the case of Windows 2000 simply run the downloaded executable to extract the files into the appropriate directory (see below).

Win XP: In the case of XP run the downloaded executable to create deploy.cab then use something like WinZip to extract the files into the appropriate directory (see below).

Windows 2003: In the case of Windows 2003 the downloaded executable is a Hot Fix which must be run (Installed) on a matching OS to create deploy.cab then use something like WinZip to extract the files into the appropriate directory (see below).

TomCat Memory Throttle

The Tomcat webserver (used to provide VC Web-services), can be very memory hungry (up to 1GB) in comparison to when it ran on VC v2.5 (normally around 50MB). This can be throttled down, though this is only recommended for small, non-production installations (<10 ESX's). Bear in mind that things like API access (inc PowerCLI) are serviced by the VMware vCentre WebServices, so if these become sluggish you may need to increase. For home lab, single ESX installations you can reduce down to 64 MB.

1. Edit the registry to put a ceiling on the Java VM's memory usage
   • Append --JvmMx=256 (MB) to end of HKLM\SYSTEM\CurrentControlSet\Services\vctomcat\ImagePath
2. Restart the service
   • VMware VirtualCenter Management Webservices

ESX Time-out

An ESX send a UDP poll to its controlling vCentre, once every 10 secs. This times-out after 20 secs, and the vCentre will mark the ESX as disconnected. If you've got ESX's in a remote site or at the far end of a dodgy/overloaded network connection it won't uncommon for the UDP polls to get lost and your ESX's appear disconnected. To increase the time-out...

Within %APP_DATA%/VMware/VMware VirtualCenter/vpxd.cfg , edit the following section

<heartbeat>
  <notRespondingTimeout>60</notRespondingTimeout>
</heartbeat>

and restart the VMware Server Service

SQL Database

Migrate SQL2000 to SQL2005

This should be fairly painless (so long as your DBA knows his stuff!).

Before you start, you must ensure the Virtual Centre is using a SQL Native Client driver to access the database (see VMware KB1003391)...

1. Open up ODBC Data Source Administrator, and go to the System DSN tab
2. Check that the driver you're using is SQL Native Client
3. If not, you'll need to upgrade...
   1. Download from Microsoft and install SQL (see the VMware KB link above)
   2. Stop your Virtual Centre Server service
   3. Rename your existing DSN (eg append "old" to it)
   4. Create a new DSN using the new driver but otherwise exactly the same as the previous (the name MUST be the same)
   5. Restart the Virtual Centre Server service
   6. Login using the VI Client and ensure everything looks good (clusters, ESX's, VM's, Alarms, Guest Customisations etc)

To complete the migration...

1. Stop the Virtual Centre Server service
2. Backup the database (and log files) from your old SQL 2000 server
3. Restore the database (and log files) to your new SQL 2005 server
4. Ensure that your Virtual Centre's database account is the db_owner of the Virtual Centre database on SQL 2005 (as it was on the SQL2000 database)
5. Change the ODBC DSN settings on the VC server so it points to the SQL 2005 server (just change the server name, if you need to specify a non-standard TCP port use <SERVER>, <PORT> rather than <SERVER>:<PORT>)
6. Restart the Virtual Centre Server service
7. Login using the VI Client and ensure everything looks good (clusters, ESX's, VM's, Alarms, Guest Customisations etc)

SQL Database Maintenance

Summary of maintenance info from http://www.vmware.com/files/pdf/vc_microsoft_sql_server.pdf

Index Fragmentation

This maintenance procedure can be performed inline (ie without stopping the DB or VirtualCenter)

1. Log into the SQL database using Query Analyser
   1. Run up SQL Query Analyzer, and log into the SQL server
2. Perform query to check the level of fragmentation (may take 2 mins or so to run). Create and execute the following query;
   • USE DB (replace DB with whatever the database is called
   • GO
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
3. Check the query results for either of the following bad conditions
   • Scan Density < 90%
   • Logical Scan Fragmentation > 10%
4. If required, perform defragmentation by running the following query (will take some time depending on level of fragmentation)
   • DBCC INDEXDEFRAG ('DB', 'VPX_HIST_STAT', 'VPXII_HIST_STAT')
   • GO
5. Reheck the level of fragmentation using the following query (if Logical Scan Fragmentation is still > 30%, then may need to do reindex)
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
6. Update the statistics to continue providing accurate metrics, use the following query;
   • UPDATE STATISTICS VPX_HIST_STAT WITH FULLSCAN
   • GO

You may occasionally encounter a deadlock issue that causes the INDEXDEFRAG session to terminate. If the session terminates in this way, there is no adverse impact on the database, and the database continues running normally. You can issue the INDEXDEFRAG command again and the operation resumes where it left off.

Index Reindexing

This maintenance procedure can only be performed with the database offline, the VirtualCenter Server Service must be shutdown

1. Perform query to check the level of fragmentation (may take 2 mins or so to run). Create and execute the following query;
   • USE DB
   • GO
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
2. Check the query results for either of the following bad conditions
   • Scan Density < 70%
   • Logical Scan Fragmentation > 30%
3. Stop the VirtualCenter Server Service
4. Reindex the table by running this query;
   • DBCC DBREINDEX ('VPX_HIST_STAT', , 70)
   • GO
5. Once complete, restart the VirtualCenter Server Service

Other Tasks

Its also possible to delete old data from the database, this is achieved by running a script against the database, see http://kb.vmware.com/kb/1000125 for further info.

Troubleshooting

VMware VirtualCentre Server service won't start

Service-specific error 2 (0x2)

• Caused by the SQL service being unavailable, therefore investigate why this is so.

If SQL is running on the same server, and the service failed after a reboot, its likely that its starting too quickly and SQL isn't ready. For starters make the service depend on SQL and SQL Agent services, failing that, make the vCentre service start in a delayed fashion.

To make the VirtualCentre Server service depend on the SQL service

1. Find the name of the SQL service
   • Find the MSSQL and SQLAgent keys in the following hive
   • HKLM\System\CurrentControlSet\Services
   • Could be be MSSQL, MSSQL$SQLEXPRESS, or if you've used a named instance, something like MSSQL$VIM
2. Make VC service dependant on it
   • Browse to HKLM\System\CurrentControlSet\Services\vpxd
   • Add the name of the SQL service to the DependOnService value (there must be a blank line at the end still)

Interaction between vCentre and SQL can be quite poor when it comes to its start-up behaviour...

• SQL tends to report itself as started, despite the fact that it hasn't made its database instances available yet.
• vCentre will try to connect to SQL, then fail if it can't get in straight away

...meaning that you end up being reliant the vCentre service restarting in order for it to be able to connect and start up - which is far from ideal for normal operation.

Virtual Machine won't export

Generic Workarounds...

• Download the VM files from the datastore, this can then be uploaded to the intended destination and imported.
  • Remove any unnecessary log files
• Use the OVF Tool, this can succeed where vCentre fails, either
  • Convert locally downloaded VM files into an OVF
  • Connect to storage via the vCentre and create an OVF from there

Failed to Export Virtual Appliance: An item with the same key has already been added

• Caused by VM being exported having a running snapshot.

To resolve...

• Delete / Consolidate VM's snapshot(s)

Failed to export virtual appliance: Unexpected meta section

• Sometimes the exported doesn't handle creating a local folder to export to properly
• Discrepancy between VMX file and actual virtual hardware config
• Snapshot files (especially VMSD) may remain despite there being no active snapshot

To resolve...

• Re-attempt
• Backup and then delete any snapshot files (assuming you're sure there are no active snapshots - see Snapshot Still Active?), .vmsd and *-0000x.vmdk files
• Check VMX file for any discrepancies between itself and reality, backup and then correct the VMX file

System Error

Call "PropertyCollector.RetrieveContents" for object "propertyCollector" on vCenter Server

• Occurs when you try to perform a task on a VM (or possibly any vCentre object), seems to be due to corruption or conflicts within the vCentre database

To resolve...

• Re-add the VM to the inventory
  1. Ensure you know the following VM info; datastore, folder, resource pool, anything else!
  2. select the VM and Remove from Inventory
  3. Locate the .vmx file in the datastore and Add to Inventory'

Can't Remove Inaccessible Virtual Machine

VM no longer exists, or is no longer required, but can't remove from Virtual Centre as its in inaccessible state, and powered on.

1. Via SSH to the ESX that the VM belongs to, find its config file
   • vmware-cmd -l | grep My_VM
2. Unregister the VM from the owning ESX
   • vmware-cmd -s unregister /path/to/My_VM.vmx
   • Should return something like unregister(/path/to/My_VM.vmx) = 1
3. VM can now be unregistered from the VC

Orphaned VM

VM's appear orphaned if they're in the vCentre database, but are no longer being reported by the ESX to vCentre. Its likely that its still running (or has been restarted by HA).

1. Locate the VM (if there's been an HA failover its probably not where vCentre believes)
   • Connect a VI Client direct to each ESX, if the VM isn't shown, locate as follows
     1. SSH to an ESX, and cd to the VM's folder
     2. Run the following command
        • vmkfstools -D *.vmx
     3. The owning ESX's MAC address is shown in the end of the owner field in the following part of the return data
        • gen 443, mode 1, owner 4d0f65e4-d55ab729-0ea0-d48564638518 mtime 8831379]
     4. Run the following command on each ESX to find the server with that MAC address
        • esxcfg-nics -l
2. If the VM is shown on the ESX's VI Client, restart the ESX's management agent.
3. If the VM is not shown on the ESX's VI Client, browse the datastore and register the VM

VI Client Slow on Windows 7

Screen redraws can sometimes get quite slow and irritating, disabling desktop composition (and so also the Aero/see-through desktop features) helps to alleviate this.

1. Go to the Properties of the shortcut used to open the VI Client
2. In the Computability tab, tick Disable desktop composition
3. Restart any running VI Client instances

Note that the above have now been fixed in vSphere Client v4.1 Update 2 (build 491557) - even if you haven't upgraded you ESX or vCentre servers you can upgrade your client, find the Client download in the ESX update area - http://downloads.vmware.com/d/details/esx41u2/dHdlYnRoKmRidGRkKg==

VI Client Console White on Windows 7

When trying to open a VM console to VM hosted on a v3 ESX, the screen is white, viewing ESX4 hosted VM's is fine. Problem appears to be due to incomparability with .NET, which is expose if you install VI Clients in a particular order, to resolve...

1. Uninstall all versions of VI Client
2. Install the following in this order…
   1. VI Client v2.5 update 6 (build 227637)
   2. VI Client v4.0 update 2 (build 258672)
   3. VI Client v4.1 (build 258902)

If you don't have VC's with those build numbers available you'll need to source the VI Client installers, either from old vCentre install packages or direct from the VMware download site (you can download individual VI Client installers rather than the whole vCentre package

com.vmware.converter Error

In vCentre v4.1, you can get a "Health status monitoring" alert for the vCentre server, which from the vCentre Service Status page shows to be caused by an error with com.vmware.converter. Browsing to the server's page (eg https://vc-server/converter/health.xml) shows the Converter service status to be green.

1. Run ldp.exe (available in Win2k8, needs to be downloaded from Microsoft for Win2k3)
2. Go to Connection | Connect... and enter the hostname of the local vCentre Server
3. Go to Connection | Bind... and enter user/pass if required or just leave Bind as currently logged in user
4. Go to View | Tree, leave the Base DN blank and click OK
5. Double click through the following to get through to the vmw-vc-SSLThumbprint...
   1. DC=virtualcenter,DC=vmware,DC=int
   2. OU=Health,DC=virtualcenter,DC=vmware,DC=int
   3. OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   4. CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   5. CN=<GUID>.vpxd,CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   6. ... copy thumbprint value from right-hand pane to notepad
6. Double click through the following to get through to the vmw-vc-SSLThumbprint value stored by Converter health...
   1. DC=virtualcenter,DC=vmware,DC=int
   2. OU=Health,DC=virtualcenter,DC=vmware,DC=int
   3. OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   4. CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   5. CN=com.vmware.converter,CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
7. Backup the existing vmw-vc-SSLThumbprint entry to notepad
8. Replace the existing entry...
   1. Right-click over CN=com.vmware.converter,CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int and hit Modify
   2. In Attribute enter vmw-vc-SSLThumbprint
   3. In Values paste in the 1st value saved in notepad (strip the trailing ;)
   4. In Operation make sure 'Replace is selected
   5. Click Enter and then Run, then Close

Its takes a while (5 - 10 mins) for the fix to take effect, reboot the vCentre server if you're feeling impatient.

Resource Pool Corruption

Can manifest itself in various ways, you might have trouble re-registering VM's, or be unable to completely delete a resource. The corruption actually occurs at the ESX, from which you might see an error message similar to Error during the configuration of the host. Can not delete non-empty group: pool<n>.

1. Delete the resource pool from the vCentre interface (if not already done)
2. SSH to the ESX and delete the /etc/vmware/hostd/pools.xml folder
3. Restart the management agent
   • services.sh restart
4. Resource pools info get refreshed to the ESX after a min or so