Virtual Centre

Yes, I know, it should be Virtual Center (or as its now known vCenter) not Centre, but in English centre is spelt that way and that's the language we're using here, not a localised dialect. Any complaints, speak to the Queen, its her language.

Install

For a production installation you should have a ready to use SQL database server (MS or Oracle), for small installations you can use MS SQL Express (which is installed as part of the vCentre install when no external database is specified).

This is a basic install that will get you going, see Installing ESX 4.1 and vCenter Server 4.1 best practices for more info.

1. Start the installer
2. Accept EULA, and then click Next
3. Enter vCentre license key (if you have one), and then click Next
4. Setup a database to use, and then click Next
   • Use MS SQL 2005 Express option for small scale or non-production deployments only
   • Select an existing DSN (to set-up a new DSN, see #Create DSN)
5. Enter the appropriate authentication information, and then click Next
6. Select the appropriate Linked Mode (this proc assumes Standalone), and then click Next
7. Accept the default TCP ports (its recommended that you don't share the server with other services which, for example) might necessitate changing the vCentre ports), and then click Next
8. Click Install

Create DSN

Use this procedure to create a new SQL DSN on the server on which you are installing vCentre, if your database is Oracle rather than Microsoft then you'll need to install an Oracle db driver 1st and select that rather than the SQL Native Client in this procedure. A blank/empty database should be create on the database server prior to carrying out this procedure.

1. Go to Data Sources (ODBC) in the Administrative Tools menu
2. Go to the System DSN tab and click on Add...
3. Select the SQL Native Client driver and click Finish
4. Enter the required options described below, and then select Next...
   • Name: This is the name that you'll know the DSN by, and use in the vCentre setup, eg VC
   • Description: Optional description for DSN, eg vCentre database
   • Server: Database server hostname, eg db1.company.com
5. Enter the appropriate authentication options for your database (ask your DBA if in doubt!), and then click Next
6. Check the Change the default database to option and select the appropriate database (that's been created on the database server, then click Next
7. Leave default options as is on final screen, and click Finish
8. Click the Test Data Source... button to test connectivity to the database, then OK to finish
   • If the test fails, recheck your config, and also test connectivity to the database server (ping, telnet porttest etc)

Install Update Manager

Update Manager can be installed either on a separate server, or on the same machine as vCentre. You need to have a SQL database available (as for vCentre, this can be an external database or MS SQL Express). You also need enough storage to keep local copies of all updates that will available for install to clients (especially when your VUM server is a VM, it can be useful to keep the patches on separate disk, so that's its easier to alter its size without affecting other data).

1. Start the installer
2. Accept EULA, and then click Next
3. Enter the local admin user/pass credentials for the server vCentre is running on, and then click Next
4. Setup a database to use, and then click Next
   • Use MS SQL 2005 Express option for small scale or non-production deployments only
   • Select an existing DSN (to set-up a new DSN, see #Create DSN)
5. Enter the appropriate authentication information, and then click Next
6. Accept the default TCP ports (its recommended that you don't share the server with other services which, for example) might necessitate changing the Update Manager ports). Tick the box if you need to configure a proxy, and then click Next
7. Change the ..location for downloading patches if you want you downloaded updates to stored on a separate disk, and click Next
8. Click Install

Once installed Update Manager will download patches and updates for hosts and VMs connected to the vCentre. To manage Update Manager...

1. With a VI Client connected to vCentre, go to Plug-ins | Manage Plug-ins...
2. For the ...Update Manager Extension, click on Download and Install...
3. Click on Next
4. Accept EULA, and then click Next
5. Click Install
6. Click Finish

Upgrade

Upgrades are performed using the same package as used for a fresh installation. Obviously VI management will be unavailable during the upgrade and the database should be backed up prior to starting work. Its quite common for the VC to lose contact with the ESX's it manages following an upgrade. Reconnecting requires some manual effort (right-click options through the VI Client, restart the ESX management agent, nothing too involved, just time consuming if you've lots of ESX's), and may require the root user/pass for the ESX's.

1. Start the installer
2. The installer will detect that its an upgrade and ask for confirmation to continue, click Yes
3. Once the wizard starts, click Next
4. Accept EULA, and then click Next
5. Confirm the database selection is correct, and then click Next
6. Enter the appropriate database authentication information (if required), and then click Next
7. You may receive warnings about any of the following (or other), this is normal, take note and proceed
   • Update Manager extension won't be supported - upgrade following vCentre upgrade
   • SQL server recovery model
   • SQL server to use TCP/IP and named pipes - config post upgrade via MS SQL Server Surface Area Config tool, Database Engines | Remote Connections
8. Unless you really want to blank your database and lose all configs, performance logs etc, leave on the default Do not overwrite... option and click Next
9. Unless you know otherwise, leave vCentre service account info as is and click Next
10. Leave TCP ports as is and click Next
11. Click Install
    • The upgrade will proceed and might take some time (5 - 15 mins)
    • Once the vCentre is upgraded its common for the vCentre agent that resides on the ESX's to also be upgraded, this occurs automatically once the ESX's reconnect after the vCentre upgrade.

Upgrade Update Manager

Upgrade vCentre before Update Manager! The database should be backed up prior to starting work.

1. Start the installer
2. The installer will detect that its an upgrade and ask for confirmation to continue, click Yes
3. Once the wizard starts, click Next
4. Accept EULA, and then click Next
5. Enter the local admin user/pass credentials for the server vCentre is running on, and then click Next
6. Enter the appropriate database authentication information (if required), and then click Next
7. Leave on the default Yes, I want to upgrade..., and tick to confirm you've taken a backup, then click Next
8. Click Install
9. Click Finish

General Configuration

Sysprep Setup

The following Sysprep support directories were created during Virtual Center installation, these need to be populated in order for the VC to be able to sysprep VM's as they are deployed using Guest Customisation:

• Windows 2003 vCentre
  • %ALLUSERSPROFILE%\Application Data\Vmware\VMware VirtualCenter\sysprep\
  • C:\ProgramData\VMware\VMware VirtualCenter\sysprep\
• Windows 2008 vCentre
  • %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\sysprep\
  • C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep\

...\1.1\
...\2k\
...\xp\
...\svr2003\
...\xp-64\
...\svr2003-64\

Windows 2008 (and Vista) don’t need sysprep files as the function is built into the OS.

Download the ‘System Preparation Tool’ or ‘Deployment Tools’ package that contains ‘sysprep’ from the Microsoft download centre. Make sure that you download the correct version for the guest operating system and ‘Service Pack’ that you want to customize.

1.1: In the case of the generic version 1.1 simply run the downloaded executable to extract the files then copy the contents of the Tools directory into the appropriate directory (see below).

Windows 2000: In the case of Windows 2000 simply run the downloaded executable to extract the files into the appropriate directory (see below).

Win XP: In the case of XP run the downloaded executable to create deploy.cab then use something like WinZip to extract the files into the appropriate directory (see below).

Windows 2003: In the case of Windows 2003 the downloaded executable is a Hot Fix which must be run (Installed) on a matching OS to create deploy.cab then use something like WinZip to extract the files into the appropriate directory (see below).

TomCat Memory Throttle

The Tomcat webserver (used to provide VC Web-services), can be very memory hungry (up to 1GB) in comparison to when it ran on VC v2.5 (normally around 50MB). This can be throttled down, though this is only recommended for small, non-production installations (<10 ESX's).

1. Edit the registry to put a ceiling on the Java VM's memory usage
   1. Append --JvmMx=256 (MB) to end of HKLM\SYSTEM\CurrentControlSet\Services\vctomcat\ImagePath
2. Restart the service
   1. VMware VirtualCenter Management Webservices

ESX Time-out

An ESX send a UDP poll to its controlling vCentre, once every 10 secs. This times-out after 20 secs, and the vCentre will mark the ESX as disconnected. If you've got ESX's in a remote site or at the far end of a dodgy/overloaded network connection it won't uncommon for the UDP polls to get lost and your ESX's appear disconnected. To increase the time-out...

Within %APP_DATA%/VMware/VMware VirtualCenter/vpxd.cfg , edit the following section

<heartbeat>
  <notRespondingTimeout>60</notRespondingTimeout>
</heartbeat>

and restart the VMware Server Service

Update Manager

Import updates from another VC

Useful when you've a isolated VC (without internet access), and another with a full compliment of updates.

1. Copy the updates from to the VC, copy the entire Data directory
   • Default location for updates C:\Documents and Settings\All Users\Application Data\VMware\Vmware Update Manager\Data
2. On the isolated VC, run a version of the following example command to update the Update Manager database
   • vmware-updateDownloadCli -p <path to patches>\Data -f esx -U <user ac>

Scan/Remediate fails

• Error: VMware Update Manager had a failure
• Detail: Patch metadata for <ESX> missing, please download patch metadata first, despite the fact that Update Manager is fully up to date.
• /var/log/vmware/esxupdate.log shows connection errors

1. On the VC, stop the VMware Update Manager Service
2. Edit vci-integrity.xml file
   • Normally found in C:\Program Files\VMware\Infrastructure\Update Manager\
3. Edit the following entry, adding the VC IP address
   • <PatchDepotUrl>http://VC-IP/vci/hostupdates/hostupdate</PatchDepotUrl>
4. Restart the VMware Update Manager Service

Best Practice

As advised by VMware, see http://www.vmware.com/files/pdf/techpaper/vsp_41_perf_UpdateManager_Best-Practices.pdf for further info

• Separate the Update Manager database from the vCenter database when there are 300+ virtual machines or 30+ hosts.
• Separate both the Update Manager server and the Update Manager database from the vCenter Server system and the vCenter Server database when there are 1000+ virtual machines or 100+ hosts.
• Make sure the Update Manager server host has at least 2GB of RAM to cache patch files in memory.
• Allocate separate physical disks for the Update Manager patch store and the Update Manager database.
• Deploy the Update Manager server close to the ESX hosts if possible. This reduces network latency and packet drops.
• On a high-latency network, powered-on virtual machine scans are preferred because they are not sensitive to network latency.
• Host operations in a slow network will take a long time. Refer to the white paper for the maximum time estimation. Don’t interrupt ongoing operations.

SQL Database

Migrate SQL2000 to SQL2005

This should be fairly painless (so long as your DBA knows his stuff!).

Before you start, you must ensure the Virtual Centre is using a SQL Native Client driver to access the database (see VMware KB1003391)...

1. Open up ODBC Data Source Administrator, and go to the System DSN tab
2. Check that the driver you're using is SQL Native Client
3. If not, you'll need to upgrade...
   1. Download from Microsoft and install SQL (see the VMware KB link above)
   2. Stop your Virtual Centre Server service
   3. Rename your existing DSN (eg append "old" to it)
   4. Create a new DSN using the new driver but otherwise exactly the same as the previous (the name MUST be the same)
   5. Restart the Virtual Centre Server service
   6. Login using the VI Client and ensure everything looks good (clusters, ESX's, VM's, Alarms, Guest Customisations etc)

To complete the migration...

1. Stop the Virtual Centre Server service
2. Backup the database (and log files) from your old SQL 2000 server
3. Restore the database (and log files) to your new SQL 2005 server
4. Ensure that your Virtual Centre's database account is the db_owner of the Virtual Centre database on SQL 2005 (as it was on the SQL2000 database)
5. Change the ODBC DSN settings on the VC server so it points to the SQL 2005 server (just change the server name, if you need to specify a non-standard TCP port use <SERVER>, <PORT> rather than <SERVER>:<PORT>)
6. Restart the Virtual Centre Server service
7. Login using the VI Client and ensure everything looks good (clusters, ESX's, VM's, Alarms, Guest Customisations etc)

SQL Database Maintenance

Summary of maintenance info from http://www.vmware.com/files/pdf/vc_microsoft_sql_server.pdf

Index Fragmentation

This maintenance procedure can be performed inline (ie without stopping the DB or VirtualCenter)

1. Log into the SQL database using Query Analyser
   1. Run up SQL Query Analyzer, and log into the SQL server
2. Perform query to check the level of fragmentation (may take 2 mins or so to run). Create and execute the following query;
   • USE DB (replace DB with whatever the database is called
   • GO
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
3. Check the query results for either of the following bad conditions
   • Scan Density < 90%
   • Logical Scan Fragmentation > 10%
4. If required, perform defragmentation by running the following query (will take some time depending on level of fragmentation)
   • DBCC INDEXDEFRAG ('DB', 'VPX_HIST_STAT', 'VPXII_HIST_STAT')
   • GO
5. Reheck the level of fragmentation using the following query (if Logical Scan Fragmentation is still > 30%, then may need to do reindex)
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
6. Update the statistics to continue providing accurate metrics, use the following query;
   • UPDATE STATISTICS VPX_HIST_STAT WITH FULLSCAN
   • GO

You may occasionally encounter a deadlock issue that causes the INDEXDEFRAG session to terminate. If the session terminates in this way, there is no adverse impact on the database, and the database continues running normally. You can issue the INDEXDEFRAG command again and the operation resumes where it left off.

Index Reindexing

This maintenance procedure can only be performed with the database offline, the VirtualCenter Server Service must be shutdown

1. Perform query to check the level of fragmentation (may take 2 mins or so to run). Create and execute the following query;
   • USE DB
   • GO
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
2. Check the query results for either of the following bad conditions
   • Scan Density < 70%
   • Logical Scan Fragmentation > 30%
3. Stop the VirtualCenter Server Service
4. Reindex the table by running this query;
   • DBCC DBREINDEX ('VPX_HIST_STAT', , 70)
   • GO
5. Once complete, restart the VirtualCenter Server Service

Other Tasks

Its also possible to delete old data from the database, this is achieved by running a script against the database, see http://kb.vmware.com/kb/1000125 for further info.

Troubleshooting

VMware VirtualCentre Server service won't start

Service-specific error 2 (0x2)

• Caused by the SQL service being unavailable, therefore investigate why this is so.

If SQL is running on the same server, and the service failed after a reboot, its likely that its starting too quickly and SQL isn't ready. Either, make the service automatically restart after X mins (good practise anyway), and/or make the service depend on SQL.

To make the VirtualCentre Server service depend on the SQL service

1. Find the name of the SQL service
   • Find the MSSQL key in the following hive (will probably be MSSQL, MSSQL$SQLEXPRESS, etc)
   • HKLM\System\CurrentControlSet\Services
2. Make VC service dependant on it
   • Browse to HKLM\System\CurrentControlSet\Services\vpxd
   • Add the name of the SQL service to the DependOnService value (there must be a blank line at the end still)

Virtual Machine won't export

Generic Workarounds...

• Download the VM files from the datastore, this can then be uploaded to the intended destination and imported.
  • Remove any unnecessary log files
• Use the OVF Tool, this can succeed where vCentre fails, either
  • Convert locally downloaded VM files into an OVF
  • Connect to storage via the vCentre and create an OVF from there

Failed to Export Virtual Appliance: An item with the same key has already been added

• Caused by VM being exported having a running snapshot.

To resolve...

• Delete / Consolidate VM's snapshot(s)

Failed to export virtual appliance: Unexpected meta section

• Sometimes the exported doesn't handle creating a local folder to export to properly
• Discrepancy between VMX file and actual virtual hardware config
• Snapshot files (especially VMSD) may remain despite there being no active snapshot

To resolve...

• Re-attempt
• Backup and then delete any snapshot files (assuming you're sure there are no active snapshots - see Snapshot Still Active?), .vmsd and *-0000x.vmdk files
• Check VMX file for any discrepancies between itself and reality, backup and then correct the VMX file

System Error

Call "PropertyCollector.RetrieveContents" for object "propertyCollector" on vCenter Server

• Occurs when you try to perform a task on a VM (or possibly any vCentre object), seems to be due to corruption or conflicts within the vCentre database

To resolve...

• Re-add the VM to the inventory
  1. Ensure you know the following VM info; datastore, folder, resource pool, anything else!
  2. select the VM and Remove from Inventory
  3. Locate the .vmx file in the datastore and Add to Inventory'

Can't Remove Inaccessible Virtual Machine

VM no longer exists, or is no longer required, but can't remove from Virtual Centre as its in inaccessible state, and powered on.

1. Via SSH to the ESX that the VM belongs to, find its config file
   • vmware-cmd -l | grep My_VM
2. Unregister the VM from the owning ESX
   • vmware-cmd -s unregister /path/to/My_VM.vmx
   • Should return something like unregister(/path/to/My_VM.vmx) = 1
3. VM can now be unregistered from the VC