Virtual Centre

Yes, I know, it should be Virtual Center (or as its now known vCenter) not Centre, but in English centre is spelt that way and that's the language we're using here, not a localised dialect. Any complaints, speak to the Queen, its her language.

Install

For a production installation you should have a ready to use SQL database server (MS or Oracle), for small installations you can use MS SQL Express (which is installed as part of the vCentre install when no external database is specified).

This is a basic install that will get you going, see Installing ESX 4.1 and vCenter Server 4.1 best practices for more detailed info, or the Architecture Suggestions section below for some hints.

SQL DSN requirements

• VC < v4.1 - Requires a 32-bit DSN even if vCentre is being run on a 64-bit machine
• VC v4.1 - Requires a 64-bit DSN
• vUM - * Requires a 32-bit DSN

1. Start the installer
2. Accept EULA, and then click Next
3. Enter vCentre license key (if you have one), and then click Next
4. Setup a database to use, and then click Next
   • Use MS SQL 2005 Express option for small scale or non-production deployments only
   • Select an existing DSN (to set-up a new DSN, see #Create DSN)
5. Enter the appropriate authentication information, and then click Next
6. Select the appropriate Linked Mode (this proc assumes Standalone), and then click Next
7. Accept the default TCP ports (its recommended that you don't share the server with other services which, for example) might necessitate changing the vCentre ports), and then click Next
8. Click Install

Create DSN

Use this procedure to create a new SQL DSN on the server on which you are installing vCentre, if your database is Oracle rather than Microsoft then you'll need to install an Oracle db driver 1st and select that rather than the SQL Native Client in this procedure. A blank/empty database should be create on the database server prior to carrying out this procedure.

If you are install vCentre on a 64-bit machine, you'll need to specifically create a 32-bit DSN. The proc below would create a 64-bit DSN if carried out on a 64-bit machine, if that's not you want to do, start the ODBC admin tool from %systemdrive%\Windows\SysWoW64\Odbcad32.exe instead of the Administrative Tools menu.

1. Go to Data Sources (ODBC) in the Administrative Tools menu in order to start the ODBC admin tool
2. Go to the System DSN tab and click on Add...
3. Select the SQL Native Client driver and click Finish
4. Enter the required options described below, and then select Next...
   • Name: This is the name that you'll know the DSN by, and use in the vCentre setup, eg VC
   • Description: Optional description for DSN, eg vCentre database
   • Server: Database server hostname, eg db1.company.com
5. Enter the appropriate authentication options for your database (ask your DBA if in doubt!), and then click Next
6. Check the Change the default database to option and select the appropriate database (that's been created on the database server, then click Next
7. Leave default options as is on final screen, and click Finish
8. Click the Test Data Source... button to test connectivity to the database, then OK to finish
   • If the test fails, recheck your config, and also test connectivity to the database server (ping, telnet porttest etc)

Install Update Manager

Update Manager can be installed either on a separate server, or on the same machine as vCentre. You need to have a SQL database available (as for vCentre, this can be an external database or MS SQL Express). You also need enough storage to keep local copies of all updates that will available for install to clients (especially when your VUM server is a VM, it can be useful to keep the patches on separate disk, so that's its easier to alter its size without affecting other data).

1. Start the installer
2. Accept EULA, and then click Next
3. Enter the local admin user/pass credentials for the server vCentre is running on, and then click Next
4. Setup a database to use, and then click Next
   • Use MS SQL 2005 Express option for small scale or non-production deployments only
   • Select an existing DSN (to set-up a new DSN, see #Create DSN)
5. Enter the appropriate authentication information, and then click Next
6. Accept the default TCP ports (its recommended that you don't share the server with other services which, for example) might necessitate changing the Update Manager ports). Tick the box if you need to configure a proxy, and then click Next
7. Change the ..location for downloading patches if you want you downloaded updates to stored on a separate disk, and click Next
8. Click Install

Once installed Update Manager will download patches and updates for hosts and VMs connected to the vCentre. To manage Update Manager...

1. With a VI Client connected to vCentre, go to Plug-ins | Manage Plug-ins...
2. For the ...Update Manager Extension, click on Download and Install...
3. Click on Next
4. Accept EULA, and then click Next
5. Click Install
6. Click Finish

Upgrade

Upgrades are performed using the same package as used for a fresh installation. Obviously VI management will be unavailable during the upgrade and the database should be backed up prior to starting work. Its quite common for the VC to lose contact with the ESX's it manages following an upgrade. Reconnecting requires some manual effort (right-click options through the VI Client, restart the ESX management agent, nothing too involved, just time consuming if you've lots of ESX's), and may require the root user/pass for the ESX's.

1. Start the installer
2. The installer will detect that its an upgrade and ask for confirmation to continue, click Yes
3. Once the wizard starts, click Next
4. Accept EULA, and then click Next
5. Confirm the database selection is correct, and then click Next
6. Enter the appropriate database authentication information (if required), and then click Next
7. You may receive warnings about any of the following (or other), this is normal, take note and proceed
   • Update Manager extension won't be supported - upgrade following vCentre upgrade
   • SQL server recovery model
   • SQL server to use TCP/IP and named pipes - config post upgrade via MS SQL Server Surface Area Config tool, Database Engines | Remote Connections
8. Unless you really want to blank your database and lose all configs, performance logs etc, leave on the default Do not overwrite... option and click Next
9. Unless you know otherwise, leave vCentre service account info as is and click Next
10. Leave TCP ports as is and click Next
11. Click Install
    • The upgrade will proceed and might take some time (5 - 15 mins)
    • Once the vCentre is upgraded its common for the vCentre agent that resides on the ESX's to also be upgraded, this occurs automatically once the ESX's reconnect after the vCentre upgrade.

Upgrade Update Manager

Upgrade vCentre before Update Manager! The database should be backed up prior to starting work.

1. Start the installer
2. The installer will detect that its an upgrade and ask for confirmation to continue, click Yes
3. Once the wizard starts, click Next
4. Accept EULA, and then click Next
5. Enter the local admin user/pass credentials for the server vCentre is running on, and then click Next
6. Enter the appropriate database authentication information (if required), and then click Next
7. Leave on the default Yes, I want to upgrade..., and tick to confirm you've taken a backup, then click Next
8. Click Install
9. Click Finish

Build Numbers

VC version	Build
2.5 Update 1	84767
2.5 Update 2	104217
2.5 Update 3	119598
2.5 Update 4	147633
4.0	162902
4.0 Update 1	208156
4.1	258902
4.1 Update 1	345043

Architecture Suggestions

The following are some suggestions on how set-up your VI management infrastructure, take all of the below with a large amount of consideration for your specific requirements. For example, even if you've a small number of ESX's, if you're always performing lots of tasks you may want to consider beefing up your management infrastructure.

• Install on Virtual machines (rather than physical)
  • VI management is resilient to hardware failures/maintenance through HA and vMotion
  • Besides, you can hardly expect service owners to be comfortable with virtualisation if you're not confident enough to locate your own systems on the platform!
• Install both vCentre and SQL on the same VM
  • Installing on separate machines is of little benefit, and if they're always together the VC service becomes impervious to network and other problems (it readily fails if it has trouble access the SQL database).
• Join the VM to the domain
  • Makes user management so much easier (and its more secure that way)
• Install vCentre to run using a Local System account
  • If a domain service account, you can't start vCentre without the domain, which might be a big problem if you're DC's are virtual and your starting up following a power-down
  • If a local admin/service account, you can't browse/add domain users/groups to your vCentre permissions
• Virtual hardware
  • 2 - 4 vCPU's (2 is fine for small installations)
  • 4 - 6 GB RAM (2 for OS, SQL, and vCentre respectively, 4GB total fine for small installations)

General Configuration

Sysprep Setup

The following Sysprep support directories were created during Virtual Center installation, these need to be populated in order for the VC to be able to sysprep VM's as they are deployed using Guest Customisation:

• Windows 2003 vCentre
  • %ALLUSERSPROFILE%\Application Data\Vmware\VMware VirtualCenter\sysprep\
  • C:\ProgramData\VMware\VMware VirtualCenter\sysprep\
• Windows 2008 vCentre
  • %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\sysprep\
  • C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep\

...\1.1\
...\2k\
...\xp\
...\svr2003\
...\xp-64\
...\svr2003-64\

Windows 2008 (and Vista) don’t need sysprep files as the function is built into the OS.

Download the ‘System Preparation Tool’ or ‘Deployment Tools’ package that contains ‘sysprep’ from the Microsoft download centre. Make sure that you download the correct version for the guest operating system and ‘Service Pack’ that you want to customize.

1.1: In the case of the generic version 1.1 simply run the downloaded executable to extract the files then copy the contents of the Tools directory into the appropriate directory (see below).

Windows 2000: In the case of Windows 2000 simply run the downloaded executable to extract the files into the appropriate directory (see below).

Win XP: In the case of XP run the downloaded executable to create deploy.cab then use something like WinZip to extract the files into the appropriate directory (see below).

Windows 2003: In the case of Windows 2003 the downloaded executable is a Hot Fix which must be run (Installed) on a matching OS to create deploy.cab then use something like WinZip to extract the files into the appropriate directory (see below).

TomCat Memory Throttle

The Tomcat webserver (used to provide VC Web-services), can be very memory hungry (up to 1GB) in comparison to when it ran on VC v2.5 (normally around 50MB). This can be throttled down, though this is only recommended for small, non-production installations (<10 ESX's). Bear in mind that things like API access (inc PowerCLI) are serviced by the VMware vCentre WebServices, so if these become sluggish you may need to increase. For home lab, single ESX installations you can reduce down to 64 MB.

1. Edit the registry to put a ceiling on the Java VM's memory usage
   • Append --JvmMx=256 (MB) to end of HKLM\SYSTEM\CurrentControlSet\Services\vctomcat\ImagePath
2. Restart the service
   • VMware VirtualCenter Management Webservices

ESX Time-out

An ESX send a UDP poll to its controlling vCentre, once every 10 secs. This times-out after 20 secs, and the vCentre will mark the ESX as disconnected. If you've got ESX's in a remote site or at the far end of a dodgy/overloaded network connection it won't uncommon for the UDP polls to get lost and your ESX's appear disconnected. To increase the time-out...

Within %APP_DATA%/VMware/VMware VirtualCenter/vpxd.cfg , edit the following section

<heartbeat>
  <notRespondingTimeout>60</notRespondingTimeout>
</heartbeat>

and restart the VMware Server Service

Update Manager

Import Offline Bundle

Useful for when you need to apply additional drivers etc to your ESX's

Often drivers are only available as an ISO, in which case you need to mount the image and extract the offline bundle ZIP file (use something like WinImage

1. Open Update Manager Administrator, and go to the Patch Repository tab
2. Click on the Import Patches link
3. Locate the ZIP file and click next, the bundle will be imported

In order to deploy the bundle you'll need to create a baseline with the bundle as a member. You've three choices here

• Create an individual baseline for that bundle, then when newer versions of the driver become available, just add it to the existing baseline
  • Best for situations where you've different types of hardware being managed by the same vCentre and so you need to distinguiish between different ESX's and different bundles
• Add the bundle to an existing baseline
  • Best for when you've got uniform hardware, and you like to apply uniform waves (aka controlled releases) of deployments to your ESX's
• Create an individual baseline for that bundle, that gets added to when new versions become available, and add the baseline to a baseline group which applies to clusters etc (hybrid of both of the above)

Import updates from another VC

Useful when you've a isolated VC (without internet access), and another with a full compliment of updates.

1. Copy the updates from to the VC, copy the entire Data directory
   • Default location for updates C:\Documents and Settings\All Users\Application Data\VMware\Vmware Update Manager\Data
2. On the isolated VC, run a version of the following example command to update the Update Manager database
   • vmware-updateDownloadCli -p <path to patches>\Data -f esx -U <user ac>

Scan/Remediate fails

• Error: VMware Update Manager had a failure
• Detail: Patch metadata for <ESX> missing, please download patch metadata first, despite the fact that Update Manager is fully up to date.
• /var/log/vmware/esxupdate.log shows connection errors

1. On the VC, stop the VMware Update Manager Service
2. Edit vci-integrity.xml file
   • Normally found in C:\Program Files\VMware\Infrastructure\Update Manager\
3. Edit the following entry, adding the VC IP address
   • <PatchDepotUrl>http://VC-IP/vci/hostupdates/hostupdate</PatchDepotUrl>
4. Restart the VMware Update Manager Service

• Error: VMware Update Manager had an unknown error
• Detial: vim.fault.noHost

1. Check that you have no inaccessible/invalid/orphaned VM's on the host, if so resolve
2. Disconnect the ESX from vCentre and reconnect

Best Practice

As advised by VMware, see http://www.vmware.com/files/pdf/techpaper/vsp_41_perf_UpdateManager_Best-Practices.pdf for further info

• Separate the Update Manager database from the vCenter database when there are 300+ virtual machines or 30+ hosts.
• Separate both the Update Manager server and the Update Manager database from the vCenter Server system and the vCenter Server database when there are 1000+ virtual machines or 100+ hosts.
• Make sure the Update Manager server host has at least 2GB of RAM to cache patch files in memory.
• Allocate separate physical disks for the Update Manager patch store and the Update Manager database.
• Deploy the Update Manager server close to the ESX hosts if possible. This reduces network latency and packet drops.
• On a high-latency network, powered-on virtual machine scans are preferred because they are not sensitive to network latency.
• Host operations in a slow network will take a long time. Refer to the white paper for the maximum time estimation. Don’t interrupt ongoing operations.

SQL Database

Migrate SQL2000 to SQL2005

This should be fairly painless (so long as your DBA knows his stuff!).

Before you start, you must ensure the Virtual Centre is using a SQL Native Client driver to access the database (see VMware KB1003391)...

1. Open up ODBC Data Source Administrator, and go to the System DSN tab
2. Check that the driver you're using is SQL Native Client
3. If not, you'll need to upgrade...
   1. Download from Microsoft and install SQL (see the VMware KB link above)
   2. Stop your Virtual Centre Server service
   3. Rename your existing DSN (eg append "old" to it)
   4. Create a new DSN using the new driver but otherwise exactly the same as the previous (the name MUST be the same)
   5. Restart the Virtual Centre Server service
   6. Login using the VI Client and ensure everything looks good (clusters, ESX's, VM's, Alarms, Guest Customisations etc)

To complete the migration...

1. Stop the Virtual Centre Server service
2. Backup the database (and log files) from your old SQL 2000 server
3. Restore the database (and log files) to your new SQL 2005 server
4. Ensure that your Virtual Centre's database account is the db_owner of the Virtual Centre database on SQL 2005 (as it was on the SQL2000 database)
5. Change the ODBC DSN settings on the VC server so it points to the SQL 2005 server (just change the server name, if you need to specify a non-standard TCP port use <SERVER>, <PORT> rather than <SERVER>:<PORT>)
6. Restart the Virtual Centre Server service
7. Login using the VI Client and ensure everything looks good (clusters, ESX's, VM's, Alarms, Guest Customisations etc)

SQL Database Maintenance

Summary of maintenance info from http://www.vmware.com/files/pdf/vc_microsoft_sql_server.pdf

Index Fragmentation

This maintenance procedure can be performed inline (ie without stopping the DB or VirtualCenter)

1. Log into the SQL database using Query Analyser
   1. Run up SQL Query Analyzer, and log into the SQL server
2. Perform query to check the level of fragmentation (may take 2 mins or so to run). Create and execute the following query;
   • USE DB (replace DB with whatever the database is called
   • GO
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
3. Check the query results for either of the following bad conditions
   • Scan Density < 90%
   • Logical Scan Fragmentation > 10%
4. If required, perform defragmentation by running the following query (will take some time depending on level of fragmentation)
   • DBCC INDEXDEFRAG ('DB', 'VPX_HIST_STAT', 'VPXII_HIST_STAT')
   • GO
5. Reheck the level of fragmentation using the following query (if Logical Scan Fragmentation is still > 30%, then may need to do reindex)
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
6. Update the statistics to continue providing accurate metrics, use the following query;
   • UPDATE STATISTICS VPX_HIST_STAT WITH FULLSCAN
   • GO

You may occasionally encounter a deadlock issue that causes the INDEXDEFRAG session to terminate. If the session terminates in this way, there is no adverse impact on the database, and the database continues running normally. You can issue the INDEXDEFRAG command again and the operation resumes where it left off.

Index Reindexing

This maintenance procedure can only be performed with the database offline, the VirtualCenter Server Service must be shutdown

1. Perform query to check the level of fragmentation (may take 2 mins or so to run). Create and execute the following query;
   • USE DB
   • GO
   • DBCC SHOWCONTIG (VPX_HIST_STAT,VPXII_HIST_STAT)
   • GO
2. Check the query results for either of the following bad conditions
   • Scan Density < 70%
   • Logical Scan Fragmentation > 30%
3. Stop the VirtualCenter Server Service
4. Reindex the table by running this query;
   • DBCC DBREINDEX ('VPX_HIST_STAT', , 70)
   • GO
5. Once complete, restart the VirtualCenter Server Service

Other Tasks

Its also possible to delete old data from the database, this is achieved by running a script against the database, see http://kb.vmware.com/kb/1000125 for further info.

Troubleshooting

VMware VirtualCentre Server service won't start

Service-specific error 2 (0x2)

• Caused by the SQL service being unavailable, therefore investigate why this is so.

If SQL is running on the same server, and the service failed after a reboot, its likely that its starting too quickly and SQL isn't ready. Either, make the service automatically restart after X mins (good practise anyway), and/or make the service depend on SQL.

To make the VirtualCentre Server service depend on the SQL service

1. Find the name of the SQL service
   • Find the MSSQL key in the following hive (will probably be MSSQL, MSSQL$SQLEXPRESS, etc)
   • HKLM\System\CurrentControlSet\Services
2. Make VC service dependant on it
   • Browse to HKLM\System\CurrentControlSet\Services\vpxd
   • Add the name of the SQL service to the DependOnService value (there must be a blank line at the end still)

Virtual Machine won't export

Generic Workarounds...

• Download the VM files from the datastore, this can then be uploaded to the intended destination and imported.
  • Remove any unnecessary log files
• Use the OVF Tool, this can succeed where vCentre fails, either
  • Convert locally downloaded VM files into an OVF
  • Connect to storage via the vCentre and create an OVF from there

Failed to Export Virtual Appliance: An item with the same key has already been added

• Caused by VM being exported having a running snapshot.

To resolve...

• Delete / Consolidate VM's snapshot(s)

Failed to export virtual appliance: Unexpected meta section

• Sometimes the exported doesn't handle creating a local folder to export to properly
• Discrepancy between VMX file and actual virtual hardware config
• Snapshot files (especially VMSD) may remain despite there being no active snapshot

To resolve...

• Re-attempt
• Backup and then delete any snapshot files (assuming you're sure there are no active snapshots - see Snapshot Still Active?), .vmsd and *-0000x.vmdk files
• Check VMX file for any discrepancies between itself and reality, backup and then correct the VMX file

System Error

Call "PropertyCollector.RetrieveContents" for object "propertyCollector" on vCenter Server

• Occurs when you try to perform a task on a VM (or possibly any vCentre object), seems to be due to corruption or conflicts within the vCentre database

To resolve...

• Re-add the VM to the inventory
  1. Ensure you know the following VM info; datastore, folder, resource pool, anything else!
  2. select the VM and Remove from Inventory
  3. Locate the .vmx file in the datastore and Add to Inventory'

Can't Remove Inaccessible Virtual Machine

VM no longer exists, or is no longer required, but can't remove from Virtual Centre as its in inaccessible state, and powered on.

1. Via SSH to the ESX that the VM belongs to, find its config file
   • vmware-cmd -l | grep My_VM
2. Unregister the VM from the owning ESX
   • vmware-cmd -s unregister /path/to/My_VM.vmx
   • Should return something like unregister(/path/to/My_VM.vmx) = 1
3. VM can now be unregistered from the VC

Orphaned VM

VM's appear orphaned if they're in the vCentre database, but are no longer being reported by the ESX to vCentre. Its likely that its still running (or has been restarted by HA).

1. Locate the VM (if there's been an HA failover its probably not where vCentre believes)
   • Connect a VI Client direct to each ESX, if the VM isn't shown, locate as follows
     1. SSH to an ESX, and cd to the VM's folder
     2. Run the following command
        • vmkfstools -D *.vmx
     3. The owning ESX's MAC address is shown in the end of the owner field in the following part of the return data
        • gen 443, mode 1, owner 4d0f65e4-d55ab729-0ea0-d48564638518 mtime 8831379]
     4. Run the following command on each ESX to find the server with that MAC address
        • esxcfg-nics -l
2. If the VM is shown on the ESX's VI Client, restart the ESX's management agent.
3. If the VM is not shown on the ESX's VI Client, browse the datastore and register the VM

VI Client Slow on Windows 7

Screen redraws can sometimes get quite slow and irritating, disabling desktop composition (and so also the Aero/see-through desktop features) helps to alleviate this.

1. Go to the Properties of the shortcut used to open the VI Client
2. In the Computability tab, tick Disable desktop composition
3. Restart any running VI Client instances

VI Client Console White on Windows 7

When trying to open a VM console to VM hosted on a v3 ESX, the screen is white, viewing ESX4 hosted VM's is fine. Problem appears to be due to incomparability with .NET, which is expose if you install VI Clients in a particular order, to resolve...

1. Uninstall all versions of VI Client
2. Install the following in this order…
   1. VI Client v2.5 update 6 (build 227637)
   2. VI Client v4.0 update 2 (build 258672)
   3. VI Client v4.1 (build 258902)

If you don't have VC's with those build numbers available you'll need to source the VI Client installers, either from old vCentre install packages or direct from the VMware download site (you can download individual VI Client installers rather than the whole vCentre package

com.vmware.converter Error

In vCentre v4.1, you can get a "Health status monitoring" alert for the vCentre server, which from the vCentre Service Status page shows to be caused by an error with com.vmware.converter. Browsing to the server's page (eg https://vc-server/converter/health.xml) shows the Converter service status to be green.

1. Run ldp.exe (available in Win2k8, needs to be downloaded from Microsoft for Win2k3)
2. Go to Connection | Connect... and enter the hostname of the local vCentre Server
3. Go to Connection | Bind... and enter user/pass if required or just leave Bind as currently logged in user
4. Go to View | Tree, leave the Base DN blank and click OK
5. Double click through the following to get through to the vmw-vc-SSLThumbprint...
   1. DC=virtualcenter,DC=vmware,DC=int
   2. OU=Health,DC=virtualcenter,DC=vmware,DC=int
   3. OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   4. CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   5. CN=<GUID>.vpxd,CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   6. ... copy thumbprint value from right-hand pane to notepad
6. Double click through the following to get through to the vmw-vc-SSLThumbprint value stored by Converter health...
   1. DC=virtualcenter,DC=vmware,DC=int
   2. OU=Health,DC=virtualcenter,DC=vmware,DC=int
   3. OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   4. CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
   5. CN=com.vmware.converter,CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int
7. Backup the existing vmw-vc-SSLThumbprint entry to notepad
8. Replace the existing entry...
   1. Right-click over CN=com.vmware.converter,CN=<GUID>,OU=ComponentSpecs,OU=Health,DC=virtualcenter,DC=vmware,DC=int and hit Modify
   2. In Attribute enter vmw-vc-SSLThumbprint
   3. In Values paste in the 1st value saved in notepad (strip the trailing ;)
   4. In Operation make sure 'Replace is selected
   5. Click Enter and then Run, then Close

Its takes a while (5 - 10 mins) for the fix to take effect, reboot the vCentre server if you're feeling impatient.