ESX3 Installation

Requirements

Media

Current

(Versions current as of Feb 09)

• Memtest bootable CD (v3.2 or later)
• ESX v3.5 update 1 install CD (.\Software_Repository\ESX\esx-3.5.0_Update_1-82663.iso)
• Hardware monitoring agent
  • DELL OpenManage v5.4 install (.\Software_Repository\DELL\OM_5.4.0_ManNode_A01.tar.gz)
  • HP Insight Manager v8.1 install (.\Software_Repository\HP\hpmgmt-8.1.1-vmware3x.tar)

Info

• Management IP address, mask, and VLAN ID (if applicable)
• VMKernel IP address, mask
• Default gateway, and DNS server IP’s
• Fully qualified domain name (e.g. esxp1.company.com)

ESX Software Installation Procedure

Hardware Checks

1. Install any required cards, memory upgrades etc
2. Default BIOS options are fine, though of note;
   • Hyperthreading should be disabled if available
   • CPU Virtualisation Technology should be enabled (disabled for ESX's versions prior to v3.5, enabled by default on DELL's, disabled by default on HP's)
3. If a DELL with a pre-configured OS, boot the system up and ensure the system is as expected, especially;
   • Number of CPU's and speed
   • Memory size
   • Network cards
   • Physical storage (local hard-drives and SAN)
4. Reboot the system to Memtest, which should auto-start. Leave to run for at least one pass (eg overnight). Any errors must be resolved prior installation of ESX.
5. Configure local hard-drive RAID virtual disk (configuration not normally required on purchased for purpose DELL servers)
   • If not using SAN storage for VM's its recommended that you use RAID 10, requires 4 or more disks, ensures optimum performance
   • If using SAN storage, local disk pack is used for ESX software, should be at least RAID 1 if 2 disks, RAID 5 recommended if 3 disks.
   • Configure only 1 RAID pack, do not carve up physical storage into multiple logical drives

ESX Software Install

If you are (re)installing an ESX that's connected to an existing SAN partition you must disconnect the SAN cables prior to install. The ESX install will write new UUID's to the SAN partition making existing ESX's unable to access the SAN via the original UUID

1. Insert ESX install CD and reboot
2. If it’s the 1st usage of the CD, perform the media check (takes about 5 mins), otherwise Skip
3. Select GUI setup mode
4. Select United Kingdom keyboard
5. Select appropriate mouse
6. Acknowledge Yes to erase ALL DATA and create new partitions
7. Accept EULA
8. Check disk size is as expected. Select recommended partition config, and confirm erasing existing partitions
9. Edit suggested partitioning such that;
   • boot 100 MB -> 200 MB
   • swap 544 MB -> 1024 MB
   • log 1992 MB -> 2048 MB
   • delete VMFS partition
10. Leave bootloader config as is
11. Enter full network config for management interface;
    • IP and mask
    • Default gateway
    • DNS servers
    • Domain name - you must enter a fully qualified domain name (otherwise new VM deployments can fail)
    • If the management IP is on a trunked VLAN switchport you must enter the VLAN (802.1q) ID
    • Uncheck 'Create a default network for virtual machines'
12. In Location tab select Europe\London, select UTC in UTC Offset tab and in the Map tab leave ‘System clock uses UTC’ checked
13. Enter root password (syndrome can only be used for test hosts)
14. Review config (especially double check the partitioning and network config, it’s a pain to change these afterwards)
15. Install will complete, eject the CD, and await reboot
16. Logon as root and issue the following command:
    • esxcfg-firewall -o 53,tcp,out,dnsClientTCP

Create Local User Accounts

1. Disable password expiry
   • esxcfg-auth --passmaxdays=99999
2. Check the login.defs file has been updated
   • more /etc/login.defs
   • Confirm the following the exists: PASS_MAX_DAYS 99999
3. Create local user accounts for console access - NOTE that setting a password is optional and not required if an Active Directory account userid is going to be used.
   • useradd username
   • passwd username you'll then be asked for a password

Remaining tasks can be completed remotely

Post ESX Installation Server Config

Config NTP Sources

Sets up the system to use ntp-svr-a and ntp-svr-b, can be configured via the VI Client but doesn't seem to work reliably

1. Login to system, and use root privelidges (su - )
2. Enable NTP through service console firewall
   • esxcfg-firewall --enableService ntpClient
3. Backup ntp config file
   • cp /etc/ntp.conf /etc/ntp.conf.bak
4. Edit file using VI, so that in the appropriate sections…
   • # Prohibit general access to this service.
   • restrict default kod nomodify trap
   • # server mytrustedtimeserverip
   • server ntp-svr-a
   • server ntp-svr-b
5. Backup step tickers file
   • cp /etc/ntp/step-tickers /etc/ntp/step-tickers.bak
6. Edit file using VI, so that…
   • ntp-svr-a
   • ntp-svr-b
7. Restart NTP service
   • service ntpd restart
8. Service should restart successfully (see below), otherwise fallback config file and restart
   • Shutting down ntpd: [FAILED]
   • ntpd: Synchronizing with time server: [ OK ]
   • Starting ntpd: [ OK ]
9. Check system time / clock
   • Date
10. Configure service to start automatically
    • chkconfig ntpd on
11. Check service has been entered into run level config
    • chkconfig --list ntpd
    • ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Config SNMP Trap Destinations

Remember that Netcool needs to be made aware of the sending ESX server for traps to populate views.

1. Login to system, and use root privelidges (su - )
2. Enable SNMP through firewall
   • esxcfg-firewall --enableService snmpd
3. Backup SNMP config file
   • cp /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.bak
4. Edit file snmpd.conf using VI, so that is eactly as shown here:

#
syscontact Server Systems Team
#
# Create a Read only community called sstesx that is only accessible by authorised pollers.
#
rocommunity sstesx poller1
rocommunity sstesx poller2

#
# Send traps to both Netcool pollers with a community name of ssttrap. 
#
trapsink receiver1 ssttrap
trapsink receiver2 ssttrap
authtrapenable 1
#
# Enable VMware MIB modules.
# 
dlmod SNMPESX            /usr/lib/vmware/snmp/libSNMPESX.so
#
# Allow Dell's Systems Management Data Engine SNMP to connect to snmpd using SMUX
#
smuxpeer .1.3.6.1.4.1.674.10892.1
#

1. Start SNMP service
   • service snmpd restart
2. Service should start successfully (see below), otherwise fallback config file and restart
   • Starting snmpd: [ OK ]
3. Set SNMP agent to start automatically
   • chkconfig snmpd on
4. Check service has been entered into run level config
   • chkconfig --list snmpd
   • snmpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Install Centennial

1. Copy across the installation files to the ESX
   • EG pscp C:\Software_Repository\Unix_Centennial\* user@ESXserver:/home/user/centennial
2. On the ESX console become root, change into the directory, set execution permissions and run the installation procedure.
   • su -
   • cd /home/user/centennial (substitute user appropriately).
   • chmod 775 *
   • ./deploy_mpclient.sh redhat_i386
3. Confirm services are running...
   • service disco status

Enable Active Directory Authentication

1. Enter the following command. Note that 'enablead' is not a typo it being 'enable ad' and not 'enabled'.
   • esxcfg-auth --enablead --addomain=ukprod.company.com --addc=adserver.company.com

Install Hardware Monitoring Agents

Use the DELL or HP sections as appropriate!

Install DELL OpenManage

Previously done by CD, latest version is only available via tar file (currently).

1. Copy the tar file onto the ESX using PSCP
   • Generic command: pscp C:\Software_Repository\DELL\OM_5.4.0_ManNode_A01.tar.gz user@ESXserver:/home/user
2. On the ESX, change directory so you're in your home directory
   • cd /home/user
3. Uncompress the file
   • tar -zxvf OM_5.4.0_ManNode_A01.tar.gz
4. Change directory into the created folders
   • cd linux/supportscripts
5. Install OpenManage
   • ./srvadmin-install.sh –b –w –r –s
6. Enable OpenMange through the ESX server’s firewall
   • esxcfg-firewall -o 1311,tcp,in,OpenManageRequest
7. Start the OpenManage services
   • srvadmin-services.sh start , dsm_sa_datamgr32d can take a few mins to start
8. Confirm the OpenManage services are running
   • srvadmin-services.sh status
   • All processes should be in a running state
9. Log out (exit), and return to ESX screen (Alt + F11)

Check monitored by / Add to DELL IT Assistant

1. On ITA machine, run up IT Assistant, and login as administrator
2. Check to see if the ESX appears in the list of servers (this is refreshed in one hour intervals, so you might need to check back later)

To add an ESX

1. Go to Discovery and Monitoring | Ranges
2. Assuming the IP of the ESX isn't covered by one of the existing ranges, right-click over 'Include Ranges
3. Add ESX's IP address (or range of IP addresses if the first of a new cluster of ESX's)
4. Accept all default options, except step 7, select Perform both Discovery and Inventory

Install HP InSight Manager

1. Copy the tar file onto the ESX using PSCP
   • Generic command: pscp C:\Software_Repository\HP\hpmgmt-8.1.1-vmware3x.tar user@ESXserver:/home/user
2. On the ESX, change directory so you're in your home directory
   • cd /home/user
3. Uncompress the file
   • tar -zxvf hpmgmt-8.1.1-vmware3x.tar
4. Change directory into the created folders
   • cd hpmgmt/811
5. Install InSight
   • ./installvm811.sh --install
   • Accept all default options, including firewall ports

Install NIC and HBA monitor script

Note: Now using ESXmon_v3 script

1. Create a SST cron folder
   • mkdir /etc/cron.sst
2. Make contents of RTS cron folder runnable every minute, edit /etc/crontab to append the following line
   • * * * * * root run-parts /etc/cron.sst
3. Copy the monitor script, ESXmon_v3, into the ESX server from the VC in a similar fashion to #Install Centennial
   • pscp C:\Software_Repository\ESXmon_v2 user@ESXserver:/home/user/
4. Move the file into the cron folder
   • mv /home/user/ESXmon_v3 /etc/cron.sst/
5. Make the files executable
   • chmod +x /etc/cron.sst/*

Monitor script gets run every 1 min, it will send traps for all HBA's and NIC's when 1st run. It takes 2 mins for the changes to the crontab file to take effect, therefore (assuming Netcool knows about the ESX server) you should see some initial traps within 3 mins.

The script creates a number of hba* and nic* files, therefore if they exist the script is being run. If you delete these files, the script will have no record of the previous HBA and NIC states, and so will generate traps when next run as their states will change from Unknown to Up/Down.

A log file is also generated, ESXmon.log, this should log any HBA/NIC state changes, and all SNMP traps sent.

VMWare Configuration

Setup Management and Network Connectivity

1. Add to Virtual Centre server
   1. Highlight the cluster the host should be a member of
   2. Right-click and select Add Host...
      • Use a fully qualified domain name, and the root user/pass
2. Configure network connectivity (following steps may need to change if your config is non standard)
   1. Go to Networking (on the Configuration tab) and select Properties... of vSwitch0 (you may have to scroll the screen to the right).
   2. On the Ports tab Add... the required networks
      • VM Port Group names must be consistent across all ESX's in a cluster otherwise HA failover will not work
      • Use the table below as a reference for consistent Port Group names
   3. On the Network Adapters tab, add the unassigned adapter.

Subnet mask crib sheet;

• 16 - 255.255.0.0
• 21 - 255.255.248.0
• 24 - 255.255.255.0
• 25 - 255.255.255.128
• 26 - 255.255.255.192

License ESX server

1. Go to the Licensed Features option, and edit the ESX Server License Type
2. Check the ESX Server Standard option, the ESX will request required licenses from the Virtual Centre server

Configure ESX Firewall

Note that if you've issued command line esxcfg-firewall commands and not rebooted, the results of these commands may not show up via the GUI. If in doubt, give the ESX a bounce.

1. Go to the Security Profile and select Properties...
2. Tick/Check the following options (do not untick options not specified)
   • SNMP Server
   • NFS Client
   • NTP Client

Setup Storage

1. Set-up the local VMFS partition
   1. Go to the Storage option, and select Add Storage...
   2. Select Disk/LUN
   3. Select the local device
   4. Tick the Use free space option
   5. Give datastore same name as ESX
   6. Accept default partition formatting options
2. Set-up the NFS drive (shared storage on NFSserver for ISO's etc)
   1. Go to the Storage (SCSI, SAN, and NFS) option, and select Add Storage...
   2. Select Network File System and enter
      • Server: NFSserver
      • Folder: NFS_Share
      • Datastore Name: NFSserver

Troubleshooting / Known Issues

Can't add to Virtual Centre

Virtual Centre server unable to connect to ESX host...

1. Restart the ESX Managment agent

Test

Recorded procedure for installing / configuring, anything in this section has been attempted on one or more test ESX's, but isn't general installation.