Installation (ESX)

Build Notes

Security Hardening

Service Console

Disk Partitions

Suggesting partition sizing for Service Console on local disk to prevent Root partition being filled with user data

part /boot --fstype ext3 --size 1024 --ondisk=sda --asprimary
part / --fstype ext3 --size 5120 --ondisk=sda --asprimary
part swap --size 2048 --ondisk=sda --asprimary
part /var --fstype ext3 --size 5120 --ondisk=sda
part /tmp --fstype ext3 --size 5120 --ondisk=sda
part /home --fstype ext3 --size 2048 --ondisk=sda
part None --fstype vmkcore --size 100 --ondisk sda

Local Accounts

Password Policy

No policy is implemented by default, if not using AD Integration then its sensible to apply a policy on the ESX, using the PAMQC module. Its not particularly elegant.

Active Directory Integration

Because service console authentication is Unix-based, it cannot use Active Directory to define user accounts. However, it can use Active Directory to authenticate users by matching local passwd file account name with Active directory with appropriate support of SFU (Services For Unix).

See Scott Lowe's blog for further info

Sudo

It is possible to limit the enhanced privileges that a user can gain by using sudo. This is most appropriate where there is a large number admins. However, in such an environment there is likely to be a large number of ESX's, managing the config on ESX is a headache.

Example of possible sudo config (/etc/sudoers)

...
# Defaults specification
Defaults logfile=/var/log/sudolog

# User privilege specification
root    ALL=(ALL) ALL
User_Alias VI_JR_ADMINS=esxoper, esxoper2
User_Alias VI_ADMINS=esxadmin

Cmnd_Alias STOP=/usr/sbin/shutdown, /usr/sbin/halt, /usr/sbin/poweroff 
Cmnd_Alias REBOOT=/usr/sbin/reboot
Cmnd_Alias KILL=/usr/bin/kill 
Cmnd_Alias NTP=/usr/sbin/ntpdate, /sbin/hwclock 

VI_JR_ADMINS ALL=STOP, REBOOT, KILL, NTP
VI_ADMINS ALL=(ALL) ALL
...

Logging

It is recommended to compress and increase the maximum log file size by modifying the configuration files in the /etc/logrotate.d directory and the /etc/logrotate.conf file.

For example, changing vmkwarning to be 2096k in size, and compressed...

[root@dtcp-esxsvce01b root]# more /etc/logrotate.d/vmkwarning
/var/log/vmkwarning{
    create 0600 root root
    missingok
    compress
    sharedscripts
    postrotate
    size 2096k
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

...and changing relevent part of /etc/logrotate.conf to allow compression...

...
# uncomment this if you want your log files compressed
compress

...

Finally, its worth redirecting sudo log activity to /var/log/sudolog, see above section on sudo.

Banners

There are three modes of direct management access to an ESX, web, ssh, and direct (local) console.

Web Access

Edit the html page /usr/lib/vmware/hostd/docroot/index.html

SSH

Edit the /etc/ssh/sshd_config file so that it knows to display a defined banner file during login...

Banner /etc/banner

Create the banner file with the appropriate contents.

Console

Prepend your banner to the /etc/issue file

ESX

Network Settings

Setting	Default	Preferred	Explanantion
Promiscuous Mode	Reject	Reject	Principally used in situations where you need to perform a network traffic (snif) capture. Data from all ports propagates to all ports (VM Port group becomes a hub rather than a switch)
MAC address changes	Accept	Reject	There are situations where allowing MAC Address Changes to Accept is required. For example; legacy applications, clustered environments, and licensing. Legacy applications may require a specific MAC addresses to be used for the application. Microsoft Clusters utilize an artificial MAC address for all servers in the cluster
Forged Transmits	Accept	Reject	The setting affects traffic transmitted from a virtual machine. If this option is set to reject, the virtual switch compares the source MAC address being transmitted by the operating system with the effective MAC address for its virtual network adapter to see if they are the same. If the MAC addresses are different, the virtual switch drops the frame. The guest operating system will not detect that its virtual network adapter cannot send packets using the different MAC address. To protect against MAC address impersonation, all virtual switches should have forged transmissions set to reject

Procedures

Quick commands

vmware -v

ESX software version and build

ESX Shutdown

• Shutdown a host ready for power off
  • shutdown -h now
• Restart a host
  • shutdown -r now

High Availability Stop/Start

• Stop HA...
  • /etc/init.d/VMWAREAAM51_vmware stop
• Start HA...
  • /etc/init.d/VMWAREAAM51_vmware start

VMware Managment Agent Restart

service mgmt-vmware restart
Stopping VMware ESX Server Management services:
  VMware ESX Server Host Agent Services                   [  OK  ]
  VMware ESX Server Host Agent Watchdog                   [  OK  ]
  VMware ESX Server Host Agent                            [  OK  ]
Starting VMware ESX Server Management services:
  VMware ESX Server Host Agent (background)               [  OK  ]
  Availability report startup (background)                [  OK  ]

If this fails to stop the service, you can try to manually kill the processes. Untested - don't do during day, if you use this and it works, delete this warning note.

1. Determine the PID's of the processes
   • ps -auxwww | grep vmware-hostd
   • which should give you something like, in which case the PID's are 2807 and 2825...
   • root 2807 0.0 0.3 4244 884 ? S Mar10 0:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u
   • root 2825 0.1 12.0 72304 32328 ? S Mar10 1:14 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u
   • root 13848 0.0 0.2 3696 556 pts/0 R 08:43 0:00 grep vmware-hostd
2. Kill the PID's using kill -p pid
   • So, for example, kill -9 2807 and kill -9 2825
3. Then reattempt the service restart

VMware Web Access Restart

service vmware-webAccess restart
Stopping VMware ESX Server webAccess:
   VMware ESX Server webAccess                             [FAILED]
Starting VMware ESX Server webAccess:
   VMware ESX Server webAccess                             [  OK  ]

Maintenance Mode

To put the ESX into maintenance mode with no access from the Infrastructure Client (VCP) use the following commands - use with caution

Put esx into maintenance mode:

vimsh -n -e /hostsvc/maintenance_mode_enter

check the esx is in maintenance mode

vimsh -n -e /hostsvc/runtimeinfo | grep inMaintenanceMode | awk ‘{print $3}’

exit maintenance mode

vimsh -n -e /hostsvc/maintenance_mode_exit

TCPDump Network Sniffer

Basic network sniffer available in Service Console

TCPDump instruction manual

EG To sniff all traffic on the Service Console interface, vswif0, going to/from 159.104.227.40

tcpdump -i vswif0 host 159.104.224.70

Security

Password Complexity Override

In order to be able to change a user (or root) password to one that breaches password complexity checking

1. Disable PAM module
   • esxcfg-auth --usepamqc -1 -1 -1 -1 -1 -1
2. Disable complexity checker
   • esxcfg-auth --usecrack -1 -1 -1 -1 -1 -1
3. Change password
4. Re-enable PAM module
   • esxcfg-auth --usepamqc=-1 -1 -1 -1 8 8

Regenerate Certificate

You might need to regenerate certificates if

• Change ESX host name
• Accidentally delete the certificates

To generate new Certificates for the ESX Server host...

1. Change directories to /etc/vmware/ssl.
2. Create backups of any existing certificates:
   • mv rui.crt orig.rui.crt
   • mv rui.key orig.rui.key
3. Rstart the vmware-hostd process:
   • service mgmt-vmware restart
4. Confirm that the ESX Server host generated new certificates by executing the following command comparing the time stamps of the new certificate files with orig.rui.crt and orig.rui.key
   • ls -la

HBA and SAN Operations

SAN LUN ID

The SAN LUN ID is used by SAN admin's to identify LUN's. It's not readily available from the GUI and has to be extracted from the vml file...

So from the following...

• /vmfs/devices/disks/vml.020006000060060160c6931100cc319eea7adddd11524149442035

you need to extract the mid characters from the vml name...

• /vmfs/devices/disks/vml.020006000060060160c6931100cc319eea7adddd11524149442035

So the SAN LUN ID is 60060160c6931100cc319eea7adddd11

Emulex

Find Emulex HBA Driver and Firmware Version, and WWPN

Doesn't require Emulex HBA utility to be installed

1. cd /proc/scsi/lpfc
2. more 1 for HBA 1
3. more 2 for HBA 2

The Portname number is the WWPN number used to identify the HBA's by the SAN.

[root@uklonesxp2 lpfc]# more 1
Emulex LightPulse FC SCSI 7.1.14_vmw1
Emulex LightPulse LP1050 2 Gigabit PCI Fibre Channel Adapter on PCI bus 0f devic
e 20 irq 121
SerialNum: BG70569148
Firmware Version: 1.91A1 (M2F1.91A1)
Hdw: 1001206d
VendorId: 0xf0a510df
Portname: 10:00:00:00:c9:61:73:de   Nodename: 20:00:00:00:c9:61:73:de

Link Up - Ready:
   PortID 0x645213
   Fabric
   Current speed 2G

Install Emulex HBA Utility

Can be found at Emulex Lputil.

To install lputil (uses example of lpfcutil-7.1.14;

1. Put the downloaded tgz file on the ESX server
   • EG mkdir /var/updates/Emulex-lpfcutil-7.1.14
2. Go into folder and extract;
   • cd /var/updates/Emulex-lpfcutil-7.1.14/
   • tar -xvzf Emulex-lpfcutil-7.1.14.tgz
3. Install;
   • ./Install.sh

[root@uklonesxp2 Emulex-lpfcutil-7.1.14]# ./Install.sh
Installing Emulex HBAAPI libraries and applications...
Installation of Emulex HBAAPI libraries and utilities is completed.

• Start the utility (on startup it should detect one or more HBA's);
• /usr/sbin/lpfc/lputil

LightPulse Common Utility for Linux. Version 1.6a10 (10/7/2004).
Copyright (c) 2004, Emulex Network Systems, Inc.

Emulex Fibre Channel Host Adapters Detected: 1
Host Adapter 0 (lpfc0) is an LP1050 (Ready Mode)

HBAnywhere Installation

1. Download the Driver and Application kit for VMware from Emulex's website.
   • At time of writing the current version of package was elxvmwarecorekit-esx35-4.0a45-1.i386.rpm
2. Copy the package to the server
   • EG pscp -pw [password] elxvmwarecorekit-esx35-4.0a45-1.i386.rpm platadmn@dtcp-esxsvce01a:/home/platadmn
3. Install the package
   • EG rpm -ivh elxvmwarecorekit-2.1a42-1.i386.rpm

Check Emulex HBA Firmware Version

Requires the HBA Utility to be installed 1st (see above)

1. Start the utility (on startup it should detect one or more HBA's;
   • /usr/sbin/lpfc/lputil
2. From the Main menu, enter 2, Adapter Revision Levels
   • Example shows version 1.91a5

                   BIU: 1001206D
      Sequence Manager: 00000000
                 Endec: 00000000
  Operational Firmware: SLI-2 Overlay
                Kernel: 1.40a3
      Initial Firmware: Initial Load 1.91a5 (MS1.91A5 )
                 SLI-1: SLI-1 Overlay 1.91a5 (M1F1.91A5 )
                 SLI-2: SLI-2 Overlay 1.91a5 (M2F1.91A5 )
 Highest FC-PH Version: 4.3
  Lowest FC-PH Version: 4.3

Update Emulex HBA Firmware

• Using HBA Utility (must be installed 1st - see above). See the Emulex website for the latest version, eg Emulex LP1050Ex

To update the firmware (example uses LP1050Ex-mf191a5)

1. Put the downloaded zip file on the UKLONVCP1 NFS Share, and unzip to a folder, eg EmulexLP1050Ex-mf191a5
2. Create folder in /var/updates;
   • mkdir /var/updates/EmulexLP1050Ex-mf191a5
3. Copy the firmware update onto the ESX
   • cp /vmfs/volumes/UKLONVCP1\ NFS\ Share/EmulexLP1050Ex-mf191a5/mf191a5.all /var/updates/EmulexLP1050Ex-mf191a5/
4. Start the utility (on startup it should detect one or more HBA's;
   • /usr/sbin/lpfc/lputil
5. From the Main menu, enter 3, Firmware Maintenance.
6. If prompted, choose the HBA that is being updated.
7. Enter 1, Load Firmware Image.
8. Enter the full path to the firmware file, upgrade will then complete, eg

Enter Image Filename => /var/updates/EmulexLP1050Ex-mf191a5/mf191a5.all
Opening File...
End Of File
Checksum OK!!!
Reading AIF Header #1...
Validating Checksum...
Erasing Flash ROM Sectors...
100% complete
Loading Image...
First Download
100% complete
Image Successfully Downloaded...
Reading AIF Header #2...
Validating Checksum...
Erasing Flash ROM Sectors...
100% complete
Loading Image...
First Download
100% complete
Updating Wakeup Parameters...
Image Successfully Downloaded...
Reading AIF Header #3...
End Of File
Resetting Host Adapter...
Image Successfully Downloaded...

• Using HBAnywhere (must be installed 1st - see above)

1. Download the correct firmware version from Emulex's website
   • EG for LPe11002's
2. Extract, and copy file to server
3. Find adapter's WWPN's
   • EG /usr/sbin/hbanyware/hbacmd ListHBAs
4. Download new firware version to each HBA
   • EG /usr/sbin/hbanyware/hbacmd download 10:00:00:00:c9:82:97:9e zf280a4.all

EMCgrab Collection

1. Download correct verion from EMC's website
   • At time of writing the current version file was emcgrab_ESX_v1.1.tar
2. Copy to server
   • EG pscp emcgrab_ESX_v1.1.tar platadmn@dtcp-esxsvce02a:/home/platadmn
3. Uncompress the file
   • EG tar -xvf emcgrab_ESX_v1.1.tar
4. Run grab (can take a few minutes, best done out of hours)
   • EG ./emcgrab.sh
5. Results can be found in \emcgrab\outputs folder

Netflow

Netflow is available on ESX v3 only, and is an experimental feature. Netflow v5 is sent.

• To start Netflow
  1. Load the module
     • vmkload_mod netflow
  2. Configure monitoring of appropriate vSwitch's to Netflow collector IP and port
     • /usr/lib/vmware/bin/vmkload_app -S -i vmktcp /usr/lib/vmware/bin/net-netflow -e vSwitch0,vSwitch1 10.20.255.31:2055
  • To reconfigure the Netflow module you must stop and restart the module

• To confirm running
  1. Check the module is running...
     • [root@esx1 root]# vmkload_mod -l | grep netflow
     • netflow 0x9b4000 0x3000 0x298b640 0x1000 16 Yes
  2. Check the correct config is running...
     • [root@esx1 root]# ps -ef | grep netflow
     • root 2413 1 0 Feb05 ? 00:00:00 /usr/lib/vmware/bin/vmkload_app -S -i vmktcp /usr/lib/vmware/bin/net-netflow -e vSwitch0,vSwitch1 10.20.255.31:2055

• To stop Netflow
  1. ps -ef | grep netflow
  2. kill <pid>
  3. vmkload_mod -u netflow

Troubleshooting

Useful paths / logfiles

Item	Path	Comments
Vmkernel logfile	/var/log/vmkernel	Pretty much everything seems to be recorded here
Vmkernel warnings	/var/log/vmkwarning	Virtual machine warnings
Local VM files	/vmfs/volumes/storage	Storage name can vary, use TAB so shell selects available
SAN VM files	/vmfs/volumes/SAN
HA agent logs	/opt/LGTOaam512/log/	Various logs of limited use

CPU

Poor performance

If VM's are performing sluggishly and/or are slow to start, use esxtop on the ESX service console. Look at Ready Time (%RDY), which is how long a VM is waiting for CPUs to become available. This can creep up if the the system is pushed, or if the VM has multiple CPUs (as it needs multiple physical CPUs to become available at the same time).

Ideally %RDY should <5%, though <10% is normally acceptable, anything >15% is bad.

Storage

Poor throughput

Use esxtop on the service console and switch to the disk monitor. Enable views for latency, you will see values like GAVG, KAVG and DAVG.

• GAVG is the total latency on IO commands averaged over 2 seconds
• KAVG is the hypervisor IO latency averaged over 2 seconds
• DAVG is everything outside the ESX server IO latency averaged over the last 2 seconds

Latency occurs when the hypervisor or physical storage cannot keep pace with the demand for IO

Storage Monitor Log Entries

How to decode the following type of entries...

Sep  3 15:15:14 tfukesxent1 vmkernel: 85:01:23:01.532 cpu4:2264)StorageMonitor: 196: vmhba1:2:0:0 status = 2/0 0x6 0x2a 0x1
Sep  3 15:15:32 tfukesxent1 vmkernel: 85:01:23:19.391 cpu4:2253)StorageMonitor: 196: vmhba1:3:9:0 status = 2/0 0x6 0x2a 0x1

The status message consists of the follow four decimal and hex blocks...

Device Status / Host Status

Sense Key

Additional Sense Code

Additional Sense Code Qualifier

Where the ESX Device and SAN host status' mean...

Decimal	Device Status	Host Status	Comments
0	No Errors	Host_OK
1		Host No_Connect
2	Check Condition	Host_Busy_Busy
3		Host_Timeout
4		Host_Bad_Target
5		Host_Abort
6		Host_Parity
7		Host_Error
8	Device Busy	Host_Reset
9		Host_Bad_INTR
10		Host_PassThrough
11		Host_Soft_Error
24	Reservation Conflict		24/0 indicates a locking error, normally caused by too many ESX's mounting a LON, wrong config on storage array, or too many VM's on a LUN

Where the Sense Key mean...

Hex	Sense Key
0x0	No Sense Information
0x1	Last command completed but used error correction
0x2	Unit Not Ready
0x3	Medium Error
0x4	Hardware Error
0x5	ILLEGAL_REQUEST (Passive SP)
0x6	LUN Reset
0x7	Data_Protect - Access to data is blocked
0x8	Blank_Check - Reached an unexpected region
0xa	Copy_Aborted
0xb	Aborted_Command - Target aborted command
0xc	Comparison for SEARCH DATA unsuccessful
0xd	Volume_Overflow - Medium is full
0xe	Source and Data on Medium do not agree

The Additional Sense Code and Additional Sense Code Qualifier mean

Hex	Sense Code
0x4	Unit Not Ready
0x3	Unit Not Ready - Manual Intervention Required
0x2	Unit Not Ready - Initializing Command Required
0x29	Device Power on or SCSI Reset