Difference between revisions of "Installation (ESX)"

Jump to navigation Jump to search

Installation (ESX) (view source)

Revision as of 11:14, 24 September 2012

12,462 bytes removed ,  11:14, 24 September 2012
m
→‎Build Numbers: Added v5.1 build
(→‎HBA and SAN Operations: Added "Emulex")
m (→‎Build Numbers: Added v5.1 build)
 
(69 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{TOC limit|3}}
== Build Notes ==
* '''[[ESX3 Installation]]''' - Example, based on an old ESX v3 build guide
* '''[[ESX4i Installation]]''' - Example, bit brief in places
* [http://www.jam-software.com/heavyload/download.shtml HeavyLoad] - Load tester (stick it in a test VM, memory test doesn't really work as ESX page sharing kicks in)
 
== Build Numbers ==
ESX build numbers, note that installing subsequent patches, on top of one of the major releases below will increase the build number.
{|class="vwikitable"
|-
! ESX version  !! ESX    !! ESXi
|-
| 3.5 Update 1  || 82663  ||  82664
|-
| 3.5 Update 2 || 110268 || 110271
|-
| 3.5 Update 3 || 123630 || 123629
|-
| 3.5 Update 4 ||colspan="2"| 153875
|-
| 3.5 Update 5 ||colspan="2"| 207095
|-
| 4.0  ||colspan="2"| 164009
|-
| 4.0 Update 1 ||colspan="2"| 208167
|-
| 4.0 Update 2 ||colspan="2"| 261974
|-
| 4.0 Update 3 ||colspan="2"| 398348
|-
| 4.0 Update 4 ||colspan="2"| 504850
|-
| 4.1 ||colspan="2"| 260247
|-
| 4.1 Update 1  ||colspan="2"| 348481
|-
| 4.1 Update 2  ||colspan="2"| 502767
|-
| 4.1 Update 3 ||colspan="2"| 800380
|-
| 5.0 ||colspan="2"| 469512
|-
| 5.0 Update 1  ||colspan="2"| 623860
|-
| 5.1 ||colspan="2"| 799733
|}
 
== USB Image ==
If you're installing ESXi 4 then you don't need to do this, the installer will detect the USB stick and install to it.
 
'''Required software etc...'''
* '''''WinImage''''' - http://www.winimage.com/download.htm
* '''''DD''''' - http://www.chrysocome.net/dd
* '''''ESXi install ISO'''''
* '''''Disk Cloner''''', eg G4U - http://www.feyrer.de/g4u/
** Ideally use a cloner that ignores the actual disk contents and does a block by block copy, anything that tries to interpret the disk image may not copy it faithfully
* You must be able to connect '''two''' image files remotely to your server, a disk cloner ''CD'' ISO, and the image ''USB'' ISO (hint: use the floppy drive).
 
'''Creating the USB image file'''
# Open up the ISO with WinImage
# Extract the <code> INSTALL.TGZ </code> from the ISO
# Uncompress <code> INSTALL.TGZ </code> and locate <code> .\INSTALL\usr\lib\vmware\installer\VMware-VMvisor-big-3.5.0_Update_4-153875.i386.dd.bz2 </code>
# Uncompress <code> VMware-VMvisor-big-3.5.0_Update_4-153875.i386.dd.bz2 </code> so that you have <code> VMware-VMvisor-big-3.5.0_Update_4-153875.i386.dd </code>
# Create ISO image from DD image by using DD
#* <code> dd bs=1M if=VMware-VMvisor-big-3.5.0_Update_4-153875.i386.dd of=esx3.5ihp-usbimage.iso </code>
 
'''Deploying the USB image file'''
# Attach your disk cloner image to your server and boot
# Once the  the server is booting to the CD ISO, attach the USB ISO
# List the avaialble disks
#* <code> list </code>
# Identify the image disk (which is 750MB) and the USB disk (which will be whatever size your USB key is)
# Copy the image to the USB key
#* <code> copydisk sd1 sd0 </code>
# Disconnect all images, reboot server, cross fingers
#* <code> reboot </code>
 
== VMware CLI ==
Especially if using ESX'''i''', you'll need to install the VMware CLI on any machine you want to access the ESX command line from.  Be aware that [http://www.activestate.com/activeperl/ ActivePerl] gets installed as well, so proceed with caution if you've already got Perl installed on the machine.


= Build Notes =
== Security Hardening ==
== Security Hardening ==
=== Service Console ===
=== Service Console ===
Applicable to ESX only (not ESXi, as ESXi doesn't have a service console)
==== Disk Partitions ====
==== Disk Partitions ====
Suggesting partition sizing for Service Console on local disk to prevent Root partition being filled with user data
Suggesting partition sizing for Service Console on local disk to prevent Root partition being filled with user data
Line 89: Line 166:
==== Network Settings ====
==== Network Settings ====


{|cellpadding="2" cellspacing="0" border="1"
{|class="vwikitable"
|-
|-
! Setting                  !! Default !! Preferred !! Explanantion
! Setting                  !! Default !! Preferred !! Explanantion
Line 103: Line 180:
|}
|}


 
[[Category:ESX]]
= Procedures =
== Quick commands ==
{|cellpadding="2" cellspacing="0" border="1"
|-
|<code> vmware -v </code>                    || ESX software version and build
|}
 
=== ESX Shutdown ===
* Shutdown a host ready for power off
** <code> shutdown -h now </code>
* Restart a host
** <code> shutdown -r now </code>
 
=== High Availability Stop/Start ===
* Stop HA...
** <code>  /etc/init.d/VMWAREAAM51_vmware stop  </code>
* Start HA...
** <code>  /etc/init.d/VMWAREAAM51_vmware start  </code>
 
== VMware Managment Agent Restart ==
<pre>
service mgmt-vmware restart
Stopping VMware ESX Server Management services:
  VMware ESX Server Host Agent Services                  [ OK  ]
  VMware ESX Server Host Agent Watchdog                  [ OK  ]
  VMware ESX Server Host Agent                            [  OK  ]
Starting VMware ESX Server Management services:
  VMware ESX Server Host Agent (background)              [  OK  ]
  Availability report startup (background)                [  OK  ]
</pre>
 
If this fails to stop the service, you can try to manually kill the processes. '''Untested - don't do during day, if you use this and it works, delete this warning note.'''
 
# Determine the PID's of the processes
#* <code> ps -auxwww | grep vmware-hostd </code>
#* which should give you something like, in which case the PID's are 2807 and 2825...
#* <code> root      2807  0.0  0.3  4244  884 ?        S    Mar10  0:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u </code>
#* <code> root      2825  0.1 12.0 72304 32328 ?      S    Mar10  1:14 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u </code>
#* <code> root    13848  0.0  0.2  3696  556 pts/0    R    08:43  0:00 grep vmware-hostd </code>
# Kill the PID's using <code> kill -p pid </code>
#* So, for example, <code> kill -9 2807 </code> and <code> kill -9 2825 </code>
# Then reattempt the service restart
 
== VMware Web Access Restart ==
<pre>
service vmware-webAccess restart
Stopping VMware ESX Server webAccess:
  VMware ESX Server webAccess                            [FAILED]
Starting VMware ESX Server webAccess:
  VMware ESX Server webAccess                            [  OK  ]
</pre>
 
 
== Maintenance Mode ==
To put the ESX into maintenance mode with no access from the Infrastructure Client (VCP) use the following commands - use with caution
 
Put esx into maintenance mode:
<pre>
vimsh -n -e /hostsvc/maintenance_mode_enter
</pre>
 
check the esx is in maintenance mode
<pre>
vimsh -n -e /hostsvc/runtimeinfo | grep inMaintenanceMode | awk ‘{print $3}’
</pre>
 
exit maintenance mode
<pre>
vimsh -n -e /hostsvc/maintenance_mode_exit
</pre>
 
 
== TCPDump Network Sniffer ==
Basic network sniffer available in Service Console
 
[http://www.tcpdump.org/tcpdump_man.html TCPDump instruction manual]
 
EG To sniff all traffic on the Service Console interface, vswif0, going to/from 159.104.227.40
 
<code> tcpdump -i vswif0 host 159.104.224.70 </code>
 
 
== Security ==
=== Password Complexity Override ===
In order to be able to change a user (or root) password to one that breaches password complexity checking
 
# Disable PAM module
#* <code> esxcfg-auth --usepamqc -1 -1 -1 -1 -1 -1 </code>
# Disable complexity checker
#* <code> esxcfg-auth --usecrack -1 -1 -1 -1 -1 -1 </code>
# Change password
# Re-enable PAM module
#* <code> esxcfg-auth --usepamqc=-1 -1 -1 -1  8 8 </code>
 
=== Regenerate Certificate ===
You might need to regenerate certificates if
* Change ESX host name
* Accidentally delete the certificates
 
To generate new Certificates for the ESX Server host...
# Change directories to /etc/vmware/ssl.
# Create backups of any existing certificates:
#* <code> mv rui.crt orig.rui.crt </code>
#* <code> mv rui.key orig.rui.key </code>
# Rstart the vmware-hostd process:
#* <code> service mgmt-vmware restart </code>
# Confirm that the ESX Server host generated new certificates by executing the following command comparing the time stamps of the new certificate files with orig.rui.crt and orig.rui.key
#* <code> ls -la </code>
 
 
== HBA and SAN Operations ==
=== SAN LUN ID ===
The SAN LUN ID is used by SAN admin's to identify LUN's.  It's not readily available from the GUI and has to be extracted from the vml file...
 
So from the following...
* <code> /vmfs/devices/disks/vml.020006000060060160c6931100cc319eea7adddd11524149442035 </code>
you need to extract the mid characters from the vml name...
* <code> /vmfs/devices/disks/vml.0200060000'''60060160c6931100cc319eea7adddd11'''524149442035 </code>
So the SAN LUN ID is <code> 60060160c6931100cc319eea7adddd11 </code>
 
=== Emulex ===
==== Find Emulex HBA Driver and Firmware Version, and WWPN ====
Doesn't require Emulex HBA utility to be installed
# <code> cd /proc/scsi/lpfc </code>
# <code>  more 1 </code> for HBA 1
# <code>  more 2 </code> for HBA 2
 
The <code> Portname </code> number is the WWPN number used to identify the HBA's by the SAN.
<pre>
[root@uklonesxp2 lpfc]# more 1
Emulex LightPulse FC SCSI 7.1.14_vmw1
Emulex LightPulse LP1050 2 Gigabit PCI Fibre Channel Adapter on PCI bus 0f devic
e 20 irq 121
SerialNum: BG70569148
Firmware Version: 1.91A1 (M2F1.91A1)
Hdw: 1001206d
VendorId: 0xf0a510df
Portname: 10:00:00:00:c9:61:73:de  Nodename: 20:00:00:00:c9:61:73:de
 
Link Up - Ready:
  PortID 0x645213
  Fabric
  Current speed 2G
</pre>
 
==== Install Emulex HBA Utility ====
Can be found at [http://www.emulex.com/vmware/support/index.jsp Emulex Lputil].
 
To install lputil (uses example of lpfcutil-7.1.14;
# Put the downloaded tgz file on the ESX server
#* EG <code> mkdir /var/updates/Emulex-lpfcutil-7.1.14 </code>
# Go into folder and extract;
#* <code> cd /var/updates/Emulex-lpfcutil-7.1.14/ </code>
#* <code> tar -xvzf Emulex-lpfcutil-7.1.14.tgz </code>
# Install;
#* <code> ./Install.sh </code>
<pre>
[root@uklonesxp2 Emulex-lpfcutil-7.1.14]# ./Install.sh
Installing Emulex HBAAPI libraries and applications...
Installation of Emulex HBAAPI libraries and utilities is completed.
</pre>
* Start the utility (on startup it should detect one or more HBA's);
* <code> /usr/sbin/lpfc/lputil </code>
<pre>
LightPulse Common Utility for Linux. Version 1.6a10 (10/7/2004).
Copyright (c) 2004, Emulex Network Systems, Inc.
 
Emulex Fibre Channel Host Adapters Detected: 1
Host Adapter 0 (lpfc0) is an LP1050 (Ready Mode)
</pre>
 
==== HBAnywhere Installation ====
# Download the Driver and Application kit for VMware from [http://www.emulex.com/downloads/emulex/cnas-and-hbas/drivers/vmware/fc-74040-pkg.html Emulex's website].
#* At time of writing the current version of package was <code>elxvmwarecorekit-esx35-4.0a45-1.i386.rpm</code>
# Copy the package to the server
#* EG <code> pscp -pw [password] elxvmwarecorekit-esx35-4.0a45-1.i386.rpm platadmn@dtcp-esxsvce01a:/home/platadmn</code>
# Install the package
#* EG <code> rpm -ivh elxvmwarecorekit-2.1a42-1.i386.rpm </code>
 
==== Check Emulex HBA Firmware Version ====
Requires the HBA Utility to be installed 1st (see above)
 
# Start the utility (on startup it should detect one or more HBA's;
#* <code> /usr/sbin/lpfc/lputil </code>
# From the Main menu, enter 2, '''Adapter Revision Levels'''
#* Example shows version 1.91a5
<pre>
                  BIU: 1001206D
      Sequence Manager: 00000000
                Endec: 00000000
  Operational Firmware: SLI-2 Overlay
                Kernel: 1.40a3
      Initial Firmware: Initial Load 1.91a5 (MS1.91A5 )
                SLI-1: SLI-1 Overlay 1.91a5 (M1F1.91A5 )
                SLI-2: SLI-2 Overlay 1.91a5 (M2F1.91A5 )
Highest FC-PH Version: 4.3
  Lowest FC-PH Version: 4.3
</pre>
 
 
==== Update Emulex HBA Firmware ====
* '''Using HBA Utility''' (must be installed 1st - see above).  See the Emulex website for the latest version, eg [http://www.emulex.com/ts/downloads/lp1050/lp1050ex.jsp Emulex LP1050Ex]
 
To update the firmware (example uses LP1050Ex-mf191a5)
# Put the downloaded zip file on the UKLONVCP1 NFS Share, and unzip to a folder, eg EmulexLP1050Ex-mf191a5
# Create folder in /var/updates;
#* <code> mkdir /var/updates/EmulexLP1050Ex-mf191a5 </code>
# Copy the firmware update onto the ESX
#* <code> cp /vmfs/volumes/UKLONVCP1\ NFS\ Share/EmulexLP1050Ex-mf191a5/mf191a5.all /var/updates/EmulexLP1050Ex-mf191a5/ </code>
# Start the utility (on startup it should detect one or more HBA's;
#* <code> /usr/sbin/lpfc/lputil </code>
# From the Main menu, enter 3, '''Firmware Maintenance'''.
# If prompted, choose the HBA that is being updated.
# Enter 1, '''Load Firmware Image'''.
# Enter the full path to the firmware file, upgrade will then complete, eg
<pre>
Enter Image Filename => /var/updates/EmulexLP1050Ex-mf191a5/mf191a5.all
Opening File...
End Of File
Checksum OK!!!
Reading AIF Header #1...
Validating Checksum...
Erasing Flash ROM Sectors...
100% complete
Loading Image...
First Download
100% complete
Image Successfully Downloaded...
Reading AIF Header #2...
Validating Checksum...
Erasing Flash ROM Sectors...
100% complete
Loading Image...
First Download
100% complete
Updating Wakeup Parameters...
Image Successfully Downloaded...
Reading AIF Header #3...
End Of File
Resetting Host Adapter...
Image Successfully Downloaded...
</pre>
 
 
* '''Using HBAnywhere''' (must be installed 1st - see above)
# Download the correct firmware version from Emulex's website
#* EG for [http://www.emulex.com/downloads/emulex/cnas-and-hbas/firmware-and-boot-code/lpe11002.html LPe11002's]
# Extract, and copy file to server
# Find adapter's WWPN's
#* EG <code>/usr/sbin/hbanyware/hbacmd ListHBAs</code>
# Download new firware version to each HBA
#* EG <code>/usr/sbin/hbanyware/hbacmd download 10:00:00:00:c9:82:97:9e zf280a4.all</code>
 
 
==== EMCgrab Collection ====
# Download correct verion from EMC's website
#* At time of writing the current version file was [ftp://ftp.emc.com/pub/emcgrab/ESX/Old_Releases/v1.1/ emcgrab_ESX_v1.1.tar]
# Copy to server
#* EG <code>pscp emcgrab_ESX_v1.1.tar platadmn@dtcp-esxsvce02a:/home/platadmn</code>
# Uncompress the file
#* EG <code>tar -xvf emcgrab_ESX_v1.1.tar</code>
# Run grab (can take a few minutes, best done out of hours)
#* EG <code>./emcgrab.sh</code>
# Results can be found in <code>\emcgrab\outputs</code> folder
 
== Netflow ==
'''Netflow is available on ESX v3 only, and is an experimental feature.  Netflow v5 is sent.'''
 
* '''To start Netflow'''
*# Load the module
*#* <code> vmkload_mod netflow </code>
*# Configure monitoring of appropriate vSwitch's to Netflow collector IP and port
*#* <code> /usr/lib/vmware/bin/vmkload_app -S -i vmktcp /usr/lib/vmware/bin/net-netflow -e vSwitch0,vSwitch1 10.20.255.31:2055 </code>
** To reconfigure the Netflow module you must stop and restart the module
 
* '''To confirm running'''
*# Check the module is running...
*#* <code> [root@esx1 root]# vmkload_mod -l | grep netflow </code>
*#* <code> netflow            0x9b4000          0x3000      0x298b640        0x1000        16 Yes </code>
*# Check the correct config is running...
*#* <code> [root@esx1 root]# ps -ef | grep netflow </code>
*#* <code> root      2413    1  0 Feb05 ?        00:00:00 /usr/lib/vmware/bin/vmkload_app -S -i vmktcp /usr/lib/vmware/bin/net-netflow -e vSwitch0,vSwitch1 10.20.255.31:2055 </code>
 
* '''To stop Netflow'''
*# <code> ps -ef | grep netflow </code>
*# <code> kill <pid> </code>
*# <code> vmkload_mod -u netflow </code>
 
= Troubleshooting =
== Useful paths / logfiles==
{|cellpadding="2" cellspacing="0" border="1"
|-
! Item              !!  Path                                !!  Comments
|-
| Vmkernel logfile  || <code> /var/log/vmkernel  </code>    ||  Pretty much everything seems to be recorded here
|-
| Vmkernel warnings  || <code> /var/log/vmkwarning  </code>  ||  Virtual machine warnings
|-
| Local VM files    || <code> /vmfs/volumes/storage </code> ||  Storage name can vary, use TAB so shell selects available
|-
| SAN VM files      || <code> /vmfs/volumes/SAN </code>    ||
|-
| HA agent logs      || <code> /opt/LGTOaam512/log/ </code>  ||  Various logs of limited use
|-
|}
 
== CPU ==
=== Poor performance ===
If VM's are performing sluggishly and/or are slow to start, use <code>esxtop</code> on the ESX service console.  Look at Ready Time (%RDY), which is how long a VM is waiting for CPUs to become available.  This can creep up if the the system is pushed, or if the VM has multiple CPUs (as it needs multiple physical CPUs to become available at the same time).
 
Ideally %RDY should <5%, though <10% is normally acceptable, anything >15% is bad.
 
== Storage ==
=== Poor throughput ===
Use <code>esxtop</code> on the service console and switch to the disk monitor.  Enable views for latency, you will see values like GAVG, KAVG and DAVG.
* '''GAVG''' is the total latency on IO commands averaged over 2 seconds
* '''KAVG''' is the hypervisor IO latency averaged over 2 seconds
* '''DAVG''' is everything outside the ESX server IO latency averaged over the last 2 seconds
 
Latency occurs when the hypervisor or physical storage cannot keep pace with the demand for IO
 
 
=== Storage Monitor Log Entries ===
 
How to decode the following type of entries...
Sep  3 15:15:14 tfukesxent1 vmkernel: 85:01:23:01.532 cpu4:2264)StorageMonitor: 196: vmhba1:2:0:0 status = 2/0 0x6 0x2a 0x1
Sep  3 15:15:32 tfukesxent1 vmkernel: 85:01:23:19.391 cpu4:2253)StorageMonitor: 196: vmhba1:3:9:0 status = 2/0 0x6 0x2a 0x1
 
The status message consists of the follow four decimal and hex blocks...
{| cellpadding="4" cellspacing="0" border="1"
|-
|''Device Status'' / ''Host Status'' || ''Sense Key'' || ''Additional Sense Code'' || ''Additional Sense Code Qualifier''
|}
 
Where the ESX Device and SAN host status' mean...
{| cellpadding="4" cellspacing="0" border="1"
|-
! Decimal !! Device Status        !! Host Status      !! Comments
|-
| 0      || No Errors            || Host_OK          ||
|-
| 1      ||                      || Host No_Connect  ||
|-
| 2      || Check Condition      || Host_Busy_Busy  ||
|-
| 3      ||                      || Host_Timeout    ||
|-
| 4      ||                      || Host_Bad_Target  ||
|-
| 5      ||                      || Host_Abort      ||
|-
| 6      ||                      || Host_Parity      ||
|-
| 7      ||                      || Host_Error      ||
|-
| 8      || Device Busy          || Host_Reset      ||
|-
| 9      ||                      || Host_Bad_INTR    ||
|-
| 10      ||                      || Host_PassThrough ||
|-
| 11      ||                      || Host_Soft_Error  ||
|-
| 24      || Reservation Conflict ||                  ||  24/0 indicates a locking error, normally caused by too many ESX's mounting a LON, wrong config on storage array, or too many VM's on a LUN
|}
 
Where the Sense Key mean...
{| cellpadding="4" cellspacing="0" border="1"
|-
! Hex !! Sense Key
|-
| 0x0 || No Sense Information
|-
| 0x1 || Last command completed but used error correction
|-
| 0x2 || Unit Not Ready
|-
| 0x3 || Medium Error
|-
| 0x4 || Hardware Error
|-
| 0x5 || ILLEGAL_REQUEST (Passive SP)
|-
| 0x6 || LUN Reset
|-
| 0x7 || Data_Protect - Access to data is blocked
|-
| 0x8 || Blank_Check - Reached an unexpected region
|-
| 0xa || Copy_Aborted
|-
| 0xb || Aborted_Command - Target aborted command
|-
| 0xc || Comparison for SEARCH DATA unsuccessful
|-
| 0xd || Volume_Overflow - Medium is full
|-
| 0xe || Source and Data on Medium do not agree
|}
 
The Additional Sense Code and Additional Sense Code Qualifier mean
{| cellpadding="4" cellspacing="0" border="1"
|-
! Hex !! Sense Code
|-
| 0x4 || Unit Not Ready
|-
| 0x3 || Unit Not Ready - Manual Intervention Required
|-
| 0x2 || Unit Not Ready - Initializing Command Required
|-
| 0x29 || Device Power on or SCSI Reset
|}
 
[[Category:VMware]]
Retrieved from ‘https://vwiki.co.uk/index.php?title=Special:MobileDiff/787...2155’

Navigation menu