Troubleshooting (ESX) - Revision history

Sstrutt: /* Random Problems */ Added ""Host XXX currently has no management network redundancy" warning"

2012-08-16T13:33:53Z

Random Problems: Added ""Host XXX currently has no management network redundancy" warning"

@@ Line 562: / Line 562: @@
 The above was gleamed from the script found at http://www.ivobeerens.nl/2011/11/11/enable-or-disable-remote-tech-support-mode-ssh/
 [[Category:ESX]]
 [[Category:Troubleshooting]]

Sstrutt: Added Troubleshooting category

2012-07-03T12:12:52Z

Added Troubleshooting category

← Older revision		Revision as of 12:12, 3 July 2012
Line 561:		Line 561:

	The above was gleamed from the script found at http://www.ivobeerens.nl/2011/11/11/enable-or-disable-remote-tech-support-mode-ssh/		The above was gleamed from the script found at http://www.ivobeerens.nl/2011/11/11/enable-or-disable-remote-tech-support-mode-ssh/




	[[Category:ESX]]		[[Category:ESX]]
			[[Category:Troubleshooting]]

Sstrutt: /* Storage Monitor Log Entries */ Minor review

2012-05-14T11:19:40Z

Storage Monitor Log Entries: Minor review

← Older revision		Revision as of 11:19, 14 May 2012
Line 190:		Line 190:

	=== Storage Monitor Log Entries ===		=== Storage Monitor Log Entries ===
	How to decode the following type of entries...		How to decode the following type of vmkernel log entries that are generated by the Storage Monitor...
	Sep 3 15:15:14 esx1 vmkernel: 85:01:23:01.532 cpu4:2264)StorageMonitor: 196: vmhba1:2:0:0 status = 2/0 0x6 0x2a 0x1		Sep 3 15:15:14 esx1 vmkernel: 85:01:23:01.532 cpu4:2264)StorageMonitor: 196: vmhba1:2:0:0 status = 2/0 0x6 0x2a 0x1
	Sep 3 15:15:32 esx1 vmkernel: 85:01:23:19.391 cpu4:2253)StorageMonitor: 196: vmhba1:3:9:0 status = 2/0 0x6 0x2a 0x1		Sep 3 15:15:32 esx1 vmkernel: 85:01:23:19.391 cpu4:2253)StorageMonitor: 196: vmhba1:3:9:0 status = 2/0 0x6 0x2a 0x1
Line 251:		Line 251:
	\| 0x0 \|\| No Sense Information		\| 0x0 \|\| No Sense Information
	\|-		\|-
	\| 0x1 \|\| Last command completed but ~~used~~ error correction		\| 0x1 \|\| Last command completed but required error correction to complete
	\|-		\|-
	\| 0x2 \|\| Unit Not Ready		\| 0x2 \|\| Unit Not Ready
	\|-		\|-
	\| 0x3 \|\| Medium Error		\| 0x3 \|\| Medium Error (non-recoverable data error)
	\|-		\|-
	\| 0x4 \|\| Hardware Error		\| 0x4 \|\| Hardware Error (non-recoverable hardware error)
	\|-		\|-
	\| 0x5 \|\| ~~ILLEGAL_REQUEST~~ (Passive SP)		\| 0x5 \|\| Illegal request (Passive SP)
	\|-		\|-
	\| 0x6 \|\| LUN Reset		\| 0x6 \|\| LUN Reset
Line 269:		Line 269:
	\| 0xa \|\| Copy_Aborted		\| 0xa \|\| Copy_Aborted
	\|-		\|-
	\| 0xb \|\| Aborted_Command - Target aborted command		\| 0xb \|\| Aborted_Command - Target disk aborted command
	\|-		\|-
	\| 0xc \|\| Comparison for SEARCH DATA unsuccessful		\| 0xc \|\| Comparison for SEARCH DATA unsuccessful
Line 278:		Line 278:
	\|}		\|}

	~~The~~ Additional Sense Code and Additional Sense Code Qualifier ~~mean~~		In order to decode the Additional Sense Code and Additional Sense Code Qualifier meanings see [[SCSI Disk Additional Sense Codes]] or http://www.adaptec.com/en-US/support/scsi/2940/AHA-2940AU/use_prod/SCSI_event_codes.htm?nc=/en-US/support/scsi/2940/AHA-2940AU/use_prod/SCSI_event_codes.htm
	~~{\| class="vwikitable"~~
	\|-
	~~! Hex !!~~ Sense ~~Code~~
	\|-
	~~\| 0x4 \|\| Unit Not Ready~~
	\|-
	~~\| 0x3 \|\| Unit Not Ready - Manual Intervention Required~~
	\|-
	~~\| 0x2 \|\| Unit Not Ready - Initializing Command Required~~
	\|-
	~~\| 0x25 \|\| Logical Unit Not Supported (eg LUN doesn't exist)~~
	\|-
	~~\| 0x29 \|\| Device Power on~~ or ~~SCSI Reset~~
	\|}

	~~For further info on sense codes see -~~ http://www.adaptec.com/en-US/support/scsi/2940/AHA-2940AU/use_prod/SCSI_event_codes.htm?nc=/en-US/support/scsi/2940/AHA-2940AU/use_prod/SCSI_event_codes.htm

	=== Recovering VM's from failed storage ===		=== Recovering VM's from failed storage ===

Sstrutt: /* High Availability */ Reviewed the HA section

2012-05-11T09:54:56Z

High Availability: Reviewed the HA section

← Older revision		Revision as of 09:54, 11 May 2012
Line 423:		Line 423:
	* Primary reconfigured for HA		* Primary reconfigured for HA

	~~It's quite common for~~ HA ~~to go into an error state, normal course of action is to use~~ the '~~''Reconfigure for HA''' option for~~ the ~~ESX that's experiencing the problem. This reinstalls the HA agent onto the ESX onto the ESX. It's also common to have to do this a couple of times for it to be successful. Other things~~ to ~~try...~~		If HA has never worked on the cluster then you're best skipping the [[#First Fixes\|First Fixes]] section, and should proceed to [[#Check DNS\|Check DNS]] and/or check the [[#Error Hints\|Error Hints]] to help diagnose what might be wrong.
	* Restart the HA process - see [[#~~High_Availability_Stop.2FStart~~\|~~High Availability Stop/Start~~]]
	* [[#~~Manually Deinstall~~\|~~Deinstall HA and VPXA~~]] ~~and reinstall~~

	HA is very dependant on proper DNS, to check everything is in order do the following from each ESX. Some versions of ESX3 are sensitive to case, always user lower, FQDN of ESX's should be lower case, and ~~VC's~~ FQDN and domain suffix search should be lower case		=== First Fixes ===
			It's quite common for HA to go into an error state, and so there are some normal first things to try. Neither of these should affect running VM's on any of the ESX's in the cluster (though see note above regarding Isolation Response).

			# '''Reinstall HA on an ESX'''
			#* This will cause the HA Agent to be reinstalled on the ESX, though note that the installer package is not refreshed from the vCentre if the right version is already on the ESX - it can be worth attempting twice
			## Right-click over the problem ESX and select ''Reconfigure for VMware HA''
			# '''Reinstall HA on the Cluster'''
			#* This will cause a complete rebuild of the HA cluster configuration, with HA being reinstalled on all ESX's. This can make matters worse if there are critical configuration problems with the cluster (those these deteriorations are inevitable, but maybe worth avoiding during production hours if in a high-blame environment)
			## Right-click over the problem Cluster and select ''Edit Settings...''
			## Untick the ''Turn On VMware HA'' option, and click ''OK''
			## Wait for HA to be removed from all ESX's
			## Right-click over the problem Cluster and select ''Edit Settings...''
			## Tick the ''Turn On VMware HA'' option, and click ''OK''
			## Wait for HA to be installed on all ESX's (it can be worth reinstalling on a few ESX's after this if there are a few persistent ESX's in error)

			If the above fails then proceed to the sections below to investigate further.

			=== Check DNS ===
			HA is very dependant on proper DNS, to check everything is in order do the following from each ESX. Some versions of ESX3 are sensitive to case, always user lower, [[Acronyms#F\|FQDN]] of ESX's should be lower case, and vCentre FQDN and domain suffix search should be lower case
	# Check that the hostname/IP of the local ESX is as expected		# Check that the hostname/IP of the local ESX is as expected
	#* <code> hostname </code>		#* <code> hostname </code>
Line 443:		Line 459:
	# Check the vCentre's FQDN and DNS suffix search are correct and lower case		# Check the vCentre's FQDN and DNS suffix search are correct and lower case

	If you need to correct DNS names, ~~don't be surprised if~~ you need to reinstall HA and VPXA~~, it~~ can be done without interrupting running VM's, but its obviously a lot less stressful not to.		If you need to correct DNS names, its likely that you will need to reinstall HA and VPXA on individual ESX's or the whole cluster. This can be done without interrupting running VM's, but its obviously a lot less stressful not to.

	=== Manually ~~Deinstall~~ ===		=== Manually De-Install ===
			Sometimes reinstalling via the VI Client doesn't do a full enough job, or it can fail, so you have to revert to doing it yourself.
	# Put the ESX into maintenance mode (optional - VM's can be left running on ESX)		# Put the ESX into maintenance mode (optional - VM's can be left running on ESX)
	# Disconnect the ESX from the Virtual Centre		# Disconnect the ESX from the Virtual Centre
Line 452:		Line 469:
	# <code> ./VMware-vpxa-uninstall.sh </code>		# <code> ./VMware-vpxa-uninstall.sh </code>
	# <code> ./VMware-aam-ha-uninstall.sh </code>		# <code> ./VMware-aam-ha-uninstall.sh </code>
	# ~~Reconect~~ the ESX to the VC		#* See note below if the uninstallers fail to remove files
			# Reconnect the ESX to the VC
	# Take out of maintenance mode		# Take out of maintenance mode

Line 464:		Line 482:
	# Take out of maintenance mode		# Take out of maintenance mode

	If the VC Agent or HA Agent fails due to the uninstaller being unable to remove files/folders, and you can't remove them manually, this is an indication that the disk is becoming corrupt. Especially if installed on a USB key, consider replacing ASAP. This is most likely to occur in ESX3 where HA tends to wear the USB key out, this is ~~fixed~~ in ESX4.		If the VC Agent or HA Agent fails due to the uninstaller being unable to remove files/folders, and you can't remove them manually, this is an indication that the disk is becoming corrupt. Especially if installed on a USB key, consider replacing ASAP. This is most likely to occur in ESX3 where HA tends to wear the USB key out, this is less of an issue in ESX4 where the amount of writing to the USB key is kept to a minimum. To work around, rename the folder(s) that were unable to be removed/modified.
			* <code> mv /opt/vmware/aam /opt/vmware/aam_old </code>
			* <code> mv /opt/vmware/vpxa /opt/vmware/vpxa_old </code>

	=== Command Line Interface etc ===		=== Command Line Interface etc ===

Sstrutt: /* Log Bundles Creation Fails */ Typo fix

2012-03-12T08:42:48Z

Log Bundles Creation Fails: Typo fix

← Older revision		Revision as of 08:42, 12 March 2012
Line 539:		Line 539:

	* '''Could not copy...Have you run out of disk space?'''		* '''Could not copy...Have you run out of disk space?'''
	** ESX - Check that there's space to be able to write in <code>/tmp<code>		** ESX - Check that there's space to be able to write in <code>/tmp</code>
	** ESXi - Check that the ESX has been configured with a scratch disk, and that it has space		** ESXi - Check that the ESX has been configured with a scratch disk, and that it has space
	* '''tar: write error: Broken pipe'''		* '''tar: write error: Broken pipe'''

Sstrutt: /* Random Problems */ Added ""Remote Tech Support Mode (SSH) for the host XXX has been enabled" warning"

2012-03-12T08:42:11Z

Random Problems: Added ""Remote Tech Support Mode (SSH) for the host XXX has been enabled" warning"

@@ Line 543: / Line 543: @@
 * '''tar: write error: Broken pipe'''
 ** ESXi - Check that the ESX has been configured with a scratch disk
 [[Category:ESX]]

Sstrutt: /* ESXi Tech Support Mode */ Minor rewording

2012-02-29T14:37:27Z

ESXi Tech Support Mode: Minor rewording

← Older revision		Revision as of 14:37, 29 February 2012
Line 53:		Line 53:

	=== ESXi Tech Support Mode ===		=== ESXi Tech Support Mode ===
	There's no Service Console on ESXi, so you have to do without. Well almost, there is the ''unsupported'' Tech Support Mode, which is a lightweight Service Console, to enable...		There's no Service Console on ESXi, so you have to do without. Well almost, there is the ''unsupported'' Tech Support Mode, which is like a lightweight Service Console, to enable SSH access to your ESX...

	'''ESXi 3.5 and 4.0'''		'''ESXi 3.5 and 4.0'''

Sstrutt: Update table style

2012-02-21T22:50:22Z

Update table style

← Older revision		Revision as of 22:50, 21 February 2012
Line 4:		Line 4:
	'''Timestamps in logfiles are in UTC !!!'''		'''Timestamps in logfiles are in UTC !!!'''
	=== ESX ===		=== ESX ===
	{\|~~cellpadding~~="~~2" cellspacing="0" border="1~~"		{\|class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Item !! Path !! Comments		! Item !! Path !! Comments
	\|-		\|-
Line 35:		Line 35:
	However, this is most easily achieved if you've got the PowerCLI installed, in which case see [[VI_Toolkit_(PowerShell)#ESXi_Logs\|ESXi Logs via PowerCLI]]		However, this is most easily achieved if you've got the PowerCLI installed, in which case see [[VI_Toolkit_(PowerShell)#ESXi_Logs\|ESXi Logs via PowerCLI]]

	{\|~~cellpadding~~="~~2" cellspacing="0" border="1~~"		{\|class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Name !! PowerCLI Key !! Diagnostic Dump Path !! Comments		! Name !! PowerCLI Key !! Diagnostic Dump Path !! Comments
	\|-		\|-
Line 77:		Line 77:
	== ESXTOP ==		== ESXTOP ==

	{\|~~cellpadding~~="~~2" cellspacing="0" border="1~~"		{\|class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Key !! Change View !! Key !! Sort by		! Key !! Change View !! Key !! Sort by
	\|-		\|-
Line 149:		Line 149:
	CPU Ready can creep up if the the system is pushed, or if the VM has multiple CPUs (as it needs multiple physical CPUs to become available at the same time, aka CPU Co-Scheduling). Multiple CPU's are especially a problem in environments where there are large number of SMP VM's.		CPU Ready can creep up if the the system is pushed, or if the VM has multiple CPUs (as it needs multiple physical CPUs to become available at the same time, aka CPU Co-Scheduling). Multiple CPU's are especially a problem in environments where there are large number of SMP VM's.

	{\|~~cellpadding~~="~~2" cellspacing="0" border="1~~"		{\|class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! % CPU Ready !! MSec CPU Ready !! Performance		! % CPU Ready !! MSec CPU Ready !! Performance
	\|-		\|-
Line 174:		Line 174:

	Latency occurs when the hypervisor or physical storage cannot keep pace with the demand for IO, as a rough guide to indicate if theres a problem or not...		Latency occurs when the hypervisor or physical storage cannot keep pace with the demand for IO, as a rough guide to indicate if theres a problem or not...
	{\|~~cellpadding~~="~~2" cellspacing="0" border="1~~"		{\|class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Latency up to !! Status		! Latency up to !! Status
	\|-		\|-
Line 195:		Line 195:

	The status message consists of the follow four decimal and hex blocks...		The status message consists of the follow four decimal and hex blocks...
	{\| ~~cellpadding~~="~~4" cellspacing="0" border="1~~"		{\| class="vwikitable"
	\|-		\|-
	\|''Device Status'' / ''Host Status'' \|\| ''Sense Key'' \|\| ''Additional Sense Code'' \|\| ''Additional Sense Code Qualifier''		\|''Device Status'' / ''Host Status'' \|\| ''Sense Key'' \|\| ''Additional Sense Code'' \|\| ''Additional Sense Code Qualifier''
Line 205:		Line 205:
	Mar 2 10:04:44 vmkernel: 81:00:00:15.893 cpu8:4633964)StorageMonitor: 196: vmhba0:0:4:0 status = D:0x28/H:0x0 0x0 0x0 0x0		Mar 2 10:04:44 vmkernel: 81:00:00:15.893 cpu8:4633964)StorageMonitor: 196: vmhba0:0:4:0 status = D:0x28/H:0x0 0x0 0x0 0x0

	{\| ~~cellpadding~~="~~4" cellspacing="0" border="1~~"		{\| class="vwikitable"
	\|-		\|-
	\|<code>D:</code>''Device Status'' / <code>H:</code>''Host Status'' \|\| ''Sense Key'' \|\| ''Additional Sense Code'' \|\| ''Additional Sense Code Qualifier''		\|<code>D:</code>''Device Status'' / <code>H:</code>''Host Status'' \|\| ''Sense Key'' \|\| ''Additional Sense Code'' \|\| ''Additional Sense Code Qualifier''
Line 211:		Line 211:

	Where the ESX Device and SAN host status' mean...		Where the ESX Device and SAN host status' mean...
	{\| ~~cellpadding~~="~~4" cellspacing="0" border="1~~"		{\| class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Decimal !! Device Status !! Host Status !! Comments		! Decimal !! Device Status !! Host Status !! Comments
	\|-		\|-
Line 245:		Line 245:

	Where the Sense Key mean...		Where the Sense Key mean...
	{\| ~~cellpadding~~="~~4" cellspacing="0" border="1~~"		{\| class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Hex !! Sense Key		! Hex !! Sense Key
	\|-		\|-
Line 279:		Line 279:

	The Additional Sense Code and Additional Sense Code Qualifier mean		The Additional Sense Code and Additional Sense Code Qualifier mean
	{\| ~~cellpadding~~="~~4" cellspacing="0" border="1~~"		{\| class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Hex !! Sense Code		! Hex !! Sense Code
	\|-		\|-
Line 474:		Line 474:
	The interface is a bit dodgy, you can enter the same command twice, and it'll be rejected one time and accepted another, patience is required.		The interface is a bit dodgy, you can enter the same command twice, and it'll be rejected one time and accepted another, patience is required.

	{\|~~cellpadding~~="~~4" cellspacing="0" border="1~~"		{\|class="vwikitable"
	\|- ~~style="background-color:#bbddff;"~~		\|-
	! Command !! Comments		! Command !! Comments
	\|-		\|-

Sstrutt: /* Storage */ Added "Recover lost SAN VMFS partition (ESX4)"

2011-11-21T10:28:16Z

Storage: Added "Recover lost SAN VMFS partition (ESX4)"

@@ Line 345: / Line 345: @@
 # Via the ESX, browse the datastore and re-add the VM's to the inventory (right-click over the .vmx file)
 #* For a Virtual Machine Question about what to do about a UUID, select Keep
 === USB / SD Hypervisor Checks ===

Sstrutt: /* Recover lost SAN VMFS partition */ Renamed to "Recover lost SAN VMFS partition (ESX3)"

2011-11-21T10:13:56Z

Recover lost SAN VMFS partition: Renamed to "Recover lost SAN VMFS partition (ESX3)"

← Older revision		Revision as of 10:13, 21 November 2011
Line 332:		Line 332:
	# Power up new new VM’s on SAN datastore, confirm OK, then delete ''myserver''-recov servers		# Power up new new VM’s on SAN datastore, confirm OK, then delete ''myserver''-recov servers

	=== Recover lost SAN VMFS partition ===		=== Recover lost SAN VMFS partition (ESX3) ===
	EG After a powerdown, ESX's can see the SAN storage, but the VMFS cannot be found in the Storage part of the ESX config, even after Refresh. To fix, the VMFS needs to be resignatured...		EG After a powerdown, ESX's can see the SAN storage, but the VMFS cannot be found in the Storage part of the ESX config, even after Refresh. To fix, the VMFS needs to be resignatured...

	'''Do not attempt to ''Add Storage'' to recover the VMFS, this will format the partition'''		'''Do not attempt to ''Add Storage'' to recover the VMFS, this will format the partition'''

	# On one of the ESX's, in Advanced Settings, change LVM.EnableResignature to 1		# On one of the ESX's, in '''Advanced Settings''', change <code>LVM.EnableResignature</code> to <code>1</code>
	# '''Refresh''' Storage, the VMFS should be found with a new name, something like snap-000000002-''OriginalName''.		# '''Refresh''' Storage, the VMFS should be found with a new name, something like snap-000000002-''OriginalName''.
	# '''Remove from Inventory''' all VM's from the old storage, the old storage should disappear from the list of datastores		# '''Remove from Inventory''' all VM's from the old storage, the old storage should disappear from the list of datastores
	# Rename the found storage to the original name		# Rename the found storage to the original name
	# '''Refresh''' Storage on all other ESX's, they should see the VMFS again		# '''Refresh''' Storage on all other ESX's, they should see the VMFS again
	# Revert LVM.EnableResignature on the appropriate ESX		# Revert <code>LVM.EnableResignature</code> on the appropriate ESX to <code>0</code>
	# Via the ESX, browse the datastore and re-add the VM's to the inventory (right-click over the .vmx file)		# Via the ESX, browse the datastore and re-add the VM's to the inventory (right-click over the .vmx file)
	#* For a Virtual Machine Question about what to do about a UUID, select Keep		#* For a Virtual Machine Question about what to do about a UUID, select Keep