Delayed Write Failed – Event ID 50 – Solved

Saw this issue though OpsMgr 2012 R2 console, telling me that all Windows Server 2012 and 2012 R2 servers where reporting this issue every evening when the backup kicked in.

There are a lot of blog posts on the web telling you to disabling the cache on the disks and so on, but this is a default setting i did not want to change. And in my opinion that is a workaround, not a fix.

So working with this issue led me to an known error from VMware: KB2006849

Telling me that this has been solved in a patch from the 27th of january 2015 and later versions.

Version: ESXi 5.5 Patch 4
Release: 2015-01-27
Build: 2403361

To verify which version you are running, then look in the vSphere Client under one of your hosts:

VMware version

Then compare that number with this website:Build number history

If you have a build version before 2403361 (like the above example) – then you need to update your ESX hosts and the VMware tools on the virtual machines.

This solves the issue.

Event id numbers which leads to this issue:

Event ID 50, 57, 137, 140, 157 and 12289

Best regards.

Mads

How to reclaim storage on HP 3Par systems.

3Par does not seem to be that effective at reclaiming space and requires some manual tasks to free up space, due to the detection method which is “Zero detect” or “UNMAP”.

To Release space on VMware

On VMware ESX, if you copy or data migrate 1TB of data from one Datastore to another Datastore: HP 3Par does not see the move and now sees 1TB+1TB of data, since the old data locations are not zeroed out or that VMware does not give instructions to 3Par about the released storage – this feature is also called UNMAP.

This seems to be due to the fact that VMware no longer has the UNMAP feature enabled which reclaims space after data has been deleted or moved

UNMAP: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

So VMware knows the data has moved and shows correctly in Vsphere console – But since the old data blocs are not reclaimed/zeroed out HP cannot see it and release the space.

Recommendations is to run UNMAP manually by the following command on a datastore trough Putty SSH on a ESX 5.5 host

”Esxcli storage vmfs unmap –l DATASTORENAME”

It will have a small performance impact when run.

ESX less than 5.5 have to use the “vmkfstools –y 80” command on a datastore to create a balloon file with empty data taking up 80% of the free space in this example.

https://kallesplayground.wordpress.com/2014/04/03/storage-reclamation-part1/

To release space on Windows Servers

When files are deleted in Windows 2003 and 2008, they are not zeroed out which 3Par can detect and release.

UNMAP is supported from Windows Server 2012 + R2 – so this should help 3Par release storage.

Windows Server 2003/2008  server requires Sdelete.exe tool to Zero out data.

Example to Zero out unused disk space on volume Z: “Sdelete.exe –Z J:\ “

Requires volumes to have letters, and from Sdelete version 1.6 you have to use the Z switch (previous –c option)

After data have been zeroed out, it will take some time for 3Par to release the space. HP tells me that the 3Par system will not prioritize this task under heavy load and that there are no way of activate this task manually or see the status of it. Nothing left then to wait and see, HP told me that I should look again after a weekend to see if the data have been released.

The virtual volumes also needs to have “Zero Detect” enabled.

3ParVolumeZeroDetect

So this is the theory, but I do still struggle with the 3Par to release the space.

In this example, HP 3Par detects that the volume is only 50% filled – but 3Par holds 5.5TB in custod

3ParVolumes1

HP 3PAR Thin Technologies Whitepaper

http://h20195.www2.hp.com/v2/GetPDF.aspx%2F4AA3-8987ENW.pdf

A easy to understand blog about thin provisioning including Video demonstration

http://h30507.www3.hp.com/t5/Around-the-Storage-Block-Blog/3PAR-Hands-on-lab-zero-detect/ba-p/97205

 

I have used all these commands without issues, but do it at your own risk.

How to remove vmkdump files from VMware ESX 5.x

So how can a datastore be deleted if there is an active file in a vmkdump folder that resides in the datastore running Vsphere Client and ESX 5.x?

When browsing the datastore, a vmkdump folder shows up with a file in it, this file is in use and prevents the datastore to be deleted, it is a log dump file and in my case could be remove and re-added safely to another store.

The files cannot be migrated by the Vsphere Client and trying to do so or delete dem gives an error: “Device or resource busy”

One .dumpfile is created by each host in a random datastore, and to delete and recreate them, you have to find the owner of the file, to do so, take the ID of the file and log in to your hosts by this adress to find their ID which have to match the filename:

https://SERVERNAME/mob/?moid=ha-host&doPath=hardware.systemInfo

then logon to the ESX host with putty, and run this command: “esxcli system coredump file remove –F”

Then recreate the file at a new destination with: “esxcli system coredump file add -f SERVERNAME -d DATASTORENAME”

Make the file active: “esxcli system coredump file set -p /vmfs/volumes/DATASTORE_UUID/vmkdump/FILENAME”

Run this command to see if the file is active: “esxcli system coredump file list”

So to solve this puzzle, these to blogs did the math.

http://jbcomp.com/remove-vmkdump-files/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2077516