In Part 1 we installed Vmware Player and installed Windows 2008 R2 Enterprise into a virtual machine. In this second part, we’ll look at setting up another virtual machine and deploying our first domain controller for the testing environment.

We could definitely go back and setup another virtual machine using the the same process as before. However, this takes too long; the least of which we have to wait for Windows 2008 R2 to re-install.

An easier option is to copy the existing virtual machine files to a new directory and paste it as a new virtual machine.

1. Right click on our perviously built virtual machine (inv-dc01), select Copy and then right click and Paste into the same directory. This will create two copies of the virtual machine, inv-dc01 (the original) and inv-dc01 – Copy. This process can be repeated as many times as we like until the local drive is full.

   

   For the moment, all that is required is the two copies of the virtual machine.

2. Rename inv-dc01 – Copy to _TMPL-inv-dc01. Now we have a TEMPLATE version of a virtual machine with Windows 2008 R2 installed and can continue to take copies of it as needed.

Onto setting up our domain controller.

1. Open the Vmware virtual machine configuration file by double clicking or right click and open the Vmware virtual machine configuration file with Vmware Player.
   
2. Once the operating system has started, use CTRL+ALT+INSERT to get to the login screen.
   
   At this point login as the user we created or login as Administrator. If you login as Administrator, Windows will prompt you to change the password. The Administrator password will also have to meet complexity requirements ie. uppercase, lowercase, letter, number and special characters.
3. Next up is the all important Vmware Tools. VMware Tools package adds drivers and utilities to improve the graphical performance for different guest operating systems, including mouse tracking. The package also enables some integration between the guest and host systems, including shared folders, plug-and-play devices, clock synchronisation, and cutting-and-pasting across environments.
   
   Click “Install Tools” in the bottom bar of the window to start the install process.
4. You may want to close this window for the moment to allow the Vmware Tools installer to run.
5. If this dialog fails to appear, open Windows Explorer, browse to the CD drive of the virtual machine and run setup.exe manually.
   
   Install Vmware Tools using the default settings; there is no real benefit to be gained from changing the default options in our scenario.
   
   Once the installer has completed, reboot the virtual machine and then log back in when it has restarted.

Once the desktop is initialised, there are a couple of configurations required before we can create the domain controller. Primarily, the IP address and the Windows machine name.

1. The domain controller ABSOLUTLEY has to have a statically assigned IP address, and the easiest (and most fun!) way to do this is set it via the command line:

   netsh interface ip show config

   
   This will show all the adapters currently available to the virtual machine. The interface that we are interested in is Local Area Connection. Once we have establised that it is available and has an IP address allocated from DHCP, it needs to be changed to a static IP address.

   netsh interface ip set address name="Local Area Connection" static 192.168.0.10 255.255.255.0 192.168.0.1 1

   This set the static IP of 192.168.0.10, subnet mask of 255.255.255.0 and default gateway (which we don’t need for the moment) of 192.168.0.1. For the purposes of completeness, it’s best to put the gateway in.
   DNS is still required, but configuring the domain controller will automatically install this service on the domain controller.
   Running ipconfig from the command line should verify the configuration has been applied.

2. Finally we’re at the point of installing the domain controller! From the command prompt, run dcpromo to start the install.
   
3. From here it is a simple matter of following the prompts.
   • At the Active Directory Domain Services Installation Wizard make sure Use advanced mode installation is NOT selected and press Next.
   • The next window is a warning about incompatibilites with Windows NT 4.0. Just press Next.
   • Choose a Deployment Configuration – Check the Create a new domain in a new forest and press Next.
   • Name the Forest Root Domain – In the FQDN of the forest root domain enter the name of your network. For our purposes, any name will do. invurted.local is acceptable. Press Next.
   • Set forest functional level – Select Windows 2008 R2 from the drop down list. This will mean that all domain controllers have to have at least Windows 2008 R2 as their operating systems. Press Next.
   • Additional Domain Controllers – Make sure the DNS Server option is ticked and press Next. Select Yes for any dialog boxes that appear.
   • Location for Database, Log Files and SYSLOG – Accept the default locations and press Next.
   • Directory Services Restore Mode Password – Enter the password of P@ssw0rd in both fields and press Next.
   • Summary – Verify your settings and press Next.
   • Active Directory Domain Services Installation WizardActive Directory will now start installing. At the completion, reboot the virtual machine. If everything is successful, upon reboot, your domain controller setup is complete.

Upon successful restart and login using the Administrator username and password, we want to look at our new domain. From the start menu, select Administrative Tools and launch the Active Directory Users and Computer program. This will allow us to create any users, groups and Organisation Units that are required.

I almost forgot! We need to rename our domain controller. In previous versions of Windows this was painful, especially as the virtual machine is a domain controller. In Windows 2008 there is more tolerance to name changes. It’s not the sort of thing that I would want to do regularly, but is possible.
In the start menu, right click on Computer and select Properties.

Chose Change settings and accept any warnings that appear on screen.

In the Sytem Properties, select Change and, again, accept any warning that appear.

In the computer name field enter the new name for the computer. For our purposes, inv-dc01 will do fine. Press OK twice and then close the system properties. At this point, it is a good idea to restart the machine again.

That’s Part 2 complete. In Part 3 we’ll look at configuring the virtual machine as a domain member server.

As ever, if you have any questions, please email me (contactme [at] invurted dot com).

Following a number of requests, I have written a tutorial on setting up a Windows 2008 testing environment. Part 1 looks at installing Vmware Player and setting up the first virtual machine
ASSUMPTIONS:

• A supported operating system is used. Vmware Player supports most Microsoft operating systems as well as a number of Linux distributions. For the purposes of this tutorial, Windows 7 is the operating system used as the base.
• The account on the Windows 7 base operating system is either a member of the Local Administrators group or can elevate priveledges as required.
• At least 50 – 100 GB of storage space is available for all the virtual machines. It’s better to err on the side of caution.
• Vmware Player is the chosen hypervisor. Vmware Workstation could as easily be used if available. We use Vmware Player as the hypervisor due to it’s cross platform compatibility and its free licensing.
• We are only doing the setup of the first virtual machine. Further configuration will be done in subsequent tutorials.

On to the fun bits! As with most hypervisor technologies, the result of what we are doing here is allowing me to run two (or more) instances of operating systems on the one hardware platform. Vmware Player is our chosen hypervisor because it removes a lot of the guess work from deploying virtual machines. Vmware Workstation definitely gives us more granular control over the final conifguration. For our purposes, the default RAM, CPU and storage configurations are more than acceptable.

1. Install Vmware player following all the default configuration options. There is little need to change the installation path etc. unless absolutley necessary.
2. Once installed, run the program and select “Create new Virtual Machines”.
   
   This will start the wizard driven configuration of the new virtual machine that we will use as the basis for the testing environment.
3. One of the first configuration options is the installation media. We’re focussing on the first two options; providing either a CDROM or an ISO file to install from. It’s possible to network boot a virtual machine, but that’s something for later.
   If we load a CDROM into the drive, Vmware Player should recognise the operating system. Similarly with an ISO file.
   
4. Next up, the all important licence key. Windows 2008 R2, Enterprise will install without a key being provided. It will install in Evaluation Mode giving us sixty days to activate with a valid licence. Again, for purely testing purposes, we can ignore the licence key and come back to it later if we need to. The only configuration to be aware of is the version of Windows 2008 to install. The CDROM (or ISO file, in this case) has a number of different versions available. Make sure Windows 2008 R2 Enterprise is the selected option.
   
   The only other piece of information needed is the first user who is installed. The username can be Administrator, but Vmware Player will throw back a warning. Windows 2008 doesn’t treat the Administrator account the same as previous operating systems. To perform most administrative functions, we still have to elevate the credentials. The user name is invurted and a password of P@ssw0rd to meet complexity requirements.
5. Now we apply the name and location of the virtual machine. This is where the space requirement comes in. By default, Vmware Player saves the virtual machine files to the current user’s directory. Both the name and the save location can be changed at this point. A recommendation would be make sure the volume that you are saving the virtual machines to has enough space for all the hard disks that will be created.
   
6. The next step is to configure the virtual hard disk. In the physical world this would be the hard disk used as an installation target for Windows. It’s exactly the same in the virtual machine. The main difference is that the size represents the maximum size that the “physical” hard drive will be reported as to the Windows operating system.
   
   As this server will only provide domain controller and networking services for the lab environment, it’s been configured as a 20GB hard drive. In the real world, we would attach a second virtual hard drive of larger size that would store all of the application data, leaving the operating system with a volume all to itself.
7. Almost there!
   
   We have our final configuration ready to go. Before powering on the virtual machine for the first time, we need to customise the virtual machine.
   
   The networking configuration is the one major limitation of Vmware Player. Vmware Workstation has a lot more configuration options available.
   The three options are:
   • Bridged – the virtual machine will use an available physical adapter for the virtual machine. This will appear on the network as a separate device from the host operating system. As out virtual machine will be a domain controller, we don’t want to expose it to the real, potentially production, nework.
     DO NOT use a bridged adapter unless you are absolutley certain it is required.
   • NAT – the virtual machine will use the same IP address assigned to a physical adapter. Both the virtual machine and the host OS will appear on the network as the “same” device.
     DO NOT use a bridged adapter unless you are absolutley certain it is required.
   • Host only – This will create a software based virtual switch that virtual machines will be plugged into. The virtual machine will have no network connectivity unless a virutal routing device is created. This is the safest option as it removes the ability for our test lab to interfere with the real, production network.
8. As the machine powers up for the first time, we should see the familiar Windows starting screen. If at this point you don’t see it, check that the CDROM or ISO file is properly mounted.
   
   We need to mount the CDROM or ISO file through this interface.
   
   Once this is done, press CTRL+ALT+INSERT to force a reset of the virtual machine. If you hit CTRL+ALT+DELETE, this will open up the option to get to the task manager of the host operating system.
9. Go and get a cup of tea, sit back and relax because Windows 2008 is installing and there’s very little required at this point.
   
   Once Windows has finished installing, we should be presented with this screen. From this point on, there is no difference between the virual machine and a real version of Windows 2008.
   Power off the virtual machine and move onto Part 2!

That’s Part 1 complete. In Part 2 we’ll look at configuring the virtual machine as a domain controller and deploying additional servers for the rest of the lab.

As ever, if you have any questions, please email me (contactme [at] invurted dot com).

Greetings, faithful reader,

This one is as much a reminder for me as it is one to you!

When you’re entering Maintenance Mode, the Vmware ESX server will always try to Vmotion all of it’s hosted virtual machines away. Problems can occur when hosts fail to migrate when entering Maintenace Mode. The server will stop at two percent and then time out. Regrettably, there is no notification that an individual Virtual Machine’s Vmotion event fails.

However, there is another cause as well: if your cluster is both HA (High Availability) and DRS (Distributed Resource Scheduling) enabled, when you put an ESX server into Maintenance mode, DRS will generate a five star recommendation. Manual mode, however, requires user interaction and you have to tell the ESX cluster to initiate the Vmotion events. Once I accept the recommendations, Vmotion will start and the target ESX server will be placed in Maintenance Mode.

There is one other case where it will fail. This one falls into the category of “undocumented system feature”. The summary is:

From: http://kb.vmware.com/selfservice/viewContent.do?externalId=1007156&sliceId=1

• An ESX host fails to enter maintenance mode in a VMware High Availability (HA) or DRS cluster
• Hosts fail to migrate when attempting to enter maintenance mode
• The progress indicator remains at 2% indefinitely
• Trying to remediate a host and getting a time out error when trying to enter the maintenance mode

Cause:
This is normal behavior for a VMware HA/DRS cluster that is using strict admission control.

Disabling strict admission control (allowing virtual machines to power on even if they violate constraints) should allow a host to enter maintenance mode in this situation but a bug was discovered whereby it did not.
Resolution:

For a permanent solution, upgrade to VirtualCenter 2.5 Update 3.

To workaround the issue, temporarily disable VMware HA in the cluster settings. You will then be able to put the ESX Server host into Maintence mode and do the work required. You can then re-enable HA on your cluster.

This one is getting filed under “Try it really soon!” However, randomly answering an email, I came across this document on http://www.vminfo.nl detailing the process for making a bootable USB key with ESX 3i installed.

Certainly, the document goes into a lot more detailed instructions, but here’s a bit of a summary:

1. Using an archiving program eg. IZArc, extract the contents of the Vmware ESX 3i ISO image. Given it’s an ISO, you could also mount it using DaemonTools or it’s like.
2. Once the files are extracted, find the INSTALL.TGZ file and open it.
3. Navigate to the \usr\lib\vmware\installer directory.
4. Open the Vmware-VMvisor-big-3.5.0_Update_2-11072.i386.dd.bz2 file. This may take a while, so patience, young padawan! Then extract the un-compressed .dd file
5. Restore the boot image to the USB thumb drive. Effectively, you’re restoring a “virtual hard drive” (in this case the .dd file extracted above.
6. Now that’s done, it’s time to test! Boot the sucker and play with ESX 3i!

The only real “gotcha” in this one is pretty obvious: make sure you have a motherboard that can boot from USB. Thankfully, these days most new motherboards have this ability. If in doubt check your motherboard manual or the manufacturer’s website.

Regrettably, I don’t have a font big enough OR bold enough to stress that this is UNSUPPORTED IN A PRODUCTION ENVIRONMENT!! Try it at your own risk!

ESX 3i on USB

Occassionally, when you try to shut down a virtual machine the progress bar will stop meaning that the process will never complete in the background. The following steps will allow you to force the shutdown of a running virtual machine.

1. Logon to the ESX host where the virtual machine is running and become root.
2. vmware-cmd -l to list all the registered virtual machines.
3. vmware-cmd /path/copied/from/vmware-cmd getstate to get state of the target virtual machine.
   If the state requires an answer:
   • vmware-cmd /path/copied/from/vmware-cmd answer

   If no answer is needed:

   • vmware-cmd /path/copied/from/vmware-cmd stop trysoft

   If trysoft does not work use vmware-cmd /path/copied/from/vmware-cmd stop hard

4. If the vmware-cmd does not work, the next step is to kill the master user world id.
5. cat /proc/vmware/vm/*/names |grep vmname replace vmname with the name of the virtual machine that has hung. From this list, get the VMID
6. less /proc/vmware/vm/vmid value/cpu/status where vmid value is the number from above.
7. Scroll over to the right until you find the group field that shows vm.####. The #### numbers are the master user world id.
8. /usr/lib/vmware/bin/vmkload_app -k 9 #### where #### is the master user world id. If the command is successful you will get a WARNING message that a signal 9 is being sent.
9. If vmkload_app does not help the next thing to try is to crash the virtual machine with the vm-support -X command.
10. vm-support -x to get the vmid.
11. From a directory that has ample space vm-support -X #### where #### is the vmid.
12. Answer all the questions with the default answers. The entire process takes about 10 minutes and creates an archive log that can be submitted to support. It will also crash the vm

iSCSI, as a storage option, has less performance than fibre channel ie. it is limited to the speed of the HBAs or the network card being used, however it is far cheaper and I can leaverage my existing infrastructure to put in an iSCSI SAN. The problem becomes, in a TCP/IP network, how do I protect the data moving between the SAN and the hosts?
To start with, what is iSCSI? Functionally, there is no difference between Fibre Channel, iSCSI and a local SCSI controller. All three send SCSI commands to hard disks. Both Fibre Channel and iSCSI transmit over fiber and TCP/IP networks, respectively; making the network storage appear as “local” storage for the iSCSI client (or initiator).
As I mentioned, iSCSI uses TCP/IP networks to transport the SCSI commands from the initiator to the target (the iSCSI SAN). iSCSI uses the default TCP port 3260 and provides no native security on communication between the initiator and the target.
Here’s where the challenges begin.
To start with, if my iSCSI traffic is shared on a production network there are two problems. The most dangerous is network sniffers for obvious reasons.
The other immediate problem is network contention. If my iSCSI SAN shares the same network as all my production network there is a big problem with performance as my normal traffic will be competing for access with iSCSI.
The solution to both problems is network isolation. This isn’t so much best practice as it is mandatory. Wether the iSCSI network is physically isolated to it’s own infrastructure or using IEEE 802.1Q (VLAN tagging), the iSCSI network should now be invisible and/or inaccessible to the production network. The only notable exception becomes the requirement for servers to access the iSCSI implementation. Easily fixed by the addition of network adapters to the servers. Even if I need to populate each server with additional NICs, this should still be cheaper than the most simple Fibre Channel implementation.
Okay, now we have isolation to provide security but what about securing the communication itself.
The obvious solution is IPSec. The not so obvious problem is the overhead to IPSec. For a storage network, it doesn’t seem optimal to have to encrypt and decrypt each packet as it is transmitted and received. Certainly, as of the writing of this article, there is no support for IPSec in ESX3.5 specifically.
So the only real option left (in ESX particular) is authentication to verify my identity. ESX supports CHAP authentication only for iSCSI initiators and targets.
CHAP (Challenge Handshake Authentication Protocol) is used to verify the identity of clients in a Point-to-point network. Your identity is verifed by using a three way handshake. After the initial link CHAP authentication is done at random intervals during the entire conversation. The one weakness is the reliance on a shared secret ie. the client’s password. ESX does not support per target credentials either. So one CHAP username and password sent to all iSCSI targets.
The CHAP process goes something like the following:

1. After the completion of the link establishment phase, the authenticator sends a “challenge” message to the peer.
2. The peer responds with a value calculated using a one-way hash function, such as an MD5 checksum hash.
3. The authenticator checks the response against its own calculation of the expected hash value. If the values match, the authenticator acknowledges the authentication; otherwise it should terminate the connection.
4. At random intervals the authenticator sends a new challenge to the peer and repeats steps 1 through 3.

An example of CHAP authentication can be seen between two Cisco 3640 routers. The network is a fairly simple on and looks like this:

After enabling the serial interfaces between the two routers, I enable and configure PPP encapsulation. On the second router (R2) , I enable CHAP authentication and view debugging information on the first router (R1)


*Mar 1 00:25:39.971: Se1/0 PPP: Using default call direction
*Mar 1 00:25:39.975: Se1/0 PPP: Treating connection as a dedicated line
*Mar 1 00:25:39.975: Se1/0 PPP: Session handle[26000001] Session id[0]
*Mar 1 00:25:39.975: Se1/0 PPP: Authorization required
*Mar 1 00:25:40.255: Se1/0 PPP: No authorization without authentication
*Mar 1 00:25:40.259: Se1/0 CHAP: I CHALLENGE id 1 len 23 from "r2"
*Mar 1 00:25:40.299: Se1/0 CHAP: Unable to authenticate for peer
R2 is waiting for authentication from R1 and is unable to get authentication information. Now I configure CHAP authentication on r1 and observe the following:

*Mar 1 00:26:26.867: Se1/0 LCP: Received AAA AUTHOR Response PASS
*Mar 1 00:26:26.867: Se1/0 IPCP: Received AAA AUTHOR Response PASS
*Mar 1 00:26:26.871: Se1/0 CHAP: O SUCCESS id 12 len 4
*Mar 1 00:26:27.055: Se1/0 CHAP: I SUCCESS id 15 len 4
*Mar 1 00:26:27.059: Se1/0 PPP: Sent CDPCP AUTHOR Request
*Mar 1 00:26:27.063: Se1/0 PPP: Sent IPCP AUTHOR Request
*Mar 1 00:26:27.071: Se1/0 CDPCP: Received AAA AUTHOR Response PASS
*Mar 1 00:26:28.055: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to up
The shared secret is exchanged between the two and the Serial interface is changed to an up state.

Following on in the series, here is the second subnet masking example.

You have been given the network range 192.168.128.0/24. Your task is to break it down into six networks. The questions you have to answer are, how many hosts on each network, and what are the network IDs and broadcast IDs for each of the networks.

I will post the answer on Friday 12th December, 2008 at 3:30pm EDST.

Sorry for the lateness of the solution. I have been as sick as a dog for the last few days, hence no update.

Last time we talked breaking networks up based on host requirements. This time we’re working with the network portion of the IP address.

When we were working with hosts, we worked from the thirty-second bit. Or, from right to left. This time we’re working from left to right to break up an IP range.

Let’s look at the IP address in binary first. Because it’s a twenty-four bit subnet mask, this tells us that it’s a Class C network and that the first three octets are locked and can’t be changed. Therefore, we will be breaking up the fourth octet.
192.168.128.00000000
255.255.255.00000000

Similarly to subnetting for hosts, subnetting for networks is achieved by modifying the subnet mask. So rather than working from the last bit leftwards, let’s start with the first bit I am allowed to modify and work to the right. In our example, the twenty-fifth bit (the first bit of the fourth octet).

For each bit I change, I get 2^bits networks. Unlike hosts, I don’t lose the first IP address to the network ID or the last IP address to the broadcast. Therefore, if I use one bit I get two usable networks. If we extend this to cover our example, two bits gives me four networks (still not enough!). Can I get exactly six networks? Because I’m working with binary values (powers of two), it’s not possible to get exactly six networks. If I use three bits, I get eight networks. More than the required six.

First thing’s first: the subnet mask. The original Class C has been subnetted. Now I am using the first three bits of the fourth octet as the network portion. It did look like this: 255.255.255.00000000. Now, after subnetting, it looks like this: 255.255.255.11100000 or, (after converting back to decimal) 255.255.255.224.

My networks look something like this (the spaces represent the division between my new network ID and new host ID):
192.168.128.000 00000
192.168.128.001 00000
192.168.128.010 00000
192.168.128.011 00000
192.168.128.100 00000
192.168.128.101 00000
192.168.128.110 00000
192.168.128.111 00000

How many hosts on each network? Seeing that we are using three bits for the network, the remaining five bits are the host portion ie. (2^5)-2 hosts or 32-2 hosts. A total of 30 hosts per network. Going back to the last tutorial, the host portion can’t be all zeroes, nor can it be all ones.

Therefore, my network ranges look like this:
192.168.128.000 00000 192.168.128.000 11111
192.168.128.001 00000 192.168.128.001 11111
192.168.128.010 00000 192.168.128.010 11111
192.168.128.011 00000 192.168.128.011 11111
192.168.128.100 00000 192.168.128.100 11111
192.168.128.101 00000 192.168.128.101 11111
192.168.128.110 00000 192.168.128.110 11111
192.168.128.111 00000 192.168.128.111 11111

Any questions or problems, please email me contactme@invurted.com or my work email address.

After the other week’s CCENT course, I made a promise to start emailing subnetting examples out. Rather than writing and re-writing the same thing time and time again, I figured I’d put them up here instead.

So, fixed length subnet masking example #1 is this:

You have been given the network range 192.168.128.0/22. Your first task is to break this network up into blocks of one hundred IP addresses. How many blocks of IP addresses will you get (how many networks will you have)? What are the network IDs and what are the broadcast IP addresses for each of the network ranges?

I will post the answer on Tuesday 9th December, 2008 at 10:30am EDST.

Okay. The question specifically asks for hosts. Therefore, we are working from the thirty second bit back to the start to get hosts.

The original subnet mask is a /22 (255.255.252.0). This tells me that the first twenty two bits are “locked away” leaving me the last ten bits to play with. Straight away we can discern at least one piece of information about my network: there are 2^10 total hosts in the network ie. 2^#bits that aren’t locked away, or one thousand and twenty four total hosts.

On to the good stuff. If we look at the IP address in it’s binary format we have to do a decimal to binary conversion on the third and fourth octets. By looking at the subnet mask this tells me it’s not a full block of eight bits in the third octet and the fourth is all zeroes.

Which looks something like this:
192.168.10000000.00000000
255.255.11111100.00000000
Notice twenty two contiguous ones and ten contiguous zeroes.

As previously mentioned, I want hosts. So, I am working from the last bit leftwards. Each zero represents two hosts (1 and 0). Every time I go up a column, I am adding a power of two to the total number of hosts. If I use one column (the thirty-second bit), I have 2^1 hosts, two columns gives me 2^2 hosts etc. Can I get exactly 100 hosts for the network? The answer is no. If I use the last six bits I have sixty-four hosts (2^6), well below the required one hundred hosts! Let’s use the last seven bits. This gives me 2^7 hosts. One hundred and twenty eight is well above the one hundred required, but is the nearest value over one hundred I get given I have to count in powers of two.

My new subnet mask now looks something like this:
192.168.100000 00.0 0000000
255.255.111111 11.1 0000000
The spaces show the modifications that I have made to the network range given the requirements of the scenario.

The three bits between the spaces are telling me what networks I am now defining. Three bits, straight away, tells me I have 2^3 or eight networks in total. The network part of the IP address has to be unique. The eight unique combinations of the network ID are:
192.168.100000 00.0 0000000 (192.168.128.0)
192.168.100000 00.1 0000000 (192.168.128.128)
192.168.100000 01.0 0000000 (192.168.129.0)
192.168.100000 01.1 0000000 (192.168.129.128)
192.168.100000 10.0 0000000 (192.168.130.0)
192.168.100000 10.1 0000000 (192.168.130.128)
192.168.100000 11.0 0000000 (192.168.131.0)
192.168.100000 11.1 0000000 (192.168.131.128)


The rules of TCP/IP state that the host ID can’t be all zeroes and it can’t be all ones. The all zeroes address is the network ID and the all ones address is the broadcast IP address for that network.

In our example, the last six bits are the host ID. Therefore, the network IDs are up above and the broadcast IP addresses are:
192.168.100000 00.0 1111111 (192.168.128.127)
192.168.100000 00.1 1111111 (192.168.128.255)
192.168.100000 01.0 1111111 (192.168.129.127)
192.168.100000 01.1 1111111 (192.168.129.255)
192.168.100000 10.0 1111111 (192.168.130.127)
192.168.100000 10.1 1111111 (192.168.130.255)
192.168.100000 11.0 1111111 (192.168.131.127)
192.168.100000 11.1 1111111 (192.168.131.255)


Any questions or problems, please email me contactme@invurted.com or my work email address.

Stay tuned because I’ll post another example later this week.

Sometimes a file or set of files in a VMFS become locked and any attempts to edit them or delete will give a device or resource busy error, even though the virtual machine associated with the files is not running. If the virtual machine is running then you would need to stop the virtual machine to manipulate the files. If you know that the virtual machine is stopped then you need to find the ESX server that has the files locked and then stop the process that is locking the file(s).

1. Logon to the ESX host where the virtual machine was last known to be running.
2. vmkfstools -D /vmfs/volumes/path/to/file to dump information on the file into /var/log/vmkernel
3. less /var/log/vmkernel and scroll to the bottom, you will see output like below:
   
Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130: < START vmware-16.log >
Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368Nov 29 15:49:17 vm22 vmkernel: gen 66493, mode 1, owner 46c60a7c-94813bcf-4273-0017a44c7727 mtime 8781867]
Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Addr <4, 588, 7>, gen 20, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644
Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)len 23973, nb 1 tbz 0, zla 2, bs 65536
Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 132: < END vmware-16.log >

4. The owner of the lock is on the third line, the last part is all you need, in this case 0017a44c7727
5. esxcfg-info | grep -i ‘system uuid’ | awk -F ‘-’ ‘{print $NF}’ will display the system uuid of the esx server. You need to run the esxcfg-info command on each esx server in the cluster to discover the owner.
6. When you find the ESX server that matches the uuid owner, logon to that ESX server and run the command: ps -elf|grep vmname where vmname is the problem virtual machine. Example output below:
   4 S root 7570 1 0 65 -10 - 435 schedu Nov27 ? 00:00:02 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user/pool2 -@ pipe=/tmp/vmhsdaemon-0/vmxf7fb85ef5d8b3522;vm=f7fb85ef5d8b3522 /vmfs/volumes/470e25b6-37016b37-a2b3-001b78bedd4c/iu-lsps-vstest/iu-lsps-vstest.vmx0
7. Since there is a process running, pid 7570 in the example, you need to kill it by following steps 5-12 on Stopping a Virtual Machine.
8. Once the kill is complete the files should be released.

VMware has added a new command in VI3 to help when a user has a VM that has become unresponsive. Below are the progressives steps to go through to get the VM cleanly powered off. Whatever you do, DO NOT kill the pid for the VM from the Service Console, unless you have tried absolutely everything. Killing the pid from the Service Console may prevent the VM from restarting.

The following steps assume that the usual graceful shutdowns do not work from within the VM operating system or the Virtual Infrastructure Client. Commands are bold underscore type, with descriptions of path names and some example output is included.

1. Logon to the ESX host where the VM is running and become root.
2. vmware-cmd -l to list all the registered VMs.
3. vmware-cmd /path/copied/from/vmware-cmd getstate to get state of vm.
   If the state requires an answer: vmware-cmd /path/copied/from/vmware-cmd answer
   If no answer is needed: vmware-cmd /path/copied/from/vmware-cmd stop trysoft
   If “trysoft” does not work: vmware-cmd /path/copied/from/vmware-cmd stop hard
4. If the vmware-cmd does not help next up is to kill the master user world id
5. cat /proc/vmware/vm/*/names |grep vmname where vmname is the vm that is hung and find the value for vmid
6. less /proc/vmware/vm/vmid value/cpu/status where vmid value is the number from above
7. Scroll over to the right until you find the group field that shows vm.#### where the #### numbers after vm. will be the master user world id
8. /usr/lib/vmware/bin/vmkload_app -k 9 #### where #### is the master user world id. If successful you will get a WARNING message that a signal 9 is being sent
9. If vmkload_app does not help next up is to crash the vm with the vm-support -X command
10. vm-support -x to get the vmid
11. From a directory that has ample space vm-support -X #### where #### is the vmid
12. Answer all the questions with the default answers. The entire process takes about 10 minutes and creates an archive log that can be submitted to support. It will also crash the vm.