Wednesday, December 21, 2011

Restoring from an alternate library in CommVault

CommVault is a great product that allows you to perform backups at a primary location, and then send a deduplicated copy to a DR location for recovery. But what if you need to perform a test recovery at the DR location, how do you tell the restore which library to restore from?

Step 1: Identify the number of the copy

Your basic setup should include a Storage Policy with 2 copies, a primary and a (at least 1) secondary. The secondary is updated either on schedule or on demand.
To view the copy number

  • Browse to Policies | Storage Policies
  • Right-click your policy and select Properties
  • On the Copy Precedence tab, note the Precedence number associated with the target Copy Name

Step 2: Browse for backup data

  • Browse to Client Computers | | |
  • Right-click the subclient and select Browse Backup Data
  • Specify a time frame if needed
  • Change Use MediaAgent to the MA in the DR location
  • Click Advanced
  • On the Advanced Browse Options, change the Copy Precedence to the precedence number previously identified
  • Click OK, OK

Finish your restore as normal

Monday, December 12, 2011

Decrypting NT passwords

If you have been in the tech realm for long, you realize there is a need to recover forgotten passwords. Normally this is difficult (as it should be), but I just found a shortcut.
The website, has an online database of NTLM hashes and their accompanying passwords. Once you have the hash (via pwdump or other tools), just copy and past the hash into the site and viola!

Standard warnings apply. Many countries consider unauthorized access to computers illegal, so only use this process if you have a legitimate reason.

Tuesday, December 06, 2011

Working with an Iomega StorCenter ix2-200

I just got a 4 TB Iomega StorCenter ix2-200 for use with VMware. The idea is that we can send it to customer locations, copy backups/VMDKs on it, and then ship it back. Previously we had done this with USB drives, but connecting a USB drive to a server is often problematic.

Basic setup for VMware NFS datastores:

  • Don't enable security
    • If you do, you have to work through enabling networks to see the individual shares
  • Enable NFS under Protocols
  • On the Shares page, under each share there is now a NFS item with the export name

When I was setting this up the first time I ran into some issues. Due to the posts and comments at, I was able to use SSH to get a better understanding of the configuration and review logs. To enable SSH,

  1. Go to http://IOMEGAURL/diagnostics.html (a hidden page)
  2. Under Support Access, click Allow remote access for support
  3. SSH to the device
    1. User: root
    2. Password: sohoadmin
    3. If you enabled security on the device, the password is soho + yourpassword

Problems making NFS work.
When trying to mount the NFS partition, my ESX server kept erroring out. The vmkernel log reported "WARNING: NFS: 946: MOUNT RPC failed with RPC status 13 (RPC was aborted due to timeout) trying to mount". The logs on the Iomega showed the mount request was being authenticated, but the mount never succeeded.
A little more googling and I found It turns out the Iomega attempts to resolve the DNS address of the requesting system. Normally this wouldnt be a problem, but I was on a network that didnt have access to a DNS server. Adding a hosts entry fixed the problem

Wednesday, November 16, 2011

Quick VMotion in VMware

I ran into an issue today where I was unable to VMotion a VM due to host issues. While troubleshooting, I remembered how Microsoft first dealt with its inability to support VMotion - Quick VMotion.
The idea is that instead of migrating the VM while online, pause the VM, then migrate it, then restart the VM on the target. Though this causes some outage, it allows movement of VMs will less impact than actually powering off the VM (or allowing HA to recover a failed host)

Quick VMotion powershell code:

$vm = get-vm vmName
$Myhost = get-vmhost hostName
suspend-vm -VM $vm -Confirm:$False
move-vm  -VM $vm -destination $Myhost -Confirm:$False
start-vm -VM $vm -Confirm:$False

Monday, November 14, 2011

VCP5 - Configure network security policies

There are 3 basic network security policies: Promiscuous mode, MAC address changes, and Forged transmits. These policies allow for customization of the networking layer to allow network sniffing, clustering, and load-balancing

Changing policies on a Distributed Switch

  1. Open vSphere client and select Networking
  2. Right-click the Port Group and select Edit Settings
  3. Under Policies | Security, change the policies as needed
Changing policies on a Standard Switch
  1. Open vSphere client and select Hosts and Clusters
  2. Select a host and click the Configuration tab
  3. Under Hardware | Networking, click Properties for a Virtual Switch
  4. Select the switch or a port group and click Edit
  5. Under Security, change the policies as needed

VCP5 - Enable Lockdown Mode

Lockdown mode blocks direct access to the ESXi host and forces all configuration to be done via the Virtual Center. This can be used to secure your VMware environment and ensure security protocols are followed.

Enabling Lockdown Mode via DCUI

  1. Log onto the ESXi host console as root
  2. Select Configure Lockdown Mode
  3. Use the spacebar to check the box and hit Enter to accept
Enabling Lockdown Mode via vSphere
  1. Open the vSphere console and select Hosts and Clusters
  2. Choose a host and click the Configuration tab
  3. Select Software | Security Profile
  4. Next to Lockdown Mode, click Edit
  5. Check the box and click OK

Enable/Configure/Disable services in the ESXi firewall

Unlike prior versions of ESX, v5 uses an XML file to describe the firewall configurations. These files are stored in the /etc/vmware/firewall directory and are automatically loaded. Firewall rules should be installed as part of a VIB package.
More information on creating rules:

Once installed, the firewall rules and service is managed via the vSphere client.

  1. Open vSphere and select Hosts and Clusters
  2. Select a host and click the Configuration tab
  3. Select Software | Security Profile
  4. Next to Firewall, click Properties to enable or disable specific rules

VCP5 - Describe how permissions are applied and inherited in vCenter Server

Permissions in vSphere are applied on managed entities (datacenters, folders, VMs, etc...) Permissions are propagated from the managed entity to child identities.

  • Permissions applied at the same level are summed to include both sets of permission. 
  • Unlike other permission inheritance schemes (which sum permissions at different levels), permissions applied at lower levels over-ride inherited permissions. 
  • Permissions applied directly to a user over-ride group and inherited permissions

VCP5 - Identify common vCenter Server privileges and roles

There are 9 default roles in vSphere - 6 sample roles, and 3 standard.

Default Roles
Role Type
Description of User Capabilities
No Access
Cannot view or change the assigned object.
vSphere Client tabs associated with an object appear without content.
Can be used to revoke permissions that would otherwise be propagated to an object from a parent object.
Available in ESXi and vCenter Server.
Read Only
View the state and details about the object.
View all the tab panels in the vSphere Client except the Console tab. Cannot perform any actions through the menus and toolbars.
Available on ESXi and vCenter Server.
All privileges for all objects.
Add, remove, and set access rights and privileges for all thevCenter Server users and all the virtual objects in the vSphere environment.
Available in ESXi and vCenter Server.

Users who are in the Active Directory group ESX Admins are automatically assigned the Administrator role.
Virtual Machine Power User
A set of privileges to allow the user to interact with and make hardware changes to virtual machines, as well as perform snapshot operations.
Privileges granted include:

All privileges for the scheduled task privileges group.
Selected privileges for global items, datastore, and virtual machine privileges groups.
No privileges for folder, datacenter, network, host, resource, alarms, sessions, performance, and permissions privileges groups.
Usually granted on a folder that contains virtual machines or on individual virtual machines.
Available on vCenter Server.
Virtual Machine User
A set of privileges to allow the user to interact with a virtual machine’s console, insert media, and perform power operations. Does not grant privileges to make virtual hardware changes to the virtual machine.
Privileges granted include:

All privileges for the scheduled tasks privileges group.
Selected privileges for the global items and virtual machine privileges groups.
No privileges for the folder, datacenter, datastore, network, host, resource, alarms, sessions, performance, and permissions privileges groups.
Usually granted on a folder that contains virtual machines or on individual virtual machines.
Available on vCenter Server.
Resource Pool Administrator
A set of privileges to allow the user to create child resource pools and modify the configuration of the children, but not to modify the resource configuration of the pool or cluster on which the role was granted. Also allows the user to grant permissions to child resource pools, and assign virtual machines to the parent or child resource pools.
Privileges granted include:

All privileges for folder, virtual machine, alarms, and scheduled task privileges groups.
Selected privileges for resource and permissions privileges groups.
No privileges for datacenter, network, host, sessions, or performance privileges groups.
Additional privileges must be granted on virtual machines and datastores to allow provisioning of new virtual machines.
Usually granted on a cluster or resource pool.
Available on vCenter Server.
Datastore Consumer
A set of privileges to allow the user to consume space on the datastores on which this role is granted. To perform a space-consuming operation, such as creating a virtual disk or taking a snapshot, the user must also have the appropriate virtual machine privileges granted for these operations.
Usually granted on a datastore or a folder of datastores.
This role is available on vCenter Server.
Network Consumer
A set of privileges to allow the user to assign virtual machines or hosts to networks, if the appropriate permissions for the assignment are also granted on the virtual machines or hosts.
Usually granted on a network or folder of networks.
Available on vCenter Server.

VCP5 - Upgrade an ESXi Host using vCenter Update Manager

Upgrading ESXi host via VUM is an elegant process.

  1. Install VUM
  2. Open vSphere and open Update Manager
  3. Select the ESXi Images tab
  4. Click Import ESXi Image
    1. Walk through process to import image
  5. Open Hosts and Clusters and select the Update Manager tab
  6. Click Scan to refresh compliance
  7. Click Remediate to begin updating the hosts

VCP5 - Upgrade VMware Tools / Virtual Machine hardware

This is a two-fer, two objectives listed in 1 post (since they are generally simple tasks).

Upgrading tools and hardware manually:

  1. In vSphere, open VMs and Templates and select the VM
  2. Right-click the VM and select Guest | Install/Upgrade VMware Tools
  3. Select Automatic Tools Upgrade and click OK
    1. This will upgrade the tools and reboot the VM
  4. When finished upgrading the tools, shut down the VM
  5. Right-click the VM and select Upgrade Virtual Hardware
  6. Power on the VM

VCP5 - Upgrade from VMFS3 to VMFS5

Upgrading from VMFS3 to VMFS5 is seamless and can be performed online with active VMs. Note however, that there are drawbacks to upgrading to VMFS5:

  • Datastores are accessible to ESXi 5 hosts only
  • The upgrade process will not take advantage of some of the VMFS5 improvements (such as universal block size)
Upgrading to VMFS5
  1. In VSphere, browse to Datastores and Datastore Clusters
  2. Select the datastore and click the Configuration tab
  3. On the right, click Upgrade to VMFS-5
  4. Click OK to start the upgrade

VCP5 - Upgrade a vNetwork Distributed Switch

There are 3 versions of vNetwork Distribute Switches available: 4.0, 4.1, and 5.0. Each version provides new functionality, but also limits the interop with older versions

Upgrade Distributed Switch

  1. In vSphere, browse to Networking
  2. Select the switch and on the Summary tab, click Upgrade
  3. Select the upgrade version and click Next
  4. Confirm no hosts report as incompatible, click Next
  5. Click Finish

Version Compatibility / Features
4.0N/AESX 4.0 and later
4.1Load-Based Teaming
Network I/O Control
ESX 4.1 and later
5.0User-defined network resource pools
Port Mirroring
ESX 5.0 and later

VCP5 - Identify steps required to upgrade a vSphere implementation

Upgrading is generally a top-down process starting with the Virtual Center Server, then hosts, then VMs. This ensures that everything is properly supported during the upgrade process.

  1. Make sure vSphere plug-ins are supported in vSphere 5
    1. Upgrade or remove them if necessary
  2. Confirm the environment meets the system requirements
  3. Upgrade vCenter Server
  4. Upgrade VMware Update Manager
  5. Upgrade ESXi hosts
  6. Reconfigure Licensing
  7. Upgrade VMs
Note that this process will differ somewhat if using VMware View.

VCP5 - Identify upgrade requirements for ESXi hosts

ESXi 5 upgrade requirements are above and beyond the standard ESXi installation requirements - they determine the state the ESX host must be in prior to upgrade.

Minimum ESX version: 4.0 ESX or ESXi

  • An ESX 3 host can be upgraded to 4, and then upgraded to 5
Supported ESX upgrade path tools:
  • vSphere Update Manager
  • Interactive upgrade via DVD
  • Scripted upgrade

Non-supported configurations:
  • ESX 4 host with an incompatible disk partition
    • Often result from ESX 3 host upgraded to ESX 4
  • ESX 4 host with missing, inaccessible, or corrupt Service Console VMDK

NOTE: By definition, "upgrade" refers to updating the current system. Pay note that this is different than a new install, such as using vSphere Auto Deploy.

VCP5 - License an ESXi host

Once licenses are installed in Virtual Center, they need to be assigned to hosts

  1. Open vCenter Server and go to Home | Administration | Licensing
  2. Click Manage vSphere Licenses
  3. On the Add License Keys screen, click Next
  4. On the Assign Licenses screen, select the ESXi host and pair it with the appropriate license
  5. Click Next, Next, Finish

VCP5 - Enable/Size/Disable memory compression cache

Memory compression uses standard compression techniques (like ZIP or RAR) to extend the use of RAM instead of swapping to disk. Compression is enabled by default and improves performance when over-committing memory by limiting the amount of swap-to-disk, a process that can take considerable time.

Enabling/Disabling Memory Compression

  1. Open the Virtual Center and select Hosts and Clusters
  2. Select a host and click the Configuration tab
  3. Select Software | Advanced Settings
  4. Select Mem.MemZipEnable, 1 for on, 0 for off
Configuring Memory Compression
  1. Open the Virtual Center and select Hosts and Clusters
  2. Select a host and click the Configuration tab
  3. Select Software | Advanced Settings
  4. Select Mem.MemZipMaxPct
    1. This defaults at 10% of the RAM for caching and can be altered from 5% to 100%

VCP5 - Enable/Configure/Disable hyperthreading

Hyperthreading has historically been a troublesome issue. The precursor to multiple cores, the idea is to run multiple commands through the same processor at the same time - thereby simulating multiple cores. This worked well with some applications, but caused nightmares for others. The good news is that Hyperthreading has been rebuilt with the latest generation of processors, and specifically tuned for VMware.

To enable/disable Hyperthreading in ESXi

  1. Configure the system BIOS to enable/disable hyperthreading
  2. Open Virtual Center and select Hosts and Clusters
  3. Select the host and click the Configuration tab
  4. Select Hardware | Processors - here you can view the current Hyperthreading settings
  5. Click Properties to change the Hyperthreading

Once enabled, Hyperthreading is defaulted to be used by all VMs in a fashion similar to multi-core processors. This sharing can be configured on a per-VM basis and be set to ANY (default), NONE (no hyperthreading), and Internal (hyperthreading only with itself). These configurations exist to stabilize individual VMs, but should be used with care.
Configuring Hyperthreading in ESXi
  1. Open Virtual Center and select VMs and Templates
  2. Select the VM and click Edit Settings
  3. On the Resources tab, select Advanced CPU to change the settings

VCP5 - Configure DNS and Routing on an ESXi Host

Although this objective sounds complicated, its simpler than you think, and you may already have completed it without realizing. This objective basically means, setting the default gateway, and DNS servers/suffixes - a step most likely completed during installation

Configuring DNS and Routing via DCUI

  1. Log into the DCUI and select Configure Management Network
  2. Select IP Configuration and enter the Default Gateway
    1. NOTE: If you are using DHCP to assign the address, the Gateway should be assigned via DHCP as well
  3. Select DNS Configuration to configure the DNS servers and local hostname
  4. Select Custom DNS Suffixes to customize the FQDN
Configuring DNS and Routing via Virtual Center
  1. Open the Virtual Center and browse to Hosts and Clusters
  2. Select the ESXi host
  3. Select the Configuration tab and Software | DNS and Routing
  4. Click Properties to configure the name, DNS servers, FQDN, and default gateway

VCP5 - Configure NTP on an ESXi Host

NTP configuration for ESXi hosts can be configured via multiple methods: command line, Virtual Center, Host Profile, or editing configurations directly (doesnt appear to be directly supported in ESX 5). The easiest by far is configuring the system via Virtual Center

Configuring NTP via Virtual Center

  1. Open Virtual Center and select the Hosts and Clusters view
  2. Select the host and click the Configuration tab
  3. Select Software | Time Configuration
  4. On the right, click Properties to edit the date, time, and NTP settings


Friday, October 28, 2011

Vyatta virtual router

Often times when setting up a test lab, I will use a Linux system as a router/firewall. While this works, it requires somewhat intimate knowledge of Linux setup and networking - something not everyone is going to have. I found the Vyatta platform is a nice hybrid between a Linux server and a network appliance. It allows for routing, firewalling, DHCP/DNS, VPN, and more.

The easiest method to get started is to download the latest OVF file from 

VCP5 - Perform an interactive installation of ESXi

Assuming your hardware meets the VMware HCL and minimum physical requirements, interactive installation is fairly simple. Things will get complicated if your host doesn't properly support the virtualization layer or if special drivers are needed

Installation Process:

  1. Insert the CD/DVD/ISO and boot the server
  2. On the ESXi Boot Menu, select ESXi-5.0.0-469512-standard Installer (version number may differ)
  3. On the Welcome to the VMware ESXi 5.0.0 Installation screen, hit Enter
  4. Accept the EULA and press F11
  5. Select the disk to install and press Enter
  6. Select the keyboard layout and press Enter
  7. Type and confirm the root password, press Enter
  8. Press F11 to confirm the installation
  9. When the installation is finished, remove the install media and press Enter to reboot

VCP5 - Determine use case for vSphere Client and Web Client

There are 3 client interfaces available for vSphere
vSphere Client

  • Primary method of managing vSphere
  • Must be installed on a Windows client
  • Provides all vSphere management functions

vSphere Web Client

  • Provides a subset of the vSphere management functions
  • Well suited for most day-to-day administration tasks
  • Accessible via most web browsers (requires Adobe Flash)

vSphere Command Line Interface

  • Uses the vSphere Power-CLI to manage most aspects of vSphere
  • Requires Microsoft PowerShell to function

More detail is available at:

VCP5 - Determine availability requirements for a vCenter Server in a given vSphere implementation

The first question to ask when looking into the availability requirements for vCenter Server is: How much do I trust my infrastructure? I know this is kind of a silly question, but if you think your server is going to fail, then you shouldn't be using that server.

Probably the next few questions to ask are: How much data can I lose? How long can I be down? Where will I be recovering the system? These are basic (but critical) Business Continuance / Disaster Recovery (BCDR) questions that are likely a part of your larger server ecosystem. For instance, losing a single server is different than losing your entire datacenter. Additionally, taking 10 minutes or 10 hours to recover the system can majorly impact your ability to manage your virtual environment.

OK - Off the soap box

There are 2 primary components to protect with vCenter: the Database and the vCenter Server. The database is the brains of the operation - it knows where all your VMs are, what kind of licenses you have, all your perf data and so on. The vCenter Server is the brawn - it does all the lifting, organizing, and collecting. Ideally you would make both of these highly available, but if you can only have 1, go for the database.

Officially supported vCenter HA options:

  • Virtualize the vCenter Server and use VMware HA/DRS to maintain the system
  • Use VMware Heartbeat to maintain an active/passive configuration
  • Third Party Clustering such as MSCS
    • This provides an active/passive configuration similar in idea to VMware Heartbeat

Non-supported vCenter HA options:

  • Database replication such as log shipping or transactional replication
    • This copies the database to a second location. The vCenter Server could be pointed to the copy in case of an outage, or an offline vCenter instance could be preconfigured to use the copy and powered on during the outage
    • This can protect against server and site failure
  • Server Replication such as DoubleTake
    • This clones the entire vCenter Server and database to a second location and continually updates the copy with transactional changes (with some lag)
    • During a failure, the copy can be configured to automatically power on and manage the virtual infrastructure
    • This can protect against server and site failure

VMware KB describing the supported vCenter Server high availability options:

Thursday, October 27, 2011

VCP5 - Deploy the vCenter Appliance

vSphere 5 now provides you with an alternative to a Windows based vCenter Server known as the vCenter Server Appliance. Like its name implies, this is a VM appliance that provides many of the vCenter functions without requiring a Windows system, or the accompanying license.
NOTE: This is very helpful in many cases, and even provides additional features, but does not provide all the  functions a traditional vCenter Server can.

Installation process
Go to
Click Support & Downloads | VMware vSphere
Click VMware vCenter Server 5.0 and Modules
Download the following 3 files
  • VMware vCenter Server 5.0 Appliance - OVF File
  • VMware vCenter Server 5.0 Appliance - System Disk
  • VMware vCenter Server 5.0 Appliance - Data Disk
Connect vSphere client to ESX host
File | Deploy OVF Template
Enter the OVF file location, Next
Walk through the wizard entering the location, datastore and network

Configuration process
Power on the appliance and open the console
Using the arrow keys, select Configure Network and setup the networking
Once networking is configured, open a browser to https://:5480
Login as root, password vmware
Accept the EULA and wait for services to start
Once the services are started, click the Network and Authentication tabs to further configure DNS and AD/NIS integration
Select vCenter Server | Database
Change Database Type to embedded and click Test Settings
When the test completes, click Save Settings
When the settings are committed, click System | Reboot

Open the vSphere client and select the vCenter Server Appliance name/IP
Log in as root/vmware or the AD/NIS credentials configured previously
Or, open a browser to https://:9443/vsphere-client

Base documentation at

VCP5 - License vCenter Server

Licensing is a simple task (assuming you have licenses) and almost isn't worth documenting.

  1. Open vCenter Server and go to Home | Administration | Licensing
  2. Click Manage vSphere Licenses
  3. Enter the license key(s) and click Add License Keys
  4. Click Next, assign the license to the ESX or vCenter Server(s)
  5. Click Next, Next, Finish

Wednesday, October 26, 2011

VCP5 - Install/Remove, Enable/Disable vSphere Client plug-ins

The vSphere Client can be extended by anyone with a need to do so. Extensions can be simple, such as reporting, or extensive, such as automation.

Installing plug-ins
To install VMware plug-ins

  1. Open the vSphere client and connect to a vCenter server
  2. On the menu bar, go to Plug-ins | Manage Plug-ins
  3. Under Available Plug-ins, select the plug-in of choice and select Download and Install

Third-party plugins traditionally are distributed as an executable, and any specific instructions are included with the executable file. Once installed, the plug-in can be enabled or additional confiugration can take place

Removing plug-ins
To remove plug-ins, use Add/Remove programs from the control panel. Example from a Windows 7 desktop

  1. Go to Start | Control Panel
  2. Select Programs and Features
    1. This list may take a while to fully populate
  3. Select the plug-in to remove and click Uninstall

Enable/Disable plug-ins
To enable or disable plug-ins

  1. Open the vSphere client and connect to a vCenter server
  2. On the menu bar, go to Plug-ins | Manage Plug-ins
  3. Under Installed Plug-ins, right-click the plug-in of choice and select Enable or Disable

Official directions can be found at:

VCP5 - Install additional vCenter Server components

In addition to the vCenter Server, there are several additional components that can be installed. The good news is that all of these components are included in the VMware VIM Setup zip file.

  • vSphere Client
    • A Windows client used to connect to the vCenter server and stand-alone ESX hosts
    • Can be installed from the zip file, or downloaded by pointing a browser to the vCenter server
  • VMware vSphere Web Client
    • A web based version of the vSphere Client. Only provides a limited set of options
  • VMware vSphere Update Manager
    • Allows centralize patch and update management of all VMware components
  • VMware ESXi Dump Collector
    • Provides a centralized location for ESXi memory dumps in case of a system failure
  • VMware Syslog Collector
    • Provides a centralized logging service for ESXi servers
  • VMware Auto Deploy
    • Automates provisioning of ESXi hosts at boot time, requiring no disks in the ESXi servers
  • VMware vSphere Authentication Proxy
    • Enables ESXi hosts to join an AD domain without requiring AD credentials

Most of the setup for these components follows the traditional "next, next, next" installation process. Specific details can be found at