Wednesday, July 8, 2015

Moving the VMware vCenter Server 4.x/5.x/6.0.x SQL database



A simple necessary task that isn’t so simple.  So the scenario plays out like this.  Customer is running vCenter, Update Manager and SRM databases on a SQL server that is running Windows 2003 with SQL 2005 – not extremely uncommon.  With the impending end of Windows 2003 looming we decide to migrate to Windows 2008 R2 and SQL 2008 R2.  

For the most part we followed KB 7960893 from VMware.

Now this KB will get you close, which is nice.  And what does close mean.  It means that we had databases up and running on the SQL server.  It means that we had ODBC connections that were connected and tested OK.  And then….It means that we couldn’t get the service to start – WTF.

So let’s look at the process and the final resolution.  First off it’s a pretty common task and I was doing this for a customer in their DR site, but it is something that everyone will run into at some time. 

Several steps in the process.

1.       Backup the databases on the old SQL server.  Simple enough go into SQL management studio right click the database and back it up.  Stop all of the VMware services on vCenter and SRM first.
2.       Copy the databases over to the new server – Really easy.
3.       Restore the databases to the new SQL server.
4.       Yeah this next step is a little harder.  Permissions.  We had to set the proper permissions.  The customer wanted to use a different service account and instead of having separate accounts for vCenter, UM, and SRM they wanted to use one account for all of them.  Not a huge deal, so whatever, let’s make it happen.  We used a local account in SQL with SQL server authentication.  We then granted dbo permissions to the vCenter, UM, and SRM databases.  You also have to give it dbo for the msdb database.  OK, all of this sounds pretty straightforward so far.
5.       Edit the vcdb.properties file to change the server name – Really easy.
6.       I didn’t update the SQL server agents just yet, because I didn’t care about performance statistics.
7.       Go to ODBC and ODBC32 on the vCenter server and change them to point to the new server and the correct database using the proper username and password.  Then test it.  BOOM, everything worked.  Start the vCenter and UM services up.  And…It failed.  WTF.

Troubleshooting Time

8.       So to do a little troubleshooting I looked in event viewer, got a whole bunch of nothing. 
9.       So what next?  I decided to start the vCenter service using the executable in standalone mode.  Open an administrative command prompt and the run vcdb –s.  Hmmm.  Interesting error.  Login failure and it was related to a DSN, which is telling me SQL login failure.  But the ODBC connections all tested positive.
10.   So I did a registry search and each of the 3 databases all have a section with a DB entry in it.  In that section I looked in there, and it still had the old entry for the name in there.  In the picture it is entry 2.  So I changed it.  DANG IT, the service still wouldn’t start.
11.   Well the line right below the username in the registry is the password hash for the old username, in the picture it is entry 3.  To change that password hash you have to go back to that administrative command prompt and run the vcdb –p command and then enter the new password twice.  And now.  BOOM the service starts.  WOO HOO.


12.   So now let’s start the Update Manager service.  It fails.  WTF.

13.   This was pretty easy.  I ran the Update Manager Utility and change the database in there and it started up fine after that.  Update Manager Utility is mentioned in KB 2011327, and it is located under Program Files (x86), because it is still a 32bit program, also have to update the ODBC32 as well.  So start the service and it worked.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2011327






14.   OK so now 2 of the 3 databases are connected and running.  Go to the SRM server and start that service up.  DANG IT, no go.  On to troubleshoot that.

15.   Went through several troubleshooting steps, and actually worked with VMware technical support, but since nothing was really set up in SRM we just did a full uninstall/reinstall.  It wasn’t pretty and if there had been any data on there I would have definitely taken the time to get it working, but…I didn’t so this was the easy way out.

So in a nutshell, it wasn’t as easy as I thought it would be.  But it is done, consulting is never quite as easy as it seems, but I still enjoy it.

Saturday, June 27, 2015

ESXi 6.0 false positive warning: Deprecated VMFS volume(s)

I found and interesting problem on my newly upgraded host today.  I received the warning:

Deprecated VMFS volume(s) found on the host. Please consider upgrading volume(s) to the latest version

Which looks like this:



Turns out simply restarting the management agents on the host will solve this issue. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).

This infomration comes from VMware KB Article 2109735.
Click here to access it:  http://kb.vmware.com/kb/2109735

Thursday, June 25, 2015

Unlocking and resetting the VMware vCenter Single Sign-On administrator password (vCenter 5.1, 5.5, 6.0)

I ran across this KB article from VMware to resolve this issue caused by a locked SSO password.  I am going to describe the solution specifically for vCenter 6.0 SSO.  For vCenter 5.1 or 5.1, you can go to the KB article link at the bottom of this page to review the other options.


Symptoms

  • You are unable to log in to the VMware vSphere Web Client using your single sign-on (SSO) administrator credentials.
  • Logging in to the vSphere Web Client using your SSO administrator credentials fails.
  • The password is incorrectly entered three times (by default).
  • You see the error:

    User account is locked. Please contact your administrator.
     

Cause

The SSO Administrator account is automatically locked after too many failed attempts, default is three.


Resolution

For VMware Platform Services Controller 6.0
  • Wait for 5 minutes. By default, the account lockout policy is set to unlock after 15 minutes. For more information on account lockout policies for the Platform Services Controller (PSC), see vCenter Server Password Requirements and Lockout Behavior in the vSphere Security Guide.
  • Unlock the account using another session that is still logged into the PSC server or using another user account with SSO administrator privileges. 

    To unlock an account using another session or using another user account with SSO administrator privileges:
    1. Click Home.
    2. Click Administration.
    3. Click Single Sign-On > Users and Groups.
    4. Click the Users tab.
    5. Right-click the affected user account, such as administrator@vsphere.local, and click Unlock.
  • In emergency situations or if the default policies are changed, you can also reset the password to unlock the account. 

    To reset the administrator@vsphere.local password on a Windows Platform Services Controller or vCenter Server with Embedded Platform Services Controller:
    1. Log in to the vCenter Server with a domain administrator account. If the Platform Services Controller is installed separate from the vCenter Server, log in to the Platform Services Controller server.
    2. Open an elevated command prompt. For more information, see Opening a command or shell prompt (1003892).
    3. Open the vdcadmintool service tool with this command:c:\> "%VMWARE_CIS_HOME%\vmdird\vdcadmintool.exe"
      This console loads:

      ===============================
      Please select:
      0. exit
      1. Test LDAP connectivity
      2. Force start replication cycle
      3. Reset account password
      4. Set log level and mask
      5. Set vmdir state
      ===============================
    4. Press 3 to enter the Reset account password option. 
    5. When prompted for the Account UPN, enter:
      Administrator@vSphere_Domain_Name.local

      By default, this is:

      Administrator@vSphere.local

      A new password is generated.
      Note: if you customized your vSphere Domain name, provide the customized domain name.
    6. Use the generated password to log in to the administrator@vSphere.local account.
    7. After the password is regenerated, log in to the vSphere Web Client and change the password.
To reset the administrator@vsphere.local password on the Platform Services Controller or vCenter Server with Embedded Platform Services Controller Appliance:
    1. Log in to the vCenter Server Appliance via SSH.
    2. Run this command to enable access the Bash shell:

      shell.set --enabled true
    3. Type shell and press Enter.
    4. Open the vdcadmintool service tool with this command:
      /usr/lib/vmware-vmdir/bin/vdcadmintoolThis console loads:
      ================================
      Please select:
      0. exit
      1. Test LDAP connectivity
      2. Force start replication cycle
      3. Reset account password
      4. Set log level and mask
      5. Set vmdir state
      ================================
    5. Press 3 to enter the Reset account password option.
    6. When prompted for the Account UPN, enter:
      Administrator@vSphere_Domain_Name.local

      By default, this is:

      Administrator@vSphere.local


      A new password is generated.

      Note
      : if you customized your vSphere Domain name, provide the customized domain name.
    7. Use the generated password to log in to the administrator@vSphere.local account.
    8. After the password is regenerated, log in to the vSphere Web Client and change the password.


KB 2034608: http://kb.vmware.com/kb/2034608