How To: Cisco UCS Director – 5.3 to 5.4 Upgrade

Cisco announced their release of UCS Director 5.4 back in November. As I’m currently running 5.3 and ran into an issue with a workflow Cisco support recommended upgrading to 5.4. I had a look over the Cisco UCS Director 5.4 Release Notes and there’s a new version of Java and the CentOS operating system are newer in the latest version. Due to this the upgrade procedure for 5.4 is different from previous version. In earlier versions it was possible to upload a patch via shelladmin and it would upgrade the software and database schema in place. 5.4 however requires new appliances to be deployed and a migration of database files etc. to be done between the 5.3 and 5.4 versions.

I really think that Cisco needs to look at using a HTML 5 console in the future as this upgrade path is overly complicated. Considering a lot of companies want you to be on the latest version when opening support calls, including Cisco, it would make sense for them to make it easier to perform the required upgrades.

The primary changes that have caused the modification to the upgrade path are:

  • CentOS version 5.4 to version 6.6
  • Java version 1.6 to version 1.8

Another thing to note is that version 5.54 requires 12GB RAM.

Cisco recommend standing up  the new appliances beside your current UCS Director and Bare-Metal Appliances and performing a migration. In my case there’s a few firewall rule etc already been created for the existing environment so I wanted to keep the same IP addresses and machine names. I changed the IP addresses of the current appliances to be something else within the same subnet and gave the new appliances temporary names but the existing IP addresses. Once everything had been migrated and the changes confirmed I was able to rename the appliances to be the existing ones and removed the older appliances from the infrastructure. Before commencing the upgrade I also had a sold read over the UCS Director Upgrade 5.4 Guide and the UCS Director Bare-Metal Agent 5.4 Upgrade Guide

Software Download:

Download the software form the Cisco Download Software portal. Download the UCS Director 5.4 appliance and the BareMetal Agent files.

ucsd_bma_upgrade_12

Once it’s been downloaded extract the zip to access the OVF required for a new appliance.

 

UCS Director Upgrade Preparation Steps:

The first step that is always recommended is to take a backup of the database for UCS Director. This can be done via the console under the shelladmin login. Take a snapshot within vCenter also.

ucsd_bma_upgrade_1

ucsd_bma_upgrade_2

Open a console into vCenter and login.
ucsd_bma_upgrade_3

Log in using the shelladmin account.

ucsd_bma_upgrade_4

Select option 3 to stop all services and select Y at the prompt.

ucsd_bma_upgrade_5

Select option 2 to ensure that all the services have been stopped.

ucsd_bma_upgrade_7

Then exit the shelladmin login and select Configure Network and give the current UCSD appliances a new IP address so the current one can be re-used.

ucsd_bma_upgrade_9

Select n for a DHCP address and enter the new IP address. I’ve had to mask the actual addresses here but you can see that the final octet has changed from .54 to .235. Enter the DNS servers etc and commit the change of IP address. Make sure you can ping the appliance on its new IP address.

ucsd_bma_upgrade_11

 

UCS Director Deployment Steps:

In vCenter select File -> Deploy OVF Template

ucsd_upgrade_1

Select the extracted OVF file from the earlier download of the appliance.

ucsd_upgrade_2

Have a look over the requirements and click Next.

ucsd_upgrade_3

Accept the license and click Next

ucsd_upgrade_5

Select the Host and Cluster to deploy the new appliance to. Enter a name for the appliance and click Next.

ucsd_upgrade_6

Select the relevant resource pool and click Next

ucsd_upgrade_8

Select the datastore location for where you want to place the appliance.

ucsd_upgrade_9

Select which disk format you want to use. In this instance I left the disk format defaults. Click Next.

ucsd_upgrade_10

Select the network to assign to the appliance and click Next.

ucsd_upgrade_11

Enter the IP address address and put in the root and shelladmin passwords. The IP address used here is the one from the old appliance. It is the one that ends in .54. Once all the information has been entered click Next.

ucsd_upgrade_12

Lastly, verify all the information and click Finish

ucsd_upgrade_13

Once the appliance has finished installing power it on. You’ll see the database creating the required  schemas

ucsd_upgrade_16

Log into the new appliance using the root account created during the OVF deployment. As the IP address is the one used previously by the 5.3 version it is recommended to run the /opt/scripts/configStaticIP.sh. Fill in the details as required. In the screenshot below it shows I also ran the configStaticIP.sh on the old appliance to ensure that it also retained its static address. The same entries will be needed for the new appliance. This is a recommendation from the Cisco documentation.

ucsd_upgrade_14

Once the IP has been set log out of the console and you’ll see an Emerald Bay UCSD 5.4.0 title for the appliance as well as the required web console login.

ucsd_upgrade_17

Open a browser and log in using the admin account. It will appear as a blank system. Now that the new appliance is installed the files needed can be migrated/copied from the old appliance before the old one is decommissioned.

ucsd_upgrade_18

 

Migrate Data between UCS Director versions:

Log back into the new appliance using the root account. Run the script  /opt/infra/migration/performMigration.sh to migrate the database and configuration files from the 5.3 appliance to the 5.4 release. The script backs up the database on the old appliance and restores it to the new one. It also executes the scripts required to upgrade the database.

ucsd_upgrade_19

When prompted, provide the IP address of the old appliance.

ucsd_upgrade_20The services on the old appliance will be requested to be shut down during the migration process.

ucsd_upgrade_21

Once the migration process has been completed you’ll be prompted to start the services using shelladmin on the new appliance.

ucsd_upgrade_22

Select option 4 to restart all of the services.

ucsd_upgrade_24

Run option 2 to verify that the services have started.

ucsd_upgrade_25

Once the migration script has completed you may need to copy some files manually from the old appliance. You’ll need to review the files to move. Enable the SSHD service on both the old and new UCS Director appliances. Check the old version against the new version to see if any of the following files need to be copied. If not, ignore them. Open WinSCP sessions to both and copy these files:

  • /etc/ntp.conf
  • mysql tuning in /etc/my.cnf

In my case the ntp.conf was different but I just edited the new file to point to internal NTP servers

Changes in the following properties files:

  • /opt/infra/client/service.properties
  • /opt/infra/inframgr/service.properties
  • /opt/infra/idaccessmgr/service.properties
  • /opt/infra/web_cloudmgr/apache-tomcat/bin/service.properties
  • /opt/infra/controller/service.properties
  • /opt/infra/eventmgr/service.properties
  • /opt/infra/networkServices/service.properties
  • /opt/infra/usageMgr/service.properties
  • /opt/infra/inframgr/hyperv.properties

Migrate the contents from the/opt/infra/uploads folder. This folder contains any files and OVFs uploaded from the Cisco UCS Director user interface. To migrate the files I ran a WinSCP session to download the required files locally and upload them to the new appliance via another WinSCP session. When prompted to overwrite existing files click Yes.

ucsd_upgrade_28

ucsd_upgrade_29

ucsd_upgrade_30

At this point I restarted the entire appliance to just start with a fresh slate. While doing this I powered down the old appliance so there were no potential clashes. I recommend to rename the old appliances to _temp and modify the VM names of the deployed VMs to the existing VM names. After the reboot log into UCS Director via the web console.

ucsd_upgrade_31

Click About and verify the version you are running.

ucsd_upgrade_32

You have now successfully migrated to 5.4.0. The last thing to do is to configure a weekly backup of the database. Once the script has been configured just run the job manually and ensure that the backup reaches your FTP server.

 

BareMetal Agent Upgrade Steps:

Once the main UCSD is done log into UCSD BMA using the root account and run the /opt/infra/configureInterface.sh script for eth0 so that its a different IP address. The new appliance being deployed will take over the current IP address assigned to BMA.

ucsd_bma_upgrade_13

Then run the command ./stopInfraAll.sh to shut down all services.

ucsd_bma_upgrade_14

Next we can begin the deployment of the new appliance. Go to File -> Deploy OVF template

ucsd_bma_upgrade_15

Select the BMA OVF that was downloaded earlier.

ucsd_bma_upgrade_16Click Next to continue the deployment process

ucsd_bma_upgrade_17

Review the details of the appliance and click Next

ucsd_bma_upgrade_18

Accept the license and click Next

ucsd_bma_upgrade_19

Enter a name for the appliance and click Next

ucsd_bma_upgrade_20

Select the resource pool to deploy the appliance to.

ucsd_bma_upgrade_21

Select the relevant datastore and click Next.

ucsd_bma_upgrade_22

Check the disk format and select whichever format fits to your environment.

ucsd_bma_upgrade_23

At this point there’s a need to select the relevant networks. One is required for management and one is required for PXE boot. Click Next.

ucsd_bma_upgrade_25

Enter details for the passwords for the accounts and the IP address of the old appliance. Click Next to continue.

ucsd_bma_upgrade_26

Review the details of the appliance and click Finish to complete the deployment.

ucsd_bma_upgrade_27

Open the console session and verify the correct IP address is assigned.

ucsd_bma_upgrade_28

Open a web browser and verify that you can at least see the 5.4.0.0 version showing up.

ucsd_bma_upgrade_29

Log into the new appliance with the root account.

ucsd_bma_upgrade_30Once logged in run /opt/infra/migration/performBMAMigration.sh script to migrate the images, templates, and configuration files from the 5.3 BMA appliance to the 5.4 BMA appliance.

ucsd_bma_upgrade_31The migration script will then go and copy all the required files.

ucsd_bma_upgrade_32Once it has been completed log in toCisco UCS Director as an admin and verify that all the images from the 5.3 BMA has been migrated successfully. You may need to refresh the view to make sure it shows up correctly.

As with the main UCS Director appliance you’ll need to copy some files with WinSCP. Copy the following files from the old appliance using WinSCP

  • /etc/resolv.conf
  • /etc/hosts
  • mysql tuning in /etc/my.cnf
  • /opt/infra/networkServices/service.properties

Reboot the new appliance and shutdown the old version

(Optional)

For any Windows images migrated from the 5.3 BMA update the WinPE file with the new IP address of the 5.4 BMA. You do not need to edit WinPE if the Baremetal Agent IP address was not changed.

If you use a separate interface for PXE Traffic you’ll need to configure the PXE interface IP using the Cisco UCS Director UI if you haven’t reused the 5.3 BMA appliance PXE IP address. Otherwise it will be fine.

Using the Configure Interface option, select the BMA account and copy the /etc/dhcpd.conf file from the 5.3 BMA appliance and reconfigure it based on the new PXE Interface IP address.

Finalise Deployment:

The last steps involve renaming any of the appliances as needed and then once the new appliance has been confirmed as working correctly delete the old appliances from the environment. And that’s it, everything has been updated. It’s a real pain to have to deploy new appliances and hopefully this will be addressed in upcoming versions.

Note: I just noticed as I was writing this post that a new 5.4.2 patch has been released. I’ll have to start looking at upgrading that patch also.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s