Last year I presented at the local Cisco DCUG to a warm and receptive audience about Cisco UCS Director being deployed on a global scale. At the time I was working for a global pharmaceutical company and following some organisational changes the requirements of the business and in turn IT changed to match. A key part of the changes focused on global standardisation of IT infrastructure to ensure 24 x 7 operational support. The best way to achieve that goal was to look at automation and orchestration. Cisco UCS Director was the tool chosen at the time. UCS Director is an absolute beast of a product and it reflects badly on Cisco as to how they have marketed and managed the product. It has potential to be the one stop shop for infrastructure management.
Create a global platform to enable physical and virtual automation based on standardised templates and processes.
- Drive standardisation across 14 global sites, reduce management overheads and complexities
- Put the company in a position to leverage follow the sun support for infrastructure to minimise out of hours support at each local site
- Provide a secure platform that could easily meet strict auditing guidelines
- Deliver a mechanism to allow end-users to quickly and easily request new virtual machines
- Streamline the request for infrastructure processes and remove existing bottlenecks
- Drive the business towards a Private Cloud architecture rather than individual silos
- Reduce licensing costs across the business for multiple existing automation and orchestration platforms.
- The ability to provide a cost model and service catalog and quickly inform projects on the estimated potential costs of their projects.
- Integration into the existing service management tool
- Integration into HP Quality Control for auditing and quality control purposes. This allowed for installation verification scripts to be completed.
After a recent upgrade to UCS Director 5.4 I noticed that my storage connections were showing a status of failed on the dashboard. I went to Administration -> Physical Accounts -> Physical Accounts. All of my NetApp controllers were offline.
I went to edit settings and re-entered my password to make sure that it had been picked up correctly.
All the settings were fine so I saved them and tested the connection to the controllers again.
The connection failed with the following error:
500 Connection has been shutdown: javax.net.ssl.SSLHandsakeException:
Server chose SSLv3, but that protocol version is not enabled or not supported by the client.
Cisco announced their release of UCS Director 5.4 back in November. As I’m currently running 5.3 and ran into an issue with a workflow Cisco support recommended upgrading to 5.4. I had a look over the Cisco UCS Director 5.4 Release Notes and there’s a new version of Java and the CentOS operating system are newer in the latest version. Due to this the upgrade procedure for 5.4 is different from previous version. In earlier versions it was possible to upload a patch via shelladmin and it would upgrade the software and database schema in place. 5.4 however requires new appliances to be deployed and a migration of database files etc. to be done between the 5.3 and 5.4 versions.
I really think that Cisco needs to look at using a HTML 5 console in the future as this upgrade path is overly complicated. Considering a lot of companies want you to be on the latest version when opening support calls, including Cisco, it would make sense for them to make it easier to perform the required upgrades.
The primary changes that have caused the modification to the upgrade path are:
- CentOS version 5.4 to version 6.6
- Java version 1.6 to version 1.8
Another thing to note is that version 5.54 requires 12GB RAM.
Cisco recommend standing up the new appliances beside your current UCS Director and Bare-Metal Appliances and performing a migration. In my case there’s a few firewall rule etc already been created for the existing environment so I wanted to keep the same IP addresses and machine names. I changed the IP addresses of the current appliances to be something else within the same subnet and gave the new appliances temporary names but the existing IP addresses. Once everything had been migrated and the changes confirmed I was able to rename the appliances to be the existing ones and removed the older appliances from the infrastructure. Before commencing the upgrade I also had a sold read over the UCS Director Upgrade 5.4 Guide and the UCS Director Bare-Metal Agent 5.4 Upgrade Guide
I had a problem a while ago where UCS Director crashed during a Metrocluster failover test. It was caused by the delay in the transfer of writable disks on the storage which in turn caused the VM kernel to panic and set the disk to read only. After that problem, and due to other restore issues within the infrastructure as well as not having a backup prior to the failover test I was left with a dead UCS Director appliance. It was essentially completely buggered as the Postgres database had become corrupt. Cisco support were unable to resolve the problem and it took a lot of playing around with NetApp snapshots to pull back a somewhat clean copy of the appliance from before the failover test. Really messy and I wouldn’t recommend it.
Since then I’ve been capturing weekly backups of the UCS Director database to a FTP server so I have a copy of the DB to restore should there be any problems with the appliance again. This script is not supported by Cisco so please be aware of that before implementing it. To set up the backup create a DB_BACKUP file in /usr/local/etc with the following:
# server login password localfile remote-dir
echo "open $1"
echo "user $2 $3"
upload_script $1 $2 $3 put $4 $5 | /usr/bin/ftp -i -n -p
if [ ! -f $BKFILE ]
echo "Backup failed. "
export NEWFILE="cuic_backup_`date '+%m-%d-%Y-%H-%M-%S'`.tar.gz"
export FTPLOGIN=< ftp user name >
export FTPPASS=<ftp password>
doftpput $FTPSERVER $FTPLOGIN $FTPPASS $BKFILE $NEWFILE
nohup /opt/infra/startInfraAll.sh &
Next you’ll need to edit your cron jobs on the appliance. You can use the crontab -e command to edit the schedule settings and enter:
1 2 * * 0 /usr/local/etc/DB_BACKUP > /dev/null 2>&1
And there you go, you now have a weekly scheduled backup of your UCS Director database.
Cisco have recently release a new version of their orchestration product UCS Director. The new release is version 5.3 and includes a raft of new features of which the majority are around improved reports and APIC support. Another new feature update is the support for NetApp OnTap 8.3. My primary reason for performing the upgrade is to leverage the reports and enhancements to workflow execution. It’s also been almost a year since the 5.1 installation was performed and I want to keep my systems up to date as much as possible. I’m currently running UCS Director 5.1.0 and Baremetal Agent 5.0.
Some of the new features in UCSD 5.3 are:
- Support for C880 M4 Server
- Support for Versa Stack and IBM Storwize
- Enhancements to EMC RecoverPoint
- Enhancements to VMware vSphere Support (VSAN Support)
- Enhancements to Application Controllers (Cisco APIC)
- Enhancements to workflow execution
- Enhancements to the script module
- Enhancements to UCSD REST APIs
- Enhancements to Managing NetApp Accounts (including support for OnTap 8.3)
- Enhancements to Cost Models and Chargeback features
- Changes to Report APIs
You can find more about the features in the release over on the Cisco UCS Director 5.3 Release Notes site.
There are two components to the release, UCS Director itself and the Baremetal Agent upgrade. The supported upgrade paths for both components are:
Cisco UCS Director
||Direct Upgrade Supported