Cisco UCS – CIMC did not detect storage controller error

During a recent UCS firmware upgrade I had quite a few blades show up with the error “CIMC did not detect storage”. Within UCSM I could see that the blade had a critical alert. It initially started after I upgraded UCS Manager firmware as documents in a previous post I wrote about UCS Firmware Upgrades. I did some searching around to find what may be causing the issue and the best answer I could find was to from the Cisco community forums to disassociate the blade, decommission and reseat within the chassis. I later spoke to a Cisco engineer and he advised of the same steps but that it was also possible to do without reseating the blade. This also looks like its a problem when upgrading from 2.2(1d) to other versions of UCSM but I haven’t been able to validate if it’s only that version or if it also affects others.

The full error I saw was for code F1004 and for Controller 1 on server 2/1 is inoperable. Reason: CIMC did not detect storage

cimc1

Within UCSM I could see there was an issue with the Blade

cimc2

Before proceeding with the upgrade of the FIs, IOMs and Blades themselves I wanted to clear any alerts within UCSM, particularly critical alerts. The steps I followed to bring the blade back online were to go to the blade and select Server Maintenance
cimc3

I then selected Decomission and clicked Ok

cimc4

I clicked yes to verify I wanted to decommission the blade.

cimc5

Next I received a notification to advise that the decommission process has started successfully.

cimc6

Once the decommission had completed an alert appeared to advise that the blade needed some intervention.

cimc7

I right-clicked on the blade and selected Re-acknowledge Slot.

cimc8

I selected Yes to verify that it is what I wanted to do

cimc9

A notification then popped up to advise that the server had been re-acknowledged.

cimc10

A KVM session onto the blade showed that it went through a full detection process again

cimc11 cimc12

Once the blade had completed the discovery and booted into ESXi again it appeared within UCSM without any issues. The re-discover and reboot took just over 10 minutes. One thing to note is that the rediscovery will strip any labels and naming you’ve added to the blade previously so you’ll need to add them again separately.

cimc14

Once the CIMC error had been dealt with I was able to continue with the firmware upgrade

 

 


One comment


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s