Nutanix LCM Upgrade Process Failed Trouble-shooting

Nutanix Prism Central having LCM failure issue during upgrading the software through LCM e.g Calm, Karbon, Epsilon, NCC, PC etc. via Life Cycle Management: LCM generated log with error message:

Nutanix LCM upgradation operation failed. Reason: Lcm prechecks detected 2 issues that would cause upgrade failures. Check ‘test_url_connectivity’ failed with ‘Failure reason: URL ‘http://download.nutanix.com/lcm/2.0’ is either incorrect or not reachable from ip x.x.x.x.
Please verify that the URL is correct and ensure it is accessible from all CVMs.’ Check ‘test_catalog_workflows’ failed with ‘Failure reason: Failed to create catalog item. Refer to KB 4584 for details on precheck failures.

Nutanix LCM framwork works with 70% faster robust up-gradation of software and hardware firmware with no downtime with Nutanix HCI appliance.

Nutanix software or firmware up-gradation failure via LCM might failed due many reasons and need to troubleshoot one by one.

How Nutanix LCM Works

Before to start the Nutanix LCM failure issue troubleshooting need to understand the Nutanix LCM: Life Cycle Management architecture and how Nutanix LCM works ?.

I hope you have read out the above post about How Nutanix LCM works, now you have better understanding of Nutanix LCM functionality.

Now we can proceed to Nutanix LCM troubleshooting.

LCM Failure Troubleshooting

lets explore the reasons and troubleshoot the failure process of software component(s) and hardware firmware up-gradation getting failed via LCM framework.

As per my knowledge i had found following reasons behind the LCM upgrading task failure issue.

• Internet is not allowed on Prism Central / Element
• DNS Server is not Configured
• Date-time & Timezone Mismatch
• Firewall Blocking Traffic
• Proxy Configuration Issue
• Update LCM and Foundation version

Lets take deep dive to each Nutanix LCM: Life Cycle Management failure issue below.

Internet is not allowed on Prism Central / Element

Nutanix Prism central / Element is required internet access to communicate with nutanix network to check the latest software and firmware updates.

Allow URL: download.nutanix.com access to Nutanix Prism and Prism central on firewall to download the software updates.

Note : Login to any CVM and Prism Central VM and ping to dowload.nutanix.com url to check the internet connectivity.

 cvm$ ping -c 4 download.nutanix.com 
Pinging drbtc55jmu32c.cloudfront.net [64:ff9b::6356:3782] with 32 bytes of data:
 Reply from 64:ff9b::6356:3782: time=115ms
 Reply from 64:ff9b::6356:3782: time=104ms
 Reply from 64:ff9b::6356:3782: time=109ms
 Reply from 64:ff9b::6356:3782: time=105ms
 Ping statistics for 64:ff9b::6356:3782:
     Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
 Approximate round trip times in milli-seconds:
     Minimum = 104ms, Maximum = 115ms, Average = 108ms

DNS Server is not Configured

Domain name server ( DNS ) is also a major configuration to access the internet. So cross-check the correct and reachable DNS IP address entry in Nutanix Prism.

Login to Prism / Central > Gear icon > Name Server

Date-time & Timezone Mismatch

There might be the issue of date-time and timezone mismatch between Nutanix CVMs and Prism Central Virtual Machine ( PCVM ).
It is recommended to keep your all Nutanix software and services on same date-time and timezone.

Nutanix Controller VM: CVM , Prism and Prism central must be on same date-time and timezone take update through same NTP server(s).

Login to Prism / Central > Gear icon > NTP Server

Firewall Blocking Traffic

There is one more possibility that your hardware firewall is blocking In-bound and Out-bound Nutanix domain or sub-domain traffic URL download.nutanix.com or *.nutanix.com and Network port number 80 and 443.

Allow above mentioned URL and Network ports on firewall.

Proxy Configuration Issue

if you are using proxy server in your Nutanix environment to access internet then you have to configure the proxy on Prism and Prism central as well to allow internet access on Nutanix Prism and Prism central to get software updates through LCM.

Login to Prism / Central > Gear icon > HTTP Proxy

Update LCM and Foundation version

If you having issue to upgrade software or firmware of Nutanix cluster i would suggest to upgrade the upgrade the LCM and Foundation version to latest.

Latest version of Nutanix LCM and foundation will have many fix and new features along-with improvements. so it better to upgrade the Nutanix LCM and foundation version first then upgrade the software or/and firmware.

Read more : Nutanix LCM Based Firmware Upgrade Operation Failed

Conclusion

Nutanix LCM: Life Cycle Management is the great feature to up-grade the Nutanix software and hardware firmware with no downtime. But LCM encounter the failure issue sometimes during the upgrade and need to troubleshoot.

In this blog i have described all possible failure scenarios just need to cross check each configuration to rid of the Nutanix LCM failure error.

Thanks to being with HyperHCI Tech Blog to stay tuned.!