Nutanix Host’s NIC Down False Alert: Link on NIC eth# is down

If you have done recently deployment of Nutanix HCI infra and Nutanix Prism prompts NIC Link down false Alert Message: ‘Nutanix Link on NIC eth# of host xx.xx.xx.xx is down’. Don’t worry about this Nutanix node / host link on NIC eth# is down false alert message, i am going to share the solution to fix / resolve it in few minutes troubleshooting steps.

Lets explore the reason and solution of Nutanix Link on NIC down on node / host false alert prompts by Nutanix Prism.

Step 1: Get the Nutnaix NIC Down on host / node alert list from Nutanix ncli command to run it on any Nutanix CVM in the Nutanix cluster

nutanix@CVM:~$ ncli alert ls

Above Command Output:

ID                              : dec0f4f1-4b05-44ac-8f1g-74bn9q14c142
Message                   : Link on NIC eth0 of host xx.xx.xx.xx is down.
NIC description      : Intel Corporation Ethernet Controller 10G X550T (rev 01)
Severity                   : kWarning
Title                          : NIC link down on host xx.xx.xx.xx

Note : Host is showing one warning regarding Link on NIC eth0 of host xx.xx.xx.xx is down that is false alert.

Step 2: Print the Nutanix Node / Host NICs status : UP / Down to run following command:

nutanix@CVM$ cat /home/nutanix/data/serviceability/check_cvm_health_job_state.json

Above command output

================== xx.xx.xx.xx =================
 {
   "eth3": {
     "Status": "Up",
     "Timestamp": 1570788334
   },
   "eth2": {
     "Status": "Up",
     "Timestamp": 1570788334
   },
   "eth0": {
     "Status": "Down",
     "Timestamp": 1570712613
   }

Note: eth0 is manually disconnected because we are not using it any more, but showing NIC status is DOWN, that why Nutanix Prism is prompting Link on NIC eth# is down on host / node false alert frequently.

Solution

There could be a reason of NIC down on Nutanix host / node false alert, that the Nutanix CVM has stale / old entry of this NICs state, when NICs was connected to the host / node and now you are not using that NIC any more. But Nutanix CVM did not remove the stale / old entry from the NICs status file after disconnected the NICs from Nutanix host / node.

Lets fix / resolve the Nutanix Link on NIC eth# is down on host / node false alert issue to follow the given steps:

Step 1: Just run following command to remove the NIC stale entry from Nutanix CVM

nutanix@CVM$ rm /home/nutanix/data/serviceability/check_cvm_health_job_state.json

Step 2: Check the NIC down false alert status to run following command

nutanix@CVM$ ncc health_checks network_checks nic_link_down_check --cvm_list=<CVM_IP_Address>

Step 3: Validate the NICs UP / Down link status to run following command

nutanix@CVM$ cat /home/nutanix/data/serviceability/check_cvm_health_job_state.json

Above command output

There should not be any DOWN NIC link entry in this file as shown below:

================== xx.xx.xx.xx =================
 {
   "eth3": {
     "Status": "Up",
     "Timestamp": 1570788334
   },
   "eth2": {
     "Status": "Up",
     "Timestamp": 1570788334 
  }
 }================== xx.xx.xx.xy =================
 {
   "eth3": {
     "Status": "Up",
     "Timestamp": 1570788176
},
"eth2": {
     "Status": "Up",
     "Timestamp":  1570788176 
   }
 }==================  xx.xx.xx.xz =================
 {
   "eth3": {
     "Status": "Up",
     "Timestamp": 1570788368
   },
   "eth2": {
     "Status": "Up",
     "Timestamp": 1570788368
   }

I hope, after performing above commands issue have resolved.!

Conclusion

This is not a common issue but sometimes if you have intentionally disconnected the NICs from Nutanix host(s) and NICs are not using any more due to some reason then you may receive ‘ Nutanix Link on NIC eth# on host is down’ alert untill you reconnect the NICs cable or remove the stale entry from the Nutanix CVM’s NICs status files.

Thanks to being with your favorite HyperHCI Tech Blog to stay tuned with latest and trending technology.!