]> Posts for June 2024 🌐:aligrant.com

ESXi 7 to 8, no healthy upstream

Alastair Grant | Thu 27 Jun 2024

I was looking to upgrade an ESXi 6.7 host to 8, which is quite a jump.  My first attempt was to simply build an ESXi 8 host (before Broadcom pulled the product) and drop the config from the old host into place.  You can't do this.

So instead, I took the config and loaded it into an ESXi 6.7 VM and went through the motions to bring it up-to-date through in-place upgrades.  Allowing me to export the 8.0 config and drop that into the new host's disk.  It was going well through to upgrading to 8, once that was completed and the host restarted everything looked fine until I tried to load the web UI, where I was greeted with:

no healthy upstream

Fantastic.  Further to this, when logging on to the shell, I received "Connection failed" when running any esxcli command, or when using vim-cmd to do things like enter maintenance mode, I'd get:

Failed to login: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

Something then, was clearly wrong.  ESXi produces a raft of logs, and trawling through them can be a little tiring!  You can take a quick peak through all the logs by running:

tail -n 30 /var/log/*.log

This will give you the last 30 lines of each log, which is a good way of glancing through all the logs for obvious splashes of errors.  In my situation, healthdPlugins.log and hostd.log providing some suspicious errors:

healthdPlugins: vmw.vpxaStatus ERROR Failed to retrieve vpxa status: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

Er(163) Hostd[134022]: [Originator@6876 sub=Solo] Failed to create SSL context: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:05800088:x509 certificate routines::no certificate or crl found)

I suspected this might be something to do with the certificates in use by the host to present the UI - mine was signed by an internal CA, so could be problematic; though the fact that I can access the HTTP server, and the certificate is being presented ok made me doubt that.

What I did was load the Root CA public certificate into the empty /etc/vmware/ssl/castore.pem and then restarted.  Everything came back online as expected.

Breaking from the voyeuristic norms of the Internet, any comments can be made in private by contacting me.

Entries for: May 2024

Previous Next