vCenter 6.7 Update 1 – Converge to embedded failed!

By | October 18, 2018

Update 27th October 2018: I was meant to provide this update last weekend, but it has been a hectic week. Within a few hours of posting this issue, a few people at VMware reached out to me to understand more about my configuration, asked me to upload the logs and a few of the smart guys and gals at VMware had been able to reproduce the issue. As it turns out, the key in the APPLMGMT_PASSWORD vecs store is used when a file based backup of the VCSA is scheduled. I believe a KB is on the way, but the workaround is to delete the VCSA backup schedule, perform the converge and you can then re-enable the schedule after the converge is complete.

 

So today I was having a play in the lab with the new vcsa-util utility in vCenter 6.7 Update 1, that provides the ability to converge an external PSC and vCenter Server in to an embedded deployment. I’ve got a walkthrough post on working through this in a basic environment coming soon, stay tuned.

Anyhow, I’d gone ahead and taken my backups, snapshots, got the pre-reqs in place, got my JSON files ready and then I went ahead and ran to the tool to start the converge process.

 

After a couple of minutes, the process finished but unfortunately I saw a big FAILED message in the output even though the Precheck validations has succeeded:

 

As you can see, there is nothing really helpful here, especially the “possible resolution is []“, but I guess that’s what you get when you run a brand new tool that has been out for a day or so as well.

After getting some time later in the evening when they family were in bed, I had a look at the converge_mgmt.log file which is exported to a temporary folder on the machine you are running vcsa-util from (the exact folder is in the output above). Inside of this log file, I found the following error message:

 

I took the “args” section of that output and look at the command that was trying to be run, which ended up being the following:

 

I opened an SSH session to the vCenter Server I was trying to converge and ran the above command, and sure enough I got the same output that was being shown in the converge_mgmt.log file:

 

I then used vecs-cli to list the entries in this specific store, getting the response below:

 

So I could see there was not a certificate in the store, but there was a key.

Now, I do not know what this store is used for, so if you know I’d love if you can leave me a comment below, or reach out to me on twitter @mattallford. I checked a couple of other 6.7 environments, and a few 6.5 ones, and also spun up a clean external 6.7U1 environment in my lab, and this store was empty on all of them. So provided this was a lab environment, I decided to try and remove the entry from the store on the vCenter Server using the following command:

 

After deleting the entry from the vecs store, I was able to rerun vcsa-util again from my management machine and this time the converge process completed successfully.

I thought I would put this post out there, primarily to see if anyone savvier than I can shed some light on what the APPLMGMT_PASSWORD store is used for, and specifically what the “secret key” is used for, and what level of importance it has. I’ve also put this information in to the vsphere channel in the vExpert slack, where there are many very smart folk. I might also try and feed this information back to VMware as it may or may not be beneficial for the support team.

2 thoughts on “vCenter 6.7 Update 1 – Converge to embedded failed!

  1. Ammar

    Thanks Matt for the detailed post. The correct way of applying your workaround and fixing the issue will be as follows.

    1. Log into the vCenter Server Appliance Management Interface https://vcenter-fqdn:5480
    2. Navigate to the Backup view
    3. Next to Backup Schedule, click the Delete button to delete the current backup schedule
    4. Attempt the convergence process again
    5. Once the convergence is complete, re-create the backup schedule. See Schedule a File-Based Backup for more information on creating a backup schedule.

    Thanks!

    Reply
    1. Matt Post author

      Hi Ammar,

      That’s correct, thanks for detailing the steps. I had made a (poor) assumption that if the backup was scheduled, the administrator likely knew the process to delete and recreate.

      Thanks again for taking the time to outline the steps.

      Cheers, Matt.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *