jezahost
  • cloud-init
  • cloudbase-init
  • proxmox
  • debian
  • windows-server

Three silent cloud-init failures on Proxmox

Hostnames resetting after a password change, Windows passwords landing on the wrong user, and a Debian image where cloud-init never ran. No errors in any of them.

cloud-init is the quiet part of VM provisioning: it runs once, does its job and gets out of the way. When it fails loudly you fix it. The problems below are the other kind. Nothing logged an error, nothing exited non-zero, and each one took a while to notice because the VM looked fine.

1. Change a password in the panel, lose your hostname

Symptom. A customer resets their root password from our portal. The VM reboots. Its hostname is back to the template default and /etc/hosts has been rewritten. The password change itself worked.

Cause. Proxmox does not store a cloud-init instance ID. It derives one by hashing the generated user-data and network-data. Change anything that lands in those files, including the password, and the hash changes. cloud-init sees a new instance ID, concludes it is running on a fresh machine, and re-runs every per-instance module: set_hostname, update_etc_hosts, SSH key generation, the lot.

A second, related surprise: Proxmox writes manage_etc_hosts: true into its generated user-data. cloud-init merges configuration in the order system config, vendor-data, user-data, and the last one wins. So whatever the template said about /etc/hosts is overridden every time, which is why that file kept changing too.

Fix. Stop letting Proxmox generate the metadata. We supply our own meta-data snippet via cicustom with a fixed instance-id per VM. Password and network changes still go through the normal Proxmox fields; only the identity is pinned. First-boot modules run once, on first boot, and never again. The hostname stays.

2. Windows: the password goes to the wrong user

Symptom. After a rebuild, some Windows Server VMs ended up with the customer's password set on Administrator instead of the configured admin user. Not all of them. Same template, same metadata, different outcome.

Cause. cloudbase-init was installed as an Automatic service. After sysprep, Windows runs the specialize pass and then boots. Automatic services can start before specialize has fully finished. Run one: cloudbase-init creates the configured user with a random password, as it should when no password has been applied yet. Reboot. Run two: the user "already exists", so the SetUserPassword plugin applies the metadata password to the fallback account name from its config, which was Administrator. Whether you got run one or run two first depended on timing.

Fix. Service start type Manual, and a SetupComplete.cmd that starts cloudbase-init explicitly. SetupComplete.cmd is guaranteed to run after specialize, so there is only ever one first run. The config file also names the intended user. After provisioning, our portal reads the cloudbase-init log through the guest agent and checks which account the password was applied to, so a regression would show up as a failed check rather than a confused customer.

3. Debian "genericcloud": cloud-init turns itself off

Symptom. A new Debian template built from the official genericcloud qcow2. The VM boots. Hostname is wrong, the password is not set, the disk is not resized. cloud-init status says disabled. No error anywhere.

Cause. The genericcloud image ships a trimmed kernel without sr_mod and ahci. Proxmox delivers cloud-init configuration on a virtual CD-ROM. No CD-ROM driver means the device never appears, the NoCloud datasource finds nothing, and cloud-init's generator correctly decides there is no datasource and disables itself. Everything after that is working as designed, which is exactly why nothing complains.

Fix. Use the generic image, which has the full driver set. It is a little larger. Alternatively keep genericcloud and switch the datasource to something that does not need a CD-ROM, but for a template that is more moving parts than the extra few hundred megabytes are worth.

What they have in common

In all three cases the component that "failed" was behaving exactly as documented. Proxmox hashes its config into an instance ID. Windows starts automatic services early. cloud-init disables itself without a datasource. The bugs lived in the gaps between layers, and none of the layers had any way to know. If a VM "works but is slightly wrong" after provisioning, the first question to ask is not "what errored" but "which component made a reasonable decision based on something it could not see".