Using Terraform and Cloud-Init to deploy and automatically monitor Proxmox instances — Windows

In the last post I covered deploying Linux-based Proxmox instances using Terraform, leveraging Cloud-Init and cloud-config to register these instances with SaltStack.

The instances in the previous post were both Linux distributions (Debian and Fedora). This is because Cloud-Init was initially built for Linux, and then ported to other Unix-like systems (eg FreeBSD, OpenBSD and more).

Thanks to a company called Cloudbase Solutions an equivalent of Cloud-Init is also available for Windows, known as Cloudbase-Init.

Cloudbase-Init

Cloudbase-Init is written in Python, and was originally developed to allow customization of Windows images on OpenStack. However it also supports a number of different metadata services (everything from AWS and Azure to VMware).

Cloudbase-Init also supports a subset of the cloud-config format, including write_files (i.e. creating files on the instance), users , groups , runcmd (i.e. running arbitrary commands) and more.

Preparing the template machine

The first step before creating your instances is to create a template Windows image. This image can then be used as a base image for other instances (using cloning).

Cloudbase Solutions provide tools to build Windows images ready to be deployed with Cloudbase-Init installed, but they rely on Hyper-V on Windows. If you do not have access to a Windows machine to generate the images, do not have access to Hyper-V, or simply want to go through the process yourself, then you can install Cloudbase-Init in a Windows instance yourself.

Installing Windows

In this post, I am using Windows Server Core 2019 (i.e. without the Desktop Experience installed). As Proxmox uses KVM as its underlying hypervisor, you will also need to download the VirtIO-Win ISO. This contains all the necessary drivers for Windows for VirtIO-based NICs, storage and serial devices. Without this, you would need use emulated devices (i.e. emulated SATA or network devices), which are less efficient than the VirtIO equivalents.

First, download the VirtIO ISO and place it in the /var/lib/vz/templates/iso directory on your Proxmox host. This is where Proxmox expects to find all ISO images by default.

After this, place the ISO for your Windows version of choice in the same directory. If you already have the ISOs, then use something like scp or rsync to place it on the Proxmox server. Alternatively, you can dowload trial versions of Windows from the Microsoft Evaluation Center. These trial versions can be activated later with the appropriate license key and service contract with Microsoft.

Now create the instance. The settings in the below images should work on a default Proxmox install, but you may need to change them if your environment differs (eg using Ceph/ZFS for storage, OpenVSwitch for networking): —

Create the VM

General Options

Choose the OS

System hardware (Disk controller and graphics

Create the disk

Choose the CPU and core count

Choose the amount of memory

Choose the network

After creating the instance, select it, go to Hardware, and then add a CD-ROM drive: —

Choose the virtio-win.iso — this contains all the necessary Windows drivers: —

When you start the machine, you should now be presented with the following: —

Loading files

Booting

Choose your language and timezone

Click Install Now

Choose which version (Desktop Experience = GUI)

Choose Custom installation (because there is no existing installation to upgrade)

At this point, you will see that there is no hard disk to install to. This is because Windows does not include the VirtIO drivers by default: —

We can resolve this by clicking on Load Driver, which will give the option to browse for drivers: —

Navigate to the driver with the virtio-win-0.x.x ISO: —

Scroll down to viostor: —

Select the driver and click on Next: —

You should now see the disk to install to: —

At this point, you can either click Next and finish the installation, or you can click Load Driver to add more VirtIO drivers. The main ones to install are: —

• Balloon — This allows for dynamic memory management, optimizing memory usage where appropriate
• NetKVM — This installs the network drivers for the VirtIO network card
• qxldod — If you require graphics drivers
• vioserial — This installs serial drivers, which we will need to use later for Cloudbase-Init (mainly for logging)

You can choose to install the drivers later, using either the GUI if you chose the desktop experience, or using pnputil if using Server Core.

Once done, click Next to finish the installation: —

Once Windows is installed and restarted, you will be given the option to create a password for the Administrator user: —

Unlock the instance (using Ctrl, Alt and Delete

Proxmox Ctrl, Alt and Delete in NoVNC

Confirm changing of the password

Change the password

Logged in as the Administrator

At this point, we are now ready to customize the image.

Customization

To prepare the image, the minimum we require for Cloud-Init is the Cloudbase-Init utility. Before installing Cloudbase-Init I also do the following: —

• Install OpenSSH Server for remote management
• Install Salt
• Disable the firewall and enable RDP

The reason for disabling the firewall is that in the environments I work in, host-based firewalls are not used. Instead, they are managed either via network firewalls or via security groups (if used within a cloud provider). If you require the firewall to be enabled, then you will need to customize the firewall rules to match your environment.

Installing OpenSSH Server

To install the OpenSSH Server on Windows 2019, you can follow the steps in this post. We are then able to SSH into the machine: —

Install Salt

To install the Salt Minion, again you can follow the steps in this post. However, do not follow the steps for updating the minion configuration post-install. The minion configuration will be customized by Cloudbase-Init based upon the provided User Data.

Disable the firewall and enable RDP

While most tasks can be completed using Powershell in Windows Core, there is no inbuilt text editor. You can either install something like ViM or you can use RDP so that notepad.exe is available for file editing.

We also disable the firewall at this point, as otherwise we will need to add an RDP-specific rule to the firewall to allow us to access the machine.

You can also use the sconfig tool to enable RDP as well: —

Installing Cloudbase-Init

To install Cloudbase-Init, go to the Cloudbase-Init product page, scroll down to Downloads, and choose the version appropriate for your environment. I am running Windows 2019 on a 64-bit x86 processor, so I chose the Stable Cloudbase-init x64 version.

The following steps will download and then install Cloudbase-Init: —

You can install Cloudbase-Init interactively and follow the on-screen instructions to customize the configuration. However we will need to change some options within the Cloudbase-Init configuration anyway, so there is little to benefit in using the GUI installer to configure the Cloudbase-Init service.

We can check that it is installed using: —

Configuring Cloudbase-Init

The default configuration files for Cloudbase-Init enable a number of different metadata services, most of which are not applicable to Proxmox.

We also need to enable the User-Data plugin otherwise Cloudbase-Init will not be able access any configuration we provide via Cloud-Init. The User Data can be in the form of Batch scripts (i.e. cmd.exe ), Powershell, BASH (if it is installed natively under Windows), Python and cloud-config .

The following configuration files tell Cloudbase-Init to use the ConfigDrive2 format for user-data (the default format for a Windows guest on Proxmox), and enables the plugins required to make changes to the network, files and source changes from user-data: —

C:\Program Files\Cloudbase Solutions\cloudbase-init\conf\cloudbase-init.conf

C:\Program Files\Cloudbase Solutions\cloudbase-init\conf\cloudbase-init-unattend.conf

You can either edit these files in notepad , or you can use SCP (as we enabled SSH already) to transfer the files from another machine.

We also set all Cloudbase-Init logs to be exposed on the Serial port, allowing us to see the changes being made by Cloudbase-Init (as well as any errors/warnings) separately from the boot and login process itself on screen/VNC.

After this is done, you can then run sysprep , which “generalizes” a Windows image. This removes install-specific information (eg passwords, unique IDs etc) in preparation for the image to be reused/cloned. The Cloudbase-Init installation also includes an Unattend answer file. This is similar to a Kickstart or PXE script in the Linux world, providing answers to questions usually presented to users during initial setup.

The included Unattend file looks like the below: —

To start the sysprep process using this Unattend file, run: —

This will prepare the instance and tell the system to run cloudbase-init.exe on boot.

Creating a template

There are two ways to convert the machine into a template in Proxmox. The first is to right click on the created machine, and select Convert to template: —

The other is to find the ID of the machine in the Proxmox CLI, and then do the following: —

We can now use this template when defining an instance within Terraform

Terraform

Create the Cloud-Init template

The first step to building an instance using Terraform is to prepare the Cloud-Init template.

There are some caveats to be aware of before we start: —

• Currently not all cloud-config modules are supported in Cloudbase-Init
• The format that Proxmox provides some of the Cloud-Init data is not 100% compatible with Cloudbase-Init

In regards to the first, Cloudbase-Init supports the following cloud-config modules (at the time of writing): —

• write_files — Creating files (eg configuration files)
• set_timezone
• set_hostname
• groups — Creating local groups on the machine, as well as assigning existing users to them
• users — Creating local users
• ntp — Setting NTP servers
• runcmd — Running a set of arbitrary commands, executed with cmd.exe (not Powershell)

In regards to the latter, there is an issue with how Proxmox provides the DNS Nameservers. Proxmox provides dns_nameservers as the option, whereas Cloudbase-Init expects dns-nameservers (associated Github Issue). This issue, combined with Cloudbase-Init not supporting cloud-config network module (to customize our network configuration) means that we must rely on DHCP.

In most cloud environments, IP addressing normally ephemeral (i.e. not persistent). If you treat the instance in this manner, then DHCP provides the same sort of functionality. However if you are expecting to be able to use static addressed instances, then you may need to look at other methods than cloud-config .

The basic template we are going to use looks like the below: —

We use a feature called multi-part content to leverage both cloud-config and Powershell to configure the instance. You could extend this to use more complex Powershell scripts, setting the timezone, creating users and more. In our file, we update the Salt Minion configuration (using the write_files module in cloud-config ), and then restart the salt-minion service (using Powershell).

Define the instance

The instance is defined similarly to the instances in the previous post: —

One notable difference is that we set the ipconfig0 = «ip=dhcp» option. Without this, the Terraform provider will crash. We should be able to provide a static IP here, but as noted previously, the format Cloudbase-Init provides DNS nameservers in is not compatible with Cloudbase-Init, meaning that we would not be able to access any hostname-based service (eg our Salt server, or downloading dependencies).

Other than this, we should now be able to build a Windows instance.

Build the instance

Now that we have defined the instance within Terraform, we can go ahead and build it: —

The last part of this is due to running the Qemu Guest Agent. Proxmox sees that the IP address return is an IPv6 address, and returns it to the Terraform provider. Unfortunately the provider is expecting either an IPv4 address, or an IPv6 address without %2 at the end of it. However if we look in the Terraform state, we are still managing this machine: —

We can also verify it exists in the Proxmox console: —

It appears that while the terraform apply operation produces an error, it is after all configuration has taken place. Terraform does however mark the instance as tainted (i.e. required to be destroyed and rebuilt). To avoid rebuilding the instance on your next terraform apply , run terraform untaint proxmox_vm_qemu.win2k19-vm to untaint it. You can then verify with a terraform plan that instance matches what we defined: —

Alternatively, if you do not install the Qemu Guest Agent, then the Terraform provider will complete without issues. It is worthwile running the Guest Agent (safer shutdowns, exposes information to the hypervisor regarding the instance), but it is not necessary to the day to day running of the machine.

The Salt setup and states are the same as we used in this post. We use Salt to: —

• Deploy Consul as an agent
• Register with the Consul server (the Salt server)
• Deploy the Windows Exporter so that Prometheus can monitor the instance

As per the previous post, we also run a highstate (i.e. all applicable Salt states) when the Minion is accepted by the Salt server.

Accept the key

To accept the minion on the master, we need to check that it has registered first: —

Now we can accept the key: —

We can check what states will be applied to this minion as well: —

Verify the highstate

As mentioned, the Minion configuration specifies that it will run a highstate when the key is accepted. We can verify this seeing if the instance has registered with Consul and that the Windows Exporter is running: —

Looks like they are working! It is worth nothing that we do not set a hostname in cloud-config for our machine, so it uses a randomly generated hostname instead.

As a last step of verification, we can run another highstate to ensure that no configuration files change, or any other packages are due to be installed: —

Everything looks good!

Prometheus

As noted in the previous post, we are using the same Prometheus setup as we do here.

Also, as we are using the same Salt states, we should expect to see the instance like we do in this post: —

Grafana

Again, everything in this post regarding Grafana is applicable too. We can use the Windows Node Dashboard to verify the metrics: —

Alternatively you can use this dashboard, which has now been updated to use the Windows Exporter metric naming (i.e windows_ instead of wmi_ , as the exporter changed from wmi_exporter to windows_exporter ): —

Summary

Using Cloudbase-Init, it is possible to customize a Windows image when it starts for the first time, allowing you to bootstrap a machine ready to be used in your environment. This takes away the work of installing machines manually, while also being able to manage Windows in a similar way to Linux and Unix variants (eg FreeBSD, OpenBSD) that support Cloud-Init.

As we have also seen, it is not without caveats currently. Some of this is down to Proxmox in how it presents Cloud-Init data, some of it is down to the Cloudbase-Init utility that is expecting configuration in a certain format but could potentially be in multiple formats (i.e. both dns-nameservers and dns_nameservers are possible within Cloud-Init, but Cloudbase-Init currently understands the former), and some are down to the Terraform provider that does not seem to understand some of values presented by the Proxmox API (specifically the IPv6 addressing).

Over time these will improve, as issues have been raised for all of these cases. This should not discourage you from using any of these tools, as all of them together bring Windows deployment on Proxmox on par with Linux (in terms of bootstrapping and Terraform usage).

Cloudbase-Init was initially created for use with OpenStack, and the fact it works with other providers and infrastructure is a nice bonus.