Proxmox Integration Guide for Nstance

This document describes how Nstance integrates with Proxmox VE for on-premise virtual machine lifecycle management.

Overview

Nstance Server can use Proxmox VE as a provider for VM lifecycle management, enabling on-premise container management node orchestration such as for Kubernetes clusters.

Concepts Mapping

Bootstrap Scripts

Nstance provides a set of bootstrap shell scripts for bootstrapping a Proxmox VE cluster and all its nodes to run Nstance - see deploy/proxmox

All scripts support --dry-run and --help. See each script’s header for full options.

Requirements

Proxmox Environment

• Proxmox VE 8+ cluster (7.3 introduced VM tags in GUI, required for instance tracking, 8 changed the boot format)
• API access enabled with dedicated API token
• Shared storage accessible from all nodes (for VM templates and cloud-init ISOs)
• DHCP server on target network bridge
• VM template with cloud-init support (qemu-guest-agent recommended for Proxmox management)

Network Requirements

• Nstance Server must have network connectivity to the Proxmox API (port 8006)
• VMs must have network connectivity to the Nstance Server gRPC endpoint
• Bridge/VLAN configuration consistent across all cluster nodes

If using keepalived for VIP failover:

• VRRP (IP protocol 112) multicast must be permitted between nodes on the interface where the VIP lives. If the Proxmox firewall is enabled, add a rule to allow VRRP from the VIP subnet (e.g. IN ACCEPT -source 10.0.0.0/24 -p vrrp in /etc/pve/firewall/cluster.fw). Without this, nodes cannot see each other’s VRRP advertisements and both will hold the VIP simultaneously (split-brain).

Object Storage Backend

Nstance Server requires an object storage backend that supports If-Match headers for leader election. Supported providers include AWS S3, Google Cloud Storage, and S3-compatible services such as Ceph RGW and SeaweedFS.

Option 1: Public Cloud Object Storage (Recommended for Hybrid Public/Private Deployments)

For hybrid public/private cloud setups, use a managed object storage service from a public cloud provider (e.g. AWS S3 or Google Cloud Storage).

AWS S3:

nstance-server --storage s3 --bucket nstance --shard <shard> --id <id>

# Standard AWS SDK authentication
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...

Google Cloud Storage:

nstance-server --storage gcs --bucket nstance --shard <shard> --id <id>

# Standard GCP SDK authentication
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

Option 2: SeaweedFS with etcd (Recommended for Private Cloud Deployments)

For fully private cloud deployments, run SeaweedFS with etcd as the metadata store:

Why SeaweedFS:

• Apache 2.0 License - Permissive licensing suitable for any deployment
• PutObject Preconditions - Supports If-Match headers required for Nstance leader election
• Lightweight - Simple deployment, low resource overhead
• S3 API Compatible - Works with standard AWS SDK

Why etcd: SeaweedFS needs a metadata store. To ensure high availability and strong consistency, even in the event of partioning between Proxmox VE clusters, etcd is recommended for the metadata store.

Production Deployment with etcd:

# SeaweedFS with etcd for HA metadata
weed master -mdir=/data/master -peers=master1:9333,master2:9333,master3:9333
weed volume -mserver=master1:9333,master2:9333,master3:9333 -dir=/data/volume
weed filer -master=master1:9333,master2:9333,master3:9333
weed s3 -filer=localhost:8888 -port=8333

Nstance Server Configuration:

nstance-server --storage s3 --bucket nstance --shard <shard> --id <id>

# S3-compatible credentials
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=admin
export AWS_SECRET_ACCESS_KEY=admin
export AWS_ENDPOINT_URL=http://seaweedfs.example.com:8333
export AWS_S3_USE_PATH_STYLE=true

Use the create-shard-config.sh bootstrap script to generate the shard configuration file — see the proxmox bootstrap scripts for details.

Option 3: Proxmox Ceph RGW (Alternative for Private Cloud Deployments)

If your Proxmox cluster already runs Ceph for storage, you can enable the RADOS Gateway (RGW) for S3 compatibility. This avoids deploying additional infrastructure but Ceph is known to be more complex to configure and manage.

Deployment Topology

Run one object storage deployment per region, accessible by all Nstance Server shards in that region.

Authentication

Proxmox API connection is configured using environment variables:

export PROXMOX_API_URL='https://localhost:8006/api2/json' # defaults to this if not set
export PROXMOX_TOKEN_ID='nstance@pve!nstance-token'
export PROXMOX_TOKEN_SECRET='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'

Note: use single quotes on the PROXMOX_TOKEN_ID value to escape the exclamation mark.

The Nstance Server will fail to start if PROXMOX_TOKEN_ID or PROXMOX_TOKEN_SECRET is missing when using the proxmox provider. PROXMOX_API_URL defaults to https://localhost:8006/api2/json if not set, which works when nstance-server runs on the Proxmox node itself.

Creating an API Token:

1. In the Proxmox web UI, navigate to Datacenter → Permissions → API Tokens → Add
2. Select a user (e.g. root@pam or create a dedicated nstance@pve user)
3. Enter a Token ID (e.g. nstance-token)
4. Uncheck Privilege Separation to inherit the user’s permissions
5. Click Add and copy the secret immediately (shown only once)

The PROXMOX_TOKEN_ID format is user@realm!tokenid, e.g. root@pam!nstance-token.

Required Proxmox Permissions:

The API token needs the following privileges. The simplest setup is PVEVMAdmin + PVEDatastoreAdmin + PVEAuditor + PVESDNUser on / (see bootstrap scripts for exact commands).

For reference, the complete list of individual privileges used:

Provider Configuration

ProviderConfig

API URL and credentials (PROXMOX_API_URL, PROXMOX_TOKEN_ID, and PROXMOX_TOKEN_SECRET) are read from environment variables — see Authentication.

Instance Template Args

Provider-specific arguments in instance templates or defaults:

Template Configuration:

Use TemplateVMID when you have either a single node cluster or shared storage and a single template accessible from all nodes. Use TemplateName when each node has its own local template with the same name but different VMIDs (Nstance will look up the correct VMID on the target node at VM creation time).

Either TemplateName or TemplateVMID must be specified (but not both). These can be set in defaults.args for a shard-wide default, or overridden per-template or per-group.

Example:

{
  "defaults": {
    "args": {
      "StoragePool": "local-lvm",
      "TemplateName": "debian-13-template",
      "Bridge": "vmbr0",
      "Cores": 2,
      "Memory": 2048
    }
  },
  "templates": {
    "database": {
      "args": {
        "TemplateName": "debian-12-template",
        "Cores": 4,
        "Memory": 8192,
        "DiskSize": "100G"
      }
    }
  }
}

VM Lifecycle

Instance Creation

When creating a VM, Nstance:

1. Selects a target node using TOPSIS scheduling algorithm
2. Clones the VM from the configured template (new VM name = instance ID)
3. Configures VM with requested resources (CPU, memory, disk)
4. Sets VM description with managed notes (metadata known as “annotations” such as group, kind, created timestamp)
5. Generates and attaches a cloud-init ISO with request userdata script
6. Applies “association” metadata tags (nstance, <cluster-id>, <shard>)
7. Starts the VM

Instance Deletion

When deleting a VM, Nstance:

1. Stops the VM gracefully (if running)
2. Deletes the cloud-init ISO from storage
3. Deletes the VM

Instance Status

Nstance queries VM status via the Proxmox API. Private IPs are populated at agent registration, not from the Proxmox provider.

Status Mapping:

Instance Listing

Nstance uses the /cluster/resources?type=vm API to enumerate all VMs cluster-wide efficiently. VMs are filtered by the nstance tag.

Instance Metadata

Nstance uses a structured metadata model for Proxmox VMs:

Identifier:

• Instance ID is stored as the VM name (e.g. tst06dx9xy919t3v9kzd2xdsyzb3g) — this is the authoritative, globally unique identifier
• Provider instance ID is the Proxmox VMID — unique at any point in time but may be reused after a VM is deleted (Proxmox assigns VMIDs via cluster.NextID() which recycles freed IDs)

Association Metadata (used for filtering and GC):

• nstance - Ownership tag identifying nstance-managed VMs
• <cluster-id> - Cluster ID as a tag (e.g. example-cluster)
• <shard> - Shard ID as a tag (e.g. dev-1)

Annotation Metadata (stored in VM notes, informational only): The VM description field contains a managed notes block:

# DO NOT EDIT BELOW - managed by nstance #
group: test
kind: tst
created: 2026-01-20T12:00:00Z
# DO NOT EDIT ABOVE - managed by nstance #

These annotations are never used by nstance for filtering or reconciliation - only for display purposes in the Proxmox UI/API to assist operators.

Node Scheduling

The scheduler selects the optimal node for VM placement using TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution), the same multi-criteria decision-making algorithm used by Proxmox VE’s built-in HA scheduler.

TOPSIS ranks nodes by their geometric distance to an ideal solution (best possible values) and anti-ideal solution (worst possible values). The node closest to ideal and farthest from anti-ideal wins.

Scheduling Criteria:

• Free memory (60% weight)
• Free CPU (40% weight)

Uses /cluster/resources?type=node to get all node resource usage in a single API call.

Cloud-Init Integration

Nstance uses cloud-init ISOs for VM configuration. The cloud-init ISO contains:

• User-data from the instance template
• Meta-data with instance ID and hostname

Network and Load Balancer Operations

Leader network and load balancer operations are not currently implemented for the Proxmox provider:

• AssignLeaderNetwork - Not implemented (returns error)
• ReleaseLeaderNetwork - Not implemented (returns error)
• CheckSubnetCapacity - Always returns available
• RegisterWithLB - No-op
• DeregisterFromLB - No-op
• ListLBInstances - Returns empty

Deployment Topology

Recommended Architecture

Each Proxmox VE cluster requires exactly one Nstance Server shard (1 cluster = 1 shard):

┌─────────────────────────────────────────────────────────────────┐
│                     Proxmox Datacenter Manager                  │
│                        (Region equivalent)                      │
├─────────────────────┬─────────────────────┬─────────────────────┤
│   PVE Cluster A     │   PVE Cluster B     │   PVE Cluster C     │
│                     │                     │                     │
│  ┌─────┐ ┌─────┐   │  ┌─────┐ ┌─────┐   │  ┌─────┐ ┌─────┐    │
│  │node1│ │node2│   │  │node1│ │node2│   │  │node1│ │node2│    │
│  └─────┘ └─────┘   │  └─────┘ └─────┘   │  └─────┘ └─────┘    │
│                     │                     │                     │
│  ┌───────────────┐ │  ┌───────────────┐ │  ┌───────────────┐  │
│  │Nstance Server │ │  │Nstance Server │ │  │Nstance Server │  │
│  │  (HA CT)      │ │  │  (HA CT)      │ │  │  (HA CT)      │  │
│  └───────────────┘ │  └───────────────┘ │  └───────────────┘  │
└─────────────────────┴─────────────────────┴─────────────────────┘
                              │
                    SDN / EVPN Fabric

Recommendation: Run Nstance Server as a Proxmox HA container (LXC) within each cluster for automatic failover between nodes. Alternative, run multiple containers and rely on Nstance Server leader election to ensure only one is the primary/leader.

Mapping to Cloud Concepts

Multi-Cluster Networking

For spanning workloads across multiple Proxmox VE clusters:

1. Deploy Proxmox Datacenter Manager to manage multiple clusters as a unified environment
2. Configure SDN with EVPN for layer 2/3 connectivity between clusters:
   • Create an SDN Zone (EVPN type for cross-cluster)
   • Define VNets that span clusters
   • Assign VNet to bridge on each cluster’s nodes
3. Configure Nstance Server with one shard per cluster, all pointing to the same object storage backend

Example Shard Configuration:

{
  "shards": [
    {
      "id": "dc1-cluster-a",
      "provider": {
        "kind": "proxmox",
        "region": "dc1",
        "zone": "cluster-a"
      }
    },
    {
      "id": "dc1-cluster-b", 
      "provider": {
        "kind": "proxmox",
        "region": "dc1",
        "zone": "cluster-b"
      }
    }
  ]
}

Single Cluster Deployment

For simpler deployments with a single Proxmox VE cluster:

• One Nstance Server instance with one shard
• Zone scheduling distributes VMs across nodes within the cluster
• No Datacenter Manager or SDN required

Limitations

No Load Balancer Integration

Proxmox has no native load balancer. For Kubernetes ingress:

• Use MetalLB with BGP or L2 mode
• Deploy HAProxy/Traefik as a VM or on bare metal
• Use external load balancer appliance

If BGP is not an option for your deployment, you may wish to use VRRP via something like keepalived or use a load balancer like gobetween, for sending requests to a healthy node running an ingress controller like Traefik using a consistent “virtual IP”.

No Spot/Preemptible Instances

Proxmox does not have a spot instance concept. The agent’s spot termination monitoring is disabled for the proxmox provider.

VM Templates

Proxmox “templates” are roughly equivalent to AWS AMIs - they’re pre-built VM images that new instances are cloned from. Unlike AWS which has a marketplace of official AMIs, Proxmox requires you to create templates from cloud images.

Template Requirements

The base VM template must have:

• cloud-init package installed and enabled
• qemu-guest-agent installed and running (recommended for Proxmox management, not required by Nstance)
• Network configured for DHCP
• SSH server enabled

Guide: Creating a Debian Template

Debian provides official cloud images with cloud-init pre-configured. This is the recommended base OS for Nstance cluster VMs.

Storage Options for Templates

There are two ways to specify a VM template in Nstance args: TemplateVMID or TemplateName.

If using TemplateVMID, note that VM templates in Proxmox VE multi-node clusters will either have a single VMID if you store them in shared storage, or different VMIDs if local to the node.

So if you don’t have shared storage, you can instead use TemplateName and use a consistent name across each node, and Nstance will lookup the template VMID per-node when provisioning a new VM.

For example, create a template named debian-13-template on each node:

• Node A: VMID 9000, name debian-13-template (local-lvm)
• Node B: VMID 9001, name debian-13-template (local-lvm)
• Node C: VMID 9002, name debian-13-template (local-lvm)

Then configure in your defaults or template args: "TemplateName": "debian-13-template"

Nstance queries VMs on the scheduled node and finds the matching template by name. An error is returned if zero or more than one template matches.

For single-node deployments, either option works — but note that TemplateVMID will require one less API call to Proxmox VE.

Template Creation

The easiest way to create a template is with the provided bootstrap script. Shell/SSH into each Proxmox node (or just one, if using shared storage) and run:

# Download and run the bootstrap script
curl -fSL -o vm-template-setup.sh https://raw.githubusercontent.com/nstance-dev/nstance/main/deploy/proxmox/vm-template-setup.sh
chmod +x vm-template-setup.sh

# Create a template with defaults (Debian 13 Trixie, local-lvm storage)
./vm-template-setup.sh

# Or customise options
# ./vm-template-setup.sh --storage ceph-pool --bridge vmbr1

# Or preview what will happen without making changes by doing a dry-run
# ./vm-template-setup.sh --dry-run

The script is idempotent — it skips creation if a template with the same name already exists on the node, downloads the cloud image only if not already present, and automatically selects the next available VMID starting from 9000. Run ./vm-template-setup.sh --help for all options.

Note: If bootstrapping multiple nodes simultaneously, two nodes may select the same VMID since Proxmox VMIDs are cluster-wide. The second node’s qm create will fail — simply re-run the script on that node. To avoid this, either run the script on one node at a time, or pass --min-vmid with a different starting VMID per node (e.g. --min-vmid 9000, --min-vmid 9100).

Note: We don’t resize the template disk. Nstance handles disk sizing via the DiskSize template argument when cloning VMs.

The template name (default debian-13-template) is then used in Nstance args as TemplateName.

Verifying the template:

# List templates
qm list | grep template

# Show template config (replace VMID with the one reported by the script)
qm config 9000

The template is now ready to be used.

Note: For other distributions (e.g. Ubuntu), use the --image-url flag to point to a different cloud image and --template-name for the name. Ensure the image includes cloud-init (qemu-guest-agent recommended).

Manual Template Creation (Advanced)

If you prefer to create the template manually, or need to customise steps beyond what the bootstrap script provides, follow these steps on each Proxmox node (or just one, if using shared storage):

# Download Debian 13 (Trixie) cloud image
cd /var/lib/vz/template/iso/
curl -fSLO https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2

export VMID=9000

# Create a new VM
qm create $VMID --name debian-13-template --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci

# Import the cloud image as the primary disk
qm set $VMID --scsi0 local-lvm:0,import-from=/var/lib/vz/template/iso/debian-13-genericcloud-amd64.qcow2

# Add cloud-init drive
qm set $VMID --ide2 local-lvm:cloudinit

# Set boot order (Proxmox 8+ format)
qm set $VMID --boot order=scsi0

# Configure serial console (required by many cloud images)
qm set $VMID --serial0 socket --vga serial0

# Enable QEMU guest agent
qm set $VMID --agent enabled=1

# Set default user (can be overridden via Nstance server config for userdata)
qm set $VMID --ciuser debian

# Set to use DHCP
qm set $VMID --ipconfig0 ip=dhcp

# Convert the VM to a template
qm template $VMID

Troubleshooting

Common Issues

1. VM Creation Fails: Check storage pool has sufficient space and is accessible from target node
2. No IP Address: Ensure network is configured and agent has registered with the server
3. Cloud-Init Not Applied: Verify cloud-init service is enabled in the template
4. Scheduling Fails: Check node status and resource availability
5. VIP on multiple nodes (keepalived split-brain): VRRP multicast is blocked between nodes. Verify with tcpdump -i <iface> vrrp — you should see advertisements from the peer. Common causes:
   • Proxmox firewall blocking VRRP (protocol 112). Add IN ACCEPT -source <vip-subnet> -p vrrp to cluster firewall rules.
   • keepalived configured on the wrong interface (e.g. vmbr0 instead of the VLAN interface where the VIP subnet lives). The server-with-keepalived.sh script auto-detects this from the VIP address.

Proxmox VE Debugging Commands

# Check cluster status
pvesh get /cluster/status

# List all VMs
pvesh get /cluster/resources --type vm

# Check node resources
pvesh get /nodes/<node>/status

# Get VM status
pvesh get /nodes/<node>/qemu/<vmid>/status/current

Further Reading

• go-proxmox Documentation - API client reference
• Proxmox VE API - Official API documentation