Proxmox VE Setup

sand can run VMs on a Proxmox VE host through its REST API. Once it's set up, a Proxmox host is just another place VMs can run — same board, same commands, same keybindings as a VM on your own machine (see Where VMs Run). What it needs first is some one-time setup on the Proxmox side, and that setup is what this page is about.

The design goal is a least-privilege, pool-scoped token: sand gets exactly the permissions it needs to run its create/clone/delete workflow, confined to a dedicated resource pool, so it cannot see or touch any VM outside that pool — not by accident, not by a bug, not at all. That isolation is structural (Proxmox enforces it), not a matter of sand behaving well.

What you'll set up

A dedicated pool for sand's VMs.
A custom role holding the minimum privileges.
A dedicated user and an API token that inherits the user's rights.
ACLs binding the role to the user at the pool — plus three privileges that Proxmox does not allow to be pool-scoped, granted at the narrowest path that works.
Optionally, a second isolated pool for automated tests.

Everything below is run on the Proxmox host (or any machine with pveum and cluster access) as a full admin — usually root@pam.

Prerequisites

Proxmox VE 9.0 or newer. sand checks the version at preflight and refuses an older host. The minimum-privilege role below is expressed in PVE 9's privilege vocabulary — it relies on the VM.GuestAgent.* privileges introduced in PVE 9, and PVE 9 removed the old VM.Monitor privilege — so it cannot be created on an 8.x host.
A storage that supports images content for VM disks (e.g. local-lvm, a ZFS pool, or a directory storage with Disk image enabled). The built-in local storage does not hold disk images by default, so sand cannot put a VM disk or a cloud-init drive there.
A file-based storage that supports import content for the one-time cloud-image download — a directory, NFS, or CIFS storage. sand downloads the base image with PVE's download-url (content type import), which block storages (zfspool, lvm-thin, RBD) reject with "not a file based storage". This is a separate storage from the disk storage above and defaults to local; set the profile's image_storage to override it. If your disk storage is a directory that already enables import, point image_storage at it and the same storage serves both.

Enabling import on local

The default local is a directory storage but often ships without the import content type. Enable it once with pvesm set local --content iso,vztmpl,backup,import (keep whatever it already lists), or point image_storage at another file-based storage.
- A Linux bridge for VM networking (usually vmbr0).
- Network reachability from the machine running sand to the VM subnet. sand talks to guests over SSH once they boot (it discovers each VM's IP from the guest agent), so your workstation must be able to reach the addresses the VMs get on that bridge.
- An SSH key pair. sand installs your public key into each guest via cloud-init and then connects with the private key, so the profile's identity_path must point to a private key whose .pub sits beside it (ssh-keygen -t ed25519 if you need one). Unlike a remote-ssh profile, a Proxmox profile cannot fall back to the SSH agent — it needs the .pub file to hand to cloud-init.
- The qemu-guest-agent is installed by sand's provisioning into the base image, so you don't need to prepare an image yourself — sand builds its base template from a cloud image the first time you create a VM.

All of it in one script

Every command on this page, in order, is also packaged as a single script — proxmox-setup.sh. All of its settings are environment variables in one block at the top, so the usual case is overriding two or three of them. Copy it to the Proxmox host and run it there as a full admin (root@pam):

SAND_DISK_STORAGE=local-lvm SAND_IMAGE_STORAGE=local ./proxmox-setup.sh

It creates the pool, the three roles, the user and token, and every ACL, then prints the token's effective permissions so you can confirm the scope yourself. It finishes by printing the token line to save and a ready-to-paste profiles.yaml block.

It builds one pool per run, named by SAND_POOL. A second, fully isolated scope — the test pool, or anything else that must never be able to touch your day-to-day VMs — is the same run under a different name, not a different procedure:

SAND_POOL=sandbar-test ./proxmox-setup.sh

The user, the token and every ACL are derived from the pool name, so two runs produce two scopes that cannot see each other. (The roles are shared, which changes nothing: a role is a privilege set and grants nothing until an ACL binds it to a user at a path.) The token file each run tells you to write is named after the pool too, so the second never overwrites the first.

Re-running is safe — the pool, roles, and ACLs are converged in place. The one exception is the API token: Proxmox reveals a token's secret exactly once, so an existing token is left untouched (and no secret is printed) unless you set SAND_TOKEN_RECREATE=1, which deletes and recreates it — invalidating the old secret everywhere it is configured.

The full script

proxmox-setup.sh

#!/usr/bin/env bash
#
# sandbar — one-shot Proxmox VE setup.
#
# Creates the dedicated pool, the least-privilege roles, the user and API
# token, and every ACL described in the "Proxmox VE" page of the sandbar docs.
# Run it ON the Proxmox host (or any machine with `pveum`/`pvesh` and cluster
# access) as a full admin — usually root@pam.
#
# Re-running is safe: the pool, roles and ACLs are converged in place. The one
# exception is the API token, whose secret Proxmox reveals exactly once — an
# existing token is left alone unless you set SAND_TOKEN_RECREATE=1.
#
# It builds ONE pool per run, named by SAND_POOL. A second, fully isolated pool
# — for the opt-in e2e suite, or anything else that must never be able to touch
# your day-to-day VMs — is the SAME run with a different name, not a different
# procedure: the user, the token and every ACL are derived from the pool, so two
# runs produce two scopes that cannot see each other.
#
#   ./proxmox-setup.sh                            # the pool sand uses: sandbar
#   SAND_DISK_STORAGE=tank ./proxmox-setup.sh
#   SAND_POOL=sandbar-test ./proxmox-setup.sh     # an isolated pool for the e2e suite

set -euo pipefail

# --------------------------------------------------------------- settings --
# Everything you might need to change lives in this block.

# The PVE node name — the identifier in /nodes/<node>/… paths, which is not
# always the same string as the host you point sand at.
SAND_NODE="${SAND_NODE:-$(hostname -s)}"

# The dedicated pool. Every VM sand creates lands in it, and the token is scoped
# to it — this name is the whole isolation boundary, and it is the ONE setting
# you change to build a second, unrelated scope (e.g. sandbar-test for the e2e
# suite).
SAND_POOL="${SAND_POOL:-sandbar}"

# The user that owns the token, and the token's id. The user DEFAULTS TO THE
# POOL NAME, and that default is what makes two runs genuinely isolated rather
# than two names for one set of rights: the confined ACLs are granted to the
# user, so pointing a second pool's run at the same user would hand that one
# user both pools. Override it only if you know you want that.
SAND_USER="${SAND_USER:-${SAND_POOL}@pve}"
SAND_TOKEN_ID="${SAND_TOKEN_ID:-prov}"

# Storage backing VM disks (must support content type "images"), and the
# file-based storage the cloud image is downloaded to (must support "import" —
# dir/NFS/CIFS only). Set both to the same name if one storage serves both.
SAND_DISK_STORAGE="${SAND_DISK_STORAGE:-local-lvm}"
SAND_IMAGE_STORAGE="${SAND_IMAGE_STORAGE:-local}"

# The Linux bridge VMs attach to, and optionally the VLAN tag the profile uses
# (leave empty for untagged — the grant is then made on the bridge itself).
SAND_BRIDGE="${SAND_BRIDGE:-vmbr0}"
SAND_BRIDGE_VLAN="${SAND_BRIDGE_VLAN:-}"

# Where to write the token file, if you are running this on the same machine
# that runs sand. Empty (the default) just prints the line to copy across.
SAND_TOKEN_OUT="${SAND_TOKEN_OUT:-}"

# Set to 1 to delete and recreate an existing token — the only way to learn a
# secret again, and it invalidates the old one everywhere it is configured.
SAND_TOKEN_RECREATE="${SAND_TOKEN_RECREATE:-0}"

# Role names. Change these only if the names collide with roles you already
# have — the privilege sets below are what sand actually needs.
#
# They are deliberately NOT derived from the pool: a role is a privilege set and
# grants nothing until an ACL binds it to a user at a path, so every pool can
# share one definition. A second run simply reconverges them.
SAND_ROLE="${SAND_ROLE:-SandbarProv}"
SAND_NET_ROLE="${SAND_NET_ROLE:-SandbarNet}"
SAND_NODE_ROLE="${SAND_NODE_ROLE:-SandbarNode}"

# The minimum-privilege set: create a base VM from a cloud image, clone it,
# resize, configure cloud-init, power on and off, snapshot, read node stats,
# and run a guest-agent command. Nothing more. Drop
# VM.GuestAgent.Unrestricted if you never need `sand` to exec via the agent.
SAND_PRIVS="${SAND_PRIVS:-\
VM.Allocate VM.Clone VM.Audit VM.PowerMgmt VM.Snapshot \
VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network \
VM.Config.Options VM.Config.Cloudinit VM.Config.HWType VM.Config.CDROM \
VM.GuestAgent.Audit VM.GuestAgent.Unrestricted \
Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Pool.Audit}"

# ---------------------------------------------------------------- helpers --

step() { printf '\n==> %s\n' "$*"; }
info() { printf '    %s\n' "$*"; }
warn() { printf 'WARNING: %s\n' "$*" >&2; }
die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }

# json_str <key> — pull one string field out of PVE's compact JSON on stdin.
# Avoids a jq dependency, which Proxmox does not install by default.
json_str() {
  sed -n "s/.*\"$1\"[[:space:]]*:[[:space:]]*\"\([^\"]*\)\".*/\1/p" | head -n1
}

# storage_supports <storage> <content> — true if that storage advertises the
# content type (PVE filters `pvesm status` by it).
storage_supports() {
  pvesm status --content "$2" 2>/dev/null | awk 'NR>1 {print $1}' | grep -qx "$1"
}

ensure_role() { # ensure_role <role> <privs>
  if pveum role add "$1" --privs "$2" >/dev/null 2>&1; then
    info "role $1 created"
  else
    pveum role modify "$1" --privs "$2"
    info "role $1 updated"
  fi
}

ensure_pool() { # ensure_pool <pool> <comment>
  if pveum pool add "$1" --comment "$2" >/dev/null 2>&1; then
    info "pool $1 created"
  else
    info "pool $1 already exists"
  fi
}

ensure_user() { # ensure_user <user>
  if pveum user add "$1" --comment "sandbar automation" >/dev/null 2>&1; then
    info "user $1 created"
  else
    info "user $1 already exists"
  fi
}

# token_exists <user> <token-id> — true if that API token is already present.
# PVE reveals a token's secret only at creation, so this is what decides between
# "created it and here is the value" and "it is already there, reuse your file".
token_exists() {
  pveum user token list "$1" --output-format json 2>/dev/null |
    grep -q "\"tokenid\"[[:space:]]*:[[:space:]]*\"$2\""
}

grant() { # grant <path> <role> <user>
  pveum acl modify "$1" --roles "$2" --users "$3"
  info "$2 on $1"
}

# ------------------------------------------------------------- preflight ---

step "Checking the host"

command -v pveum >/dev/null 2>&1 ||
  die "pveum not found — run this on the Proxmox host, as an admin (root@pam)"

pve_major="$(pveversion | sed -n 's|^pve-manager/\([0-9]\{1,\}\)\..*|\1|p')"
if ! printf '%s' "$pve_major" | grep -q '^[0-9]\{1,\}$' || [ "$pve_major" -lt 9 ]; then
  die "sand needs Proxmox VE 9.0 or newer (found: $(pveversion))"
fi
info "$(pveversion)"

pvesh get "/nodes/$SAND_NODE/status" >/dev/null 2>&1 ||
  die "no such node '$SAND_NODE' — set SAND_NODE to the name shown in the PVE tree"
info "node $SAND_NODE"

storage_supports "$SAND_DISK_STORAGE" images ||
  die "storage '$SAND_DISK_STORAGE' does not exist or does not hold disk images — set SAND_DISK_STORAGE"
info "disk storage $SAND_DISK_STORAGE (images)"

if storage_supports "$SAND_IMAGE_STORAGE" import; then
  info "image storage $SAND_IMAGE_STORAGE (import)"
else
  warn "storage '$SAND_IMAGE_STORAGE' does not advertise content type 'import', so the
         one-time cloud-image download will fail. If it is a directory/NFS/CIFS
         storage, add the type — keeping whatever it already lists — with:

           pvesm set $SAND_IMAGE_STORAGE --content iso,vztmpl,backup,import

         Otherwise point SAND_IMAGE_STORAGE at a file-based storage. Continuing."
fi

ip -o link show "$SAND_BRIDGE" >/dev/null 2>&1 ||
  warn "bridge '$SAND_BRIDGE' not found on this node — continuing, but check SAND_BRIDGE"

# ----------------------------------------------------------------- roles ---
# Shared by every pool this script sets up, so they are created once.

step "Creating the least-privilege roles"
ensure_role "$SAND_ROLE" "$SAND_PRIVS"
ensure_role "$SAND_NET_ROLE" "SDN.Use"
ensure_role "$SAND_NODE_ROLE" "Sys.AccessNetwork Sys.Audit"

# ----------------------------------------------------------------- scope ---

# ----------------------------------------------------------------- scope ---
# The pool, its user, its token, and the ACLs that confine that user to it.
# Everything below is derived from SAND_POOL, so a second run under a different
# name builds a second scope that shares nothing with this one but the roles.

full_token_id="${SAND_USER}!${SAND_TOKEN_ID}"

step "Pool $SAND_POOL"
ensure_pool "$SAND_POOL" "sandbar-managed VMs"

# Add the VM disk storage to the pool so the role's Datastore privileges apply
# to it. Errors if it is already a member, which is fine — and two pools may
# share one storage: a pool holds VMIDs and whole storages, so this grants the
# role's Datastore privileges on that storage, never a pool-private slice of it.
if pveum pool modify "$SAND_POOL" --storage "$SAND_DISK_STORAGE" >/dev/null 2>&1; then
  info "storage $SAND_DISK_STORAGE added to the pool"
else
  info "storage $SAND_DISK_STORAGE already in the pool"
fi

step "User and API token"
ensure_user "$SAND_USER"

token_value=""
if token_exists "$SAND_USER" "$SAND_TOKEN_ID"; then
  if [ "$SAND_TOKEN_RECREATE" = 1 ]; then
    pveum user token remove "$SAND_USER" "$SAND_TOKEN_ID" >/dev/null
    info "removed the existing token $full_token_id"
  else
    warn "token $full_token_id already exists and its secret cannot be read back.
         Keeping it — reuse the token file you saved when it was created, or
         re-run with SAND_TOKEN_RECREATE=1 to replace it (which invalidates
         the old secret everywhere it is configured)."
  fi
fi

if ! token_exists "$SAND_USER" "$SAND_TOKEN_ID"; then
  # --privsep 0: the token inherits the user's rights. With separation on, a
  # token's rights are the INTERSECTION of its own and its user's — and this
  # user exists solely to own the token, so that intersection would be empty.
  # The confined ACLs below are granted to the user for that reason.
  token_value="$(pveum user token add "$SAND_USER" "$SAND_TOKEN_ID" --privsep 0 \
    --output-format json | json_str value)"
  [ -n "$token_value" ] || die "could not read the new token's value for $full_token_id"
  info "token $full_token_id created"
fi

step "Granting the confined ACLs to $SAND_USER"
# The pool: this is the whole isolation boundary.
grant "/pool/$SAND_POOL" "$SAND_ROLE" "$SAND_USER"

# Three things a pool cannot hold, granted at the narrowest path that works.
# A plain Linux bridge lives under the synthetic SDN zone "localnetwork"; with
# a VLAN tag the path gains the tag as a further segment.
sdn_path="/sdn/zones/localnetwork/$SAND_BRIDGE"
if [ -n "$SAND_BRIDGE_VLAN" ]; then
  sdn_path="$sdn_path/$SAND_BRIDGE_VLAN"
fi
grant "$sdn_path" "$SAND_NET_ROLE" "$SAND_USER"
grant "/nodes/$SAND_NODE" "$SAND_NODE_ROLE" "$SAND_USER"

# Storage privileges on both storages sand uses — the disk storage to allocate
# VM disks, the image storage to download the cloud image.
grant "/storage/$SAND_DISK_STORAGE" "$SAND_ROLE" "$SAND_USER"
if [ "$SAND_IMAGE_STORAGE" != "$SAND_DISK_STORAGE" ]; then
  grant "/storage/$SAND_IMAGE_STORAGE" "$SAND_ROLE" "$SAND_USER"
fi

step "Verifying the scope of $full_token_id"
perms="$(pveum user permissions "$full_token_id" --output-format json)"
printf '%s\n' "$perms"
case "$perms" in
'{}' | '')
  warn "the token has NO effective permissions. This is almost always a
         privilege-separated (--privsep 1) token whose ACLs were granted to the
         token instead of the user. Re-run with SAND_TOKEN_RECREATE=1."
  ;;
esac
if printf '%s' "$perms" | grep -q '"/"[[:space:]]*:'; then
  warn "a permission is granted at '/' for $SAND_USER — the isolation guarantee
         does NOT hold. Find and remove that over-broad grant:
         pveum acl list"
fi
# The pool must be the ONLY pool this token can see. A second scope's pool
# showing up here means both runs were pointed at one user (see SAND_USER).
if printf '%s' "$perms" | grep -o '"/pool/[^"]*"' | grep -qv "\"/pool/$SAND_POOL\""; then
  warn "this token can see a pool other than $SAND_POOL. Two scopes must not
         share a user — re-run the other pool with its own SAND_USER."
fi

if [ -n "$token_value" ] && [ -n "$SAND_TOKEN_OUT" ]; then
  (
    umask 077
    printf '%s=%s\n' "$full_token_id" "$token_value" >"$SAND_TOKEN_OUT"
  )
  chmod 600 "$SAND_TOKEN_OUT"
  info "token file written to $SAND_TOKEN_OUT (mode 600)"
fi

# --------------------------------------------------------------- summary ---
#
# The pool this run built can serve either purpose, and the script has no way to
# know which you meant — so it prints the mapping for both. Take the half you
# need and ignore the other.

# Display text only — this string is pasted into the reader's own shell, which
# is what expands the tilde. The path this script may WRITE to is
# SAND_TOKEN_OUT, which is used verbatim.
# shellcheck disable=SC2088
token_file="~/.config/sandbar/${SAND_POOL}.token"

step "Done — finish the setup on the machine that runs sand"

if [ -n "$token_value" ]; then
  cat <<EOF

The token secret below is shown ONCE and cannot be retrieved again. Save it on
the machine that runs sand — the file is named after the POOL, so a second
scope's token never overwrites this one:

  mkdir -p ~/.config/sandbar
  ( umask 077; printf '%s\n' '${full_token_id}=${token_value}' > ${token_file} )
  chmod 600 ${token_file}
EOF
else
  cat <<EOF

No new token was created, so there is no secret to print. Reuse the token file
you saved when the existing token was created.
EOF
fi

cat <<EOF

If this pool is for everyday use, add it to your profiles.yaml
(~/.config/sandbar/profiles.yaml) — id is yours to choose, and must be unique
across profiles:

  profiles:
    - id: ${SAND_NODE}
      name: proxmox
      type: proxmox
      enabled: true
      host: $(hostname -f 2>/dev/null || hostname)
      node: ${SAND_NODE}
      pool: ${SAND_POOL}
      storage: ${SAND_DISK_STORAGE}
      image_storage: ${SAND_IMAGE_STORAGE}
      bridge: ${SAND_BRIDGE}
      token_file: ${token_file}
      identity_path: ~/.ssh/id_ed25519   # the key sand installs and connects with
      # insecure: true                   # only if the PVE cert is self-signed

If it is the isolated pool for the opt-in e2e suite, this is its environment:

  export PROXMOX_E2E=1
  export PROXMOX_E2E_HOST=$(hostname -f 2>/dev/null || hostname)
  export PROXMOX_E2E_NODE=${SAND_NODE}
  export PROXMOX_E2E_POOL=${SAND_POOL}
  export PROXMOX_E2E_STORAGE=${SAND_DISK_STORAGE}
  export PROXMOX_E2E_IMAGE_STORAGE=${SAND_IMAGE_STORAGE}
  export PROXMOX_E2E_BRIDGE=${SAND_BRIDGE}
  export PROXMOX_E2E_TOKEN_FILE=${token_file}
  export PROXMOX_E2E_SSH_USER=debian
  export PROXMOX_E2E_SSH_IDENTITY=~/.ssh/id_ed25519

To build the OTHER one, run this again with a different pool name — same steps,
same script, a scope that shares nothing with this one:

  SAND_POOL=<the other pool> $0
EOF

The rest of this page is the same setup done by hand, with the reasoning for each privilege — read it if you want to know why the script does what it does, or if your host differs enough that you'd rather drive it yourself.

Step 1 — Create a dedicated pool

Every VM sand creates is placed in this pool automatically, and the token is scoped to it. That membership is the whole isolation boundary.

pveum pool add sandbar --comment "sandbar-managed VMs"

Step 2 — Create the minimum-privilege role

This is the exact set of privileges sand's workflow needs — create a base VM from a cloud image, clone it, resize, configure cloud-init, power on and off, snapshot, read node stats, and run a guest-agent command. Nothing more.

pveum role add SandbarProv --privs "\
VM.Allocate VM.Clone VM.Audit VM.PowerMgmt VM.Snapshot \
VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network \
VM.Config.Options VM.Config.Cloudinit VM.Config.HWType VM.Config.CDROM \
VM.GuestAgent.Audit VM.GuestAgent.Unrestricted \
Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Pool.Audit"

Some of these are non-obvious, and if you trim them further you'll get confusing failures — so, for the record, why each of the less-obvious ones is here:

Privilege	Why `sand` needs it
`VM.Config.HWType`	Setting `scsihw`, `vga`, and `machine` on the base VM. Cloud images need `virtio-scsi-pci`, not the PVE default.
`VM.Config.Options`	Setting `agent`, `name`, and `ostype`.
`VM.Config.Disk`	Covers disk devices and the `boot` order.
`VM.Config.Cloudinit`	Injecting the SSH key, user, and network config.
`Datastore.AllocateTemplate`	Downloading the cloud image into storage via the `download-url` endpoint (content type `import`). PVE gates that endpoint on this privilege specifically — `Datastore.AllocateSpace` alone is not enough, and its absence fails the very first base-build step with a 403.
`Pool.Audit`	So the pool name appears in listings — without it `sand` can't tell which VMs are its own.
`VM.GuestAgent.Unrestricted`	Only needed for guest-agent `exec`. It's the broadest privilege in the set; drop it if you never need `sand` to run a command via the agent (it uses SSH for shells regardless).

Do not add VM.Monitor or VM.Console

VM.Monitor was removed in PVE 9 — including it makes pveum role add reject the whole command. VM.Console is only for the VNC/SPICE console, which sand never uses. Leaving both out is deliberate.

Step 3 — Create a user and an API token

pveum user add sandbar@pve --comment "sandbar automation"
pveum user token add sandbar@pve prov --privsep 0 --output-format json

The second command prints the token value exactly once:

{
  "full-tokenid": "sandbar@pve!prov",
  "value": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "info": { "privsep": "0" }
}

--privsep 0 turns privilege separation off, so the token inherits its user's permissions. That's deliberate: Proxmox computes a privilege-separated (--privsep 1) token's rights as the intersection of the user's permissions and the token's own. So if you grant the ACLs only to the token and its user has none — exactly the case here, where sandbar@pve exists solely to own this token — the token ends up with no permissions at all, and pveum user permissions 'sandbar@pve!prov' prints {}. With separation off you grant the least-privilege ACLs below to the user sandbar@pve instead, and the token carries exactly those. Because this user has no password and no other roles, its permissions are the confined set — there is nothing broader for the token to inherit. Save the value now; it cannot be retrieved again.

sand authenticates with the token's full identity, which is the two fields above joined by an =:

<full-tokenid>=<value>
sandbar@pve!prov=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Write exactly that one line into a file sand will read (referenced as token_file in Step 6):

mkdir -p ~/.config/sandbar
umask 077
printf 'sandbar@pve!prov=%s\n' 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
  > ~/.config/sandbar/pve1.token
chmod 600 ~/.config/sandbar/pve1.token

The file holds one line and nothing else — the identity, an =, and the value. sand refuses to read it unless it is mode 600 (owner-only).

Step 4 — Bind the role to the user at the pool

# Scope the user (and so its token) to the pool: this confines it to sandbar's own VMs.
pveum acl modify /pool/sandbar --roles SandbarProv --users sandbar@pve

# Add the VM storage to the pool so the role's Datastore privileges apply to it.
# This grants them on the WHOLE storage object (see the note below), not a
# pool-private slice — Proxmox has no notion of "the pool's part" of a storage.
pveum pool modify sandbar --storage local-lvm

A shared storage is trusted as a whole

A pool can only contain VMIDs and entire storages — there is no "pool's slice" of a datastore. So if local-lvm also backs VMs outside the pool, the token's Datastore.AllocateSpace and Datastore.Audit reach the whole of it: it can enumerate every volume on that storage (volume names, sizes, and the owning VMID) and allocate space anywhere on it. It still cannot read or modify another VM's disk — and it never sees those VMs' configuration or state — because reading, attaching, or reassigning a volume owned by another guest requires access to that guest or the Datastore.Allocate privilege, neither of which this token has. If you want the storage boundary as tight as the VM one, give sand a dedicated storage (its own LVM-thin pool or dataset) rather than one shared with other VMs.

Step 5 — Grant the three privileges that can't be pool-scoped

Proxmox implements pool permissions by projecting a role onto the pool's members, and a pool can only contain VMs and storage. Three things sand needs are therefore not grantable at the pool, and have to be granted at the narrowest path that does work. None of them grants any access to another VM — that's exactly why they're named individually here instead of reaching for a broader role.

# 1. Attach a VM to the bridge. A plain Linux bridge lives under the synthetic
#    SDN zone "localnetwork"; with a VLAN tag the path gains the tag as a further
#    segment (…/vmbr0/<tag>).
pveum role add SandbarNet --privs "SDN.Use"
pveum acl modify /sdn/zones/localnetwork/vmbr0 --roles SandbarNet --users sandbar@pve

# 2. Download the cloud image to storage, and read node CPU/memory for the board
#    header. Both are node-scoped, not pool-scoped.
pveum role add SandbarNode --privs "Sys.AccessNetwork Sys.Audit"
pveum acl modify /nodes/pve1 --roles SandbarNode --users sandbar@pve

# 3. Storage privileges on BOTH storages sand uses:
#    - the DISK storage, so it can allocate VM disks (Datastore.AllocateSpace);
#    - the IMAGE storage, so it can download the cloud image with content=import
#      (Datastore.AllocateTemplate). These are the same command when a single
#      file-based storage serves both — run it once in that case.
pveum acl modify /storage/local-lvm --roles SandbarProv --users sandbar@pve
pveum acl modify /storage/local     --roles SandbarProv --users sandbar@pve

Privilege	Path	What it allows
`SDN.Use`	`/sdn/zones/localnetwork/vmbr0`	Attaching a VM to that one bridge — nothing else on the network
`Sys.AccessNetwork`	`/nodes/pve1`	Downloading the cloud image to storage
`Sys.Audit`	`/nodes/pve1`	Reading node CPU/memory/disk stats for the board header

You won't see localnetwork under SDN → Zones

localnetwork is a built-in synthetic zone that Proxmox uses only as the permission path for plain Linux bridges — it is not a configured SDN zone, so it never appears under Datacenter → SDN → Zones, and that's expected. To confirm the bridge grant landed, look under Datacenter → Permissions (or run the verification command below), not the SDN panel.

Verify the scope

Confirm the token ended up with only the paths you intended — this is the check that proves the isolation guarantee actually holds:

pveum user permissions 'sandbar@pve!prov' --output-format json

The result should list only /pool/sandbar, your disk and image storages, the bridge, and the node /nodes/pve1. If any permission appears at /, the isolation guarantee does not hold — go back and remove the over-broad grant.

If this prints {}

An empty result means the token has no effective permissions — almost always because it was created with --privsep 1 and the ACLs were granted to the token instead of the user. A privilege-separated token's rights are the intersection of the user's and the token's, so a token whose user holds no ACLs gets nothing. Recreate it with --privsep 0 (Step 3) and grant the ACLs to --users sandbar@pve (Steps 4–5).

Step 6 — Point `sand` at the host

Proxmox profiles can be created in the TUI or by editing profiles.yaml

A Proxmox profile can be created in the TUI by pressing p → n, choosing Proxmox from the type picker, and filling the form. Its insecure checkbox (for self-signed certificates) is toggled with space, not enter — enter moves to the next field, as it does on every other row. The form covers every field below, including the optional user, image_storage and base_image; the focused field's description appears under the form, naming the default an optional field falls back to when left blank. The form also refuses to save without storage, bridge or identity_path — a hand-edited profiles.yaml is not checked for those three, and a profile missing one fails only later, when it tries to create a VM. Alternatively, add a Proxmox profile by hand-editing profiles.yaml as shown here — the YAML form is the reference and automation path.

Add a proxmox profile to your profiles.yaml:

profiles:
  - id: pve1
    name: proxmox
    type: proxmox
    enabled: true
    host: pve1.example.com        # the API host; add :8006 only if non-default
    node: pve1                    # the PVE node name
    pool: sandbar                 # the dedicated pool from Step 1
    storage: local-lvm            # images-capable storage for VM disks
    # image_storage: local        # file-based (dir/NFS/CIFS) storage for the
    #                             # cloud-image download; defaults to "local"
    # base_image: https://…/my-base.qcow2   # cloud image URL; defaults to the
    #                             # project golden image (see note below)
    bridge: vmbr0                 # the Linux bridge
    token_file: ~/.config/sandbar/pve1.token
    identity_path: ~/.ssh/id_ed25519   # REQUIRED: the SSH key sand installs + connects with
    # user: dev                   # the guest login user; defaults to your host username
    # insecure: true              # only if the PVE cert is self-signed
    # ca_file: /etc/pve/pve-root-ca.pem   # or pin the CA instead

The profile fields:

Field	Meaning
`host`	Hostname or IP the API answers on. A bare host uses port `8006`; append `:port` only if you've changed it.
`node`	The PVE node name (the identifier in `/nodes/<node>/…` paths) — often the same string as the host, but not always.
`pool`	The dedicated pool. Every VM `sand` creates lands here, and the token is scoped to it.
`storage`	Required. The images-capable storage backing VM disks and the cloud-init drive. May be block (zfspool, lvm-thin) or file-based.
`image_storage`	Optional. The file-based storage (dir/NFS/CIFS) the cloud image is downloaded to with content `import` — block storages reject it. Defaults to `local`. The disk is then imported onto `storage` from here.
`base_image`	Optional. URL of the cloud image the base template is built from. Defaults to the project golden image (Debian genericcloud with `qemu-guest-agent` preinstalled, checksum-verified). Override only to use your own image — which must also ship the agent (see below). The download filename is derived from the URL.
`bridge`	Required. The Linux bridge `net0` attaches to. Omitting it does not mean "no network": QEMU falls back to user-mode NAT, so the guest boots and is simply unreachable over SSH.
`token_file`	Path to a file holding `user@realm!tokenid=value`.
`identity_path`	Required. Path to an SSH private key. `sand` installs the matching `<identity_path>.pub` into the guest via cloud-init and then connects over SSH with the private key — so the `.pub` must exist beside it. Generate one with `ssh-keygen -t ed25519` if you don't have it.
`user`	Optional. The guest login user `sand`'s cloud-init creates (and SSHes in as). Defaults to your host username.
`insecure`	Optional. Skip TLS verification (PVE ships a self-signed cert by default).
`ca_file`	Optional. Pin a CA certificate instead of disabling verification.

The token file must be chmod 600

Like identity_path for a remote profile, token_file is a path, never the credential itself — profiles.yaml stays secret-free and safe to check into dotfiles. sand refuses to read a token file that is readable by group or other; a leaked API token is not a recoverable mistake. Create it with umask 077 (as in Step 3) or run chmod 600 on it.

That's it. sand builds its base template from a cloud image the first time you create a VM (this takes a few minutes — it downloads the image, runs the same Ansible provisioning the other backends use, and converts the result to a PVE template), then clones each new VM from it. The board header shows the node's real CPU, memory, and storage usage, sampled from the API.

Why the default image is a project-built one, not stock Debian

sand learns a VM's IP address only from the QEMU guest agent (it is the only IP a pure-API client can read from PVE), and it needs that IP to SSH in and provision the base. So the base image must boot with qemu-guest-agent already running — but stock cloud images (including Debian genericcloud) don't ship it, and PVE's API can neither install packages through its built-in cloud-init nor upload a cloud-init snippet. So base_image defaults to a project-built golden image — upstream Debian genericcloud with the agent baked in, published as a GitHub release asset and checksum-verified on download. You don't need to do anything for this to work.

If you override base_image with your own URL, that image must likewise have qemu-guest-agent installed and enabled, or the base build will hang waiting for the agent. Bake it in offline on any Linux box with libguestfs' guestfs-tools:

virt-sysprep -a your-image.qcow2 \
  --install qemu-guest-agent \
  --run-command 'systemctl enable qemu-guest-agent'

virt-sysprep edits the disk offline (never boots the guest, so cloud-init's first-boot behaviour is untouched) and strips machine-specific state so clones stay independent.

A separate pool for automated tests

If you run sand's opt-in end-to-end test suite (or otherwise want a throwaway pool that can never touch your day-to-day VMs), set up a second pool and token exactly as above but with different names — sandbar-test, its own user, and its own token. The two pools are fully isolated from each other, so automated runs create and destroy VMs freely in sandbar-test with no possibility of affecting the VMs in sandbar.

proxmox-setup.sh does this for you — it is the same run you did for the everyday pool, with one variable changed:

SAND_POOL=sandbar-test ./proxmox-setup.sh

It derives the user (sandbar-test@pve) and every ACL from that name, so the second scope shares nothing with the first, and it prints the PROXMOX_E2E_* block below already filled in. If you override SAND_USER to a user that already owns another pool, the script's verification step says so rather than letting you build two pools one token can see.

The test suite is documented in the repository's development guide; it's gated behind a build tag and skips unless you configure it, and it reads its target from these environment variables (which mirror the profile fields above):

export PROXMOX_E2E=1
export PROXMOX_E2E_HOST=pve1.example.com
export PROXMOX_E2E_NODE=pve1
export PROXMOX_E2E_POOL=sandbar-test          # the ISOLATED test pool
export PROXMOX_E2E_STORAGE=local-lvm
# export PROXMOX_E2E_IMAGE_STORAGE=local       # file-based; defaults to "local"
export PROXMOX_E2E_BRIDGE=vmbr0
export PROXMOX_E2E_TOKEN_FILE=~/.config/sandbar/pve-test.token
export PROXMOX_E2E_SSH_USER=debian            # the cloud-init guest login user
export PROXMOX_E2E_SSH_IDENTITY=~/.ssh/id_ed25519
# export PROXMOX_E2E_IMAGE=https://…/my-base.qcow2   # optional; unset uses the
                                                     # default golden image, the
                                                     # only one with the guest
                                                     # agent the suite needs
# export PROXMOX_E2E_INSECURE=1               # if the PVE cert is self-signed

# Optional: a VMID OUTSIDE the test pool, to prove the token cannot touch it.
# The isolation test never creates or deletes this VM — you own it.
export PROXMOX_E2E_FOREIGN_VMID=100

Setting PROXMOX_E2E_FOREIGN_VMID enables the pool-isolation test, which takes a VMID outside the test pool and asserts that the pool-scoped token is refused (with a permission error) when it tries to read, stop, or delete it — the live proof of the guarantee this whole page is built around. To confirm the other half by hand — that the foreign VM is unchanged afterward — check its status as an admin before and after; the test deliberately can't, because its token can't see the VM at all, which is the point.