- security
- ssh
- least-privilege
- proxmox
- firewall
Giving automation root without giving it a shell
Three small decisions that let our portal and deploy tooling do privileged work on a firewall and on hypervisor nodes without any of them holding a general-purpose root login.
Every hosting platform has the same awkward question: the billing portal needs to change things on the firewall and on the hypervisors, but the portal is an internet-facing web application. Whatever it can do, an attacker who compromises it can do too. The usual answers are "give it root and hope" or "give it nothing and do everything by hand". We ended up somewhere in between, with three patterns that cost almost nothing and have held up well. None of them is new; the point is how little you need.
1. The firewall: a key that can run exactly one program
Our firewall has an API. It is fine for rules and aliases, but assigning an address to an interface, or reading certain state tables, needs root on the box. Handing the portal a root SSH key was not an option.
OpenSSH has had the answer for decades in authorized_keys:
restrict,from="<portal IP>",command="/usr/local/sbin/<wrapper>" ssh-ed25519 AAAA... portalcommand=means the key can run that program and nothing else. The client's requested command arrives inSSH_ORIGINAL_COMMAND, which the program parses and validates.restrictdisables port forwarding, agent forwarding, X11 and PTY allocation in one word.from=pins the key to the host that is allowed to use it.
The wrapper itself is small and boring: it accepts a handful of verbs, checks every argument against a strict pattern, refuses anything else and logs the refusal. It also supports --dry so the caller can see what would change. A compromised portal can therefore assign an address to a tenant interface. It cannot read the firewall's configuration, cannot open a shell, cannot pivot.
Two things we learned the hard way:
- One key per function. When the wrapper later needed to do a second kind of thing, we added a second key with a second
command=, not a second verb to the first wrapper. Blast radius stays per-function. - The root shell on our firewall is not a POSIX shell. If your wrapper or your operators paste
$(...)into it, nothing works and the error is cryptic. Run things throughsh -cor switch shells first.
2. Hypervisor nodes: pull, do not push
The second question was how to get scripts onto the hypervisor nodes. The obvious way, scp from the management host, gives the management host a root login on every node. A management host is exactly the machine an attacker wants, because it is where everything is scripted from.
We reversed the direction. The nodes have a read-only deploy key that lets them ask the management host for one file from one pushed commit, and nothing else:
restrict,from="<node IPs>",command="/usr/local/bin/<wrapper>" ssh-ed25519 AAAA... <key-name>The wrapper accepts exactly one form, show <commit> <path>, checks that the commit is an ancestor of main, and streams the blob. A node operator installs a script by running, on the node:
<deploy-command> <commit> <file> <expected-sha256-prefix>The node fetches the file, compares the hash, refuses on mismatch, installs it, and fans it out to the other nodes over the cluster's own existing trust. The management host never has a credential that works on a node.
What this buys you:
- Nothing can be deployed that was not committed and pushed. There is no "I copied a quick fix over" state that the repository does not know about.
- Every installed file is traceable to a commit hash plus a content hash, so "which version is on this node" has a one-line answer.
- The deploy tool is git plus one forced-command wrapper. There is nothing to upgrade.
The cost: the operator has to be on a node to deploy, and pasting a commit hash and a sha256 prefix is slightly more typing than scp. We decided that was the right amount of friction for anything that runs as root on a hypervisor.
3. The ACL trap: deeper paths replace, they do not add
The third item is not a pattern, it is a warning. Proxmox has path-based ACLs with role propagation. Our portal's token had a broad role at the top of the tree so it could create VMs. We then wanted it to also be able to read files from inside guests via the agent, which is a separate privilege. So we added a narrow role for the token on a deeper path.
The token immediately lost the ability to create VMs.
Proxmox ACL semantics: if a subject has any ACL entry on a deeper path, the roles inherited from a shallower path are not merged with it. They are replaced by whatever is set at the deeper path. "Propagate" propagates downward until something else is defined; it does not union. The behaviour is documented, but the word "propagate" invites the opposite assumption.
The fix was not to pile roles onto one token. We gave the narrow privilege to a separate user with its own token, and kept the portal's main token untouched. As a side effect we now have separate subjects for each kind of work, which turned out to be what we wanted anyway.
The common thread
In all three cases the question was never "does this component need root". It did. The question was "root for what, from where, and how do I know afterwards what it did". A forced command answers the first two. A commit hash answers the third. And when a permission system has a surprising rule, the answer is more subjects with narrower roles, not more roles on one subject.