Proxmox Task Scheduling and Tuning¶
Note
This is a draft. Screenshots are marked with % TODO screenshot: comments
and placeholder image paths under /admin/media/; capture them from the live
NETLAB+ interface and renumber to the sequential imageNNNN.png scheme before
publishing. Confirm the version number in the introduction below.
When NETLAB+ drives a Proxmox VE cluster, a single classroom action can launch many hypervisor operations at once: cloning a pod duplicates every virtual machine it contains, and the start of a large reservation can power on dozens of VMs together. Proxmox will accept only so much concurrent work before it begins rejecting operations with an overload error.
As of version 25.1.0, NETLAB+ includes a task scheduler that paces the work it sends to each Proxmox cluster so the cluster is kept busy without being overwhelmed. This section explains how the scheduler works and how to inspect and tune it.
Note
Task scheduling and tuning apply only to Proxmox VE datacenters. The pages described here are not shown for VMware vSphere datacenters.
How Task Scheduling Works¶
Every Proxmox operation NETLAB+ performs (a clone, a migration, a power action, and so on) is assigned a weight, a small whole number that estimates how demanding that operation is on the cluster. A full disk clone is expensive and carries a high weight; powering a VM on is cheap and carries the minimum weight of 1.
NETLAB+ tracks the combined weight of all the operations it currently has running and admits new ones only while there is room under two limits:
- Host Capacity:
The maximum combined weight of tasks NETLAB+ will run at once on a single Proxmox host.
- Datacenter Capacity:
The maximum combined weight of tasks NETLAB+ will run at once across all hosts in the datacenter.
When admitting the next operation would push either limit over its cap, NETLAB+ holds the operation in a queue and starts it as soon as capacity frees up. This happens automatically and requires no configuration; the default weights and capacities are tuned to work well on typical clusters.
Note
These limits govern only the work NETLAB+ itself initiates. Operations you start directly in the Proxmox web interface are not counted against them.
Auto-Tuning¶
The default weights are estimates. A given cluster’s storage, networking, and host hardware determine how much work it can really absorb, so NETLAB+ can adjust the weights to fit the cluster it is managing.
When Proxmox Auto-Tuning is enabled, NETLAB+ watches for overload errors from the cluster. Each time an operation is rejected for overload, NETLAB+ raises that operation’s weight slightly and retries, learning a higher, more realistic cost over time. Learned increases are bounded: a weight will not climb beyond a fixed multiple of its configured value, nor beyond what a single host can admit. If a weight reaches that ceiling, it is a strong signal that the cluster has a real capacity problem worth investigating.
Auto-tuning is off by default and is enabled per datacenter.
Note
Auto-tuning only ever raises weights, and only for the specific operations that overload the cluster. It never lowers a weight on its own. To return to your configured values, use the tuning form’s restore actions described below.
Enabling Auto-Tuning¶
Click on > Virtual Machine Infrastructure > Virtual Datacenters and Management Agents.
Click the datacenter you want to configure, then click Edit.
Select the Proxmox Auto-Tuning checkbox and click Submit.
Note
Enabling or disabling auto-tuning does not discard what NETLAB+ has already
learned; the learned values are simply frozen. Disabling it stops further
adjustments, and re-enabling it resumes from where it left off. To clear the
learned values, use Restore User Defaults on the Task Tuning
page.
Viewing Task Load¶
The Task Load dashboard shows NETLAB+’s live view of how busy a Proxmox cluster is: an overall datacenter load, a load figure for each host, and the list of tasks currently consuming scheduler capacity. It is a useful first stop when clones or pod operations seem slow, to see whether NETLAB+ is pacing work against a busy cluster.
Click on > Virtual Machine Infrastructure > Virtual Datacenters and Management Agents.
Click the datacenter you want to inspect, then click View Tasks.
The Task Load page is displayed. The load figures and task lists update automatically.
The dashboard is organized as follows.
- Datacenter Load:
A bar and percentage at the top showing the combined weight of all running tasks across the datacenter relative to the datacenter capacity.
- Host Tabs:
One tab per Proxmox host. Each tab shows that host’s load relative to the host capacity, and lists the tasks NETLAB+ currently has running on it.
A load near 100% means NETLAB+ is holding new operations in the queue until running ones finish. This is the scheduler working as intended; it is not an error. Persistently high load is a sign the cluster is a bottleneck for the work being asked of it.
To adjust the weights or capacity limits for this cluster, click Tuning. To return to the datacenter, click Dismiss.
Tuning Task Weights and Capacity¶
The Task Tuning page lets you inspect and override the weights and capacity limits for a Proxmox datacenter. Most sites never need to change these; adjust them only when you have a specific reason, such as a cluster that is unusually powerful or unusually constrained.
From the Task Load dashboard, click Tuning.
Click Edit to change the values.
Adjust the capacity limits and task weights as needed (see the field descriptions below), then click Submit.
Each field also carries an on-screen help tip; click the help button on the form to show the tips inline.
Field Descriptions - Task Tuning
- Proxmox Auto-Tuning:
Enables the automatic weight-learning feedback loop described in Auto-Tuning. The weights and capacities below apply whether or not this box is checked; the checkbox controls only the automatic adjustments.
- Datacenter Capacity:
The maximum combined weight of concurrent tasks across all hosts in the datacenter. Must be greater than or equal to the host capacity.
- Host Capacity:
The maximum combined weight of concurrent tasks on a single host. Must not exceed the datacenter capacity, and no single task weight may exceed it.
- Task weights:
One field per Proxmox operation type, giving that operation’s weight. Higher weights make an operation count for more, so fewer of them run at once. Every weight must be a whole number of at least 1 and no greater than the host capacity.
About the Clone Weights¶
Three of the weight fields describe virtual machine cloning, which is usually the heaviest work NETLAB+ asks of a cluster and the most common reason to tune.
- Full clone:
The weight for a full clone that NETLAB+ initiates (shown as
qmclone:full). Proxmox copies every virtual disk into a fully independent VM. This is the most disk-intensive clone, so it carries the heaviest clone weight.- Linked clone:
The weight for a linked clone that NETLAB+ initiates (shown as
qmclone:linked). The new VM shares its parent’s disks and stores only the blocks it changes. Almost no data is copied, so it is far cheaper than a full clone.- External clone:
The weight for a clone NETLAB+ found running on the cluster but did not start itself, for example one launched from the Proxmox web interface (shown as
qmclone). Because NETLAB+ cannot tell whether such a clone is full or linked, this weight assumes the worst case, a full clone.
Note
The remaining weight fields correspond to other Proxmox operations (migrate, snapshot, template, power actions, and so on). The on-screen help tip for each field describes what the operation does and why it is weighted as it is.
Restoring Defaults¶
The Edit page provides two ways to reset the fields. Neither takes effect until
you click Submit, so you can preview the values first, and
Cancel discards them.
Restore User Defaults:Fills the fields with the values you last saved, discarding any adjustments auto-tuning has learned since. Use this to return the cluster to your own configured baseline.
Restore System Defaults:Fills the fields with the built-in values NETLAB+ ships with. Use this to start over from the factory settings.
Note
Saving the tuning form commits the values shown as your new baseline and clears all learned auto-tuning adjustments for the datacenter, not only the fields you changed. The new values apply to operations admitted after you save; work already running keeps the weight it was admitted with.