Polling and scaling

← Documentation index

How the poller works, how to tell whether it is keeping up, and what to change when it is not.

The model

Every device is polled on three cadences, dispatched through Redis-backed queues:

Poll	Default	Collects
Fast	30 s	Interface traffic, resources, clients, status
Logs	60 s	Device log events
Slow	5 min	Interfaces, VLANs, system info, wireless, STP

Enqueue rate is not connect rate. Workers process jobs with limited concurrency, so what matters is whether they drain faster than the schedule fills.

Is it keeping up?

Settings → Polling, or GET /api/system/poller.

The number to read is headroom — service capacity divided by demand:

arrival   devices / poll interval
service   worker concurrency / mean poll duration
headroom  service / arrival

Below 1.0 the backlog grows every cycle and devices go stale. Comfortably above it, the fleet is being polled on schedule.

Headroom alone is not enough, which is why backlog and drain estimate sit beside it. It describes flow — whether the workers can keep up from here — and is blind to work already queued. A healthy headroom next to a large backlog means capacity is fine but you are still working through a queue.

A single status summarises all of it: ok, draining, or saturated.

Which devices are being missed

The same page lists devices not polled recently, and distinguishes two states that look identical from the outside:

• not attempted recently — the poller never reached it
• responded with… — it was reached and did not answer

device_poll_stats records attempt and success separately per device and per kind, with duration and last error. Their difference is that distinction.

Tuning

All of these are environment variables and need a container restart.

Variable	Default	Effect
POLLER_CONCURRENCY	12	Workers per queue. The first lever when headroom is low
POLLER_INTERVAL_MS	30000	Scheduler tick. Lengthening reduces demand
POLLER_JOB_TIMEOUT_MS	45000	Ceiling on one poll, so a dead device cannot hold a worker
POLLER_JOB_RETENTION_SEC	3600	How long finished jobs are kept
POLLER_JOB_RETENTION_COUNT	5000	How many finished jobs are kept
REDIS_MAXMEMORY	512mb	Ceiling on Redis

Raise concurrency first. Lengthen the interval second — it reduces freshness, which is the thing you are paying for.

Behaviour worth knowing

A device already queued is not enqueued again. Deduplication is by job id, so a backlog cannot compound: one outstanding poll per device and kind, however long it waits.

Periodic polls are not retried in-job. The schedule is the retry — a device that did not answer is unlikely to answer a second later, and a retry occupies a worker the rest of the fleet needs.

Stale queued polls are discarded, at startup and every five minutes. A periodic poll that has waited longer than its own cadence is worthless: running it produces a reading the next cycle would take anyway. Discarding is safe precisely because the work is periodic — the schedule re-enqueues within one interval.

Job history is bounded. Redis runs with noeviction rather than an LRU policy on purpose: BullMQ's job locks and stalled-check keys carry TTLs, and evicting one causes duplicate processing or a stalled-job storm. A memory ceiling protects the host; retention keeps usage far below it.

Clearing a backlog

Settings → Polling → Clear queued, or POST /api/system/poller/drain.

Nothing is lost. The scheduler re-enqueues whatever is still due on its next tick, so the cost is at most one interval of freshness.

Scheduled work and timezones

Scheduled tasks are evaluated in the timezone set under Settings → General, not the container's. The container runs UTC, so without this a backup set for 02:00 would fire at 02:00 UTC wherever you are — and on the wrong day far enough east.