The AdOps rule engine is three moving parts. A dispatcher sweeps MongoDB every 10 seconds for rules whose next execution time has passed and enqueues work. A Redis-backed queue carries one job per ad account. A worker consumes each job, reads Meta Insights, evaluates the conditions and executes the actions, then writes down exactly what it did. This is what happens between saving a rule and seeing a budget change.

What runs every 10 seconds?

A cron sweep in the dispatcher. On each tick it queries the rules collection for every rule whose status is ACTIVE and whose next scheduled execution has already passed, with the rule’s optional start and end date window respected.

The sweep is configured to wait for completion, which means the next tick cannot re-enter a sweep that is still running. The honest way to state the guarantee: ticks never overlap. It is not a promise that a rule fires within 10 seconds of coming due — the sweep walks its due rules one after another, and an ad account with many campaigns and a slow Graph response holds the line behind it. A rule fires on the first sweep after it becomes due, and under load “first sweep” is a variable quantity.

After a rule is dispatched, the dispatcher recomputes its next execution time: the check interval added to now, or, in timetable mode, the next slot in the 7-day grid — scanning forward up to seven days when nothing remains today.

Why can the dispatcher not change a campaign?

Because it has no code that writes to Meta. The dispatcher’s only Graph API calls are reads: it enumerates campaigns or ad sets for each ad account, requesting the name and creation time, with the rule’s filters translated into Meta’s own filtering parameter and pagination followed cursor by cursor.

Everything it writes goes to two fields in MongoDB — the next and last scheduled execution timestamps. Every status flip, budget write, rename and duplication lives in the worker, a separate process. That split is a real blast-radius boundary rather than a policy: a bad deploy of the dispatcher cannot alter an advertiser’s account, because the capability is not in that binary.

Entity fetches are retried 3 times in total, waiting 1 second and then 2 seconds. If the third attempt fails, that ad account is skipped for this run rather than failing the whole rule, and the next sweep tries again.

How does one rule become jobs?

By fanning out per ad account. A rule attached to three ad accounts produces three jobs, each carrying the access token, the rule definition, a batch id, the campaigns the dispatcher found for that account, and the account itself.

Jobs are enqueued with no retry attempts and are removed from the queue on both completion and failure. That is a deliberate shape and it is worth understanding: the queue does not replay failed work. The retry is the rule’s next scheduled evaluation. A rule on a 15-minute interval effectively retries in 15 minutes, which is usually what you want for automation that acts on live spend — a burst of replays after a Meta outage is not a kindness.

The batch id is what stitches the run back together in the UI. Every record the worker writes carries it, which is why the Rule Logs screen can show one row per batch and then expand into per-campaign detail.

How many Meta API calls does one evaluation cost?

Fewer than the naive version, and not a fixed number. Before evaluating, the worker groups the rule’s conditions by reporting period, de-duplicates the metric fields required within each group, and issues all of them as a single Meta Batch API call per campaign. A rule with six conditions across Today and Last 7 days is two grouped requests inside one batch, not six calls.

Some metric keys never enter that fetch at all, because they are not Insights fields. Eight keys are excluded and resolved from elsewhere: the two time-of-day metrics, custom metrics, daily budget, campaign age, remaining budget and lifetime budget among them. Budget fields are read from the campaign object only when the rule actually references them, and a metric-versus-metric condition fetches its comparison metric separately — so the cross-metric comparison that makes a rule expressive also costs an extra round trip per condition.

The main rule-evaluation fetch is issued with Meta’s unified attribution setting, which is why a ROAS threshold in a rule lines up with the account-level ROAS reported on the dashboard.

How does the engine decide, and then act?

Each condition resolves to a number and is tested with one of 6 operators. Most metrics come straight from the Insights response, with dotted metric keys matched against the action type breakdown; the eight special keys have their own resolvers, including custom metrics, which are looked up in a Google Sheet at evaluation time. The task’s AND or OR operator then folds the individual results into one verdict.

If the verdict passes, the action runs — but not blindly. The three budget actions re-read the live daily budget from Meta immediately before writing, and the three naming actions re-read the live campaign name, so a percentage change or a name append applies to the campaign’s current state rather than to a stale snapshot. Every write then goes through a single instrumented wrapper, which logs the request and the Graph response and, on failure, pushes the function name, message, stack, the exact request body and Meta’s error response into the error log.

That error log deduplicates. Messages are normalised — timestamps, UUIDs, IP addresses, account ids and long numeric ids are masked — and grouped by function name plus normalised message, with an occurrence count, first and last seen timestamps, the last 10 parameter and response samples, and a status of active, resolved or ignored. Recurring failures collapse into one tracked row with a counter instead of a flood.

What does the engine write down?

Two levels of record, plus the cooldown.

Per batch, a benchmark row: the rule, the ad account, applied and affected item counts, start date, execute count and a status that moves from RUNNING to DONE, with progress reported to the queue as the worker walks the campaign list. The row is created with an upsert keyed on the batch id, so a redelivered job reuses the same audit row rather than leaving duplicate half-finished ones.

Per campaign, per task, per run, an execution record: the action, whether it actually executed, before and after values where the action changed something, every condition with its pass or fail and the value it saw, the campaign snapshot, the ad account, the next execute timestamp, the total execution time and a step-by-step trail of 4 named stages with their elapsed seconds. That trail is what the Benchmark screen’s slow-rule report reads when you ask which rules are heavy.

The cooldown is written as a next-execute timestamp per campaign and task after an action fires. When the rule re-evaluates before it, the run is recorded as skipped with the reason next_execute_not_reached — an explicit row, not a silent no-op.

Where are the limits?

Four worth stating plainly, because knowing them changes how you write rules.

Dispatch latency is not bounded. The sweep is sequential. Many rules across many accounts push the effective interval out.

There is no queue-level retry. The next scheduled evaluation is the retry.

The dispatcher runs as a single instance so that a due rule dispatches once; the worker is the half that scales across cores.

The health endpoints are liveness stubs. Each service exposes a health route that returns a status and process uptime; it does not probe MongoDB, Redis or the queue, so it answers “the process is up”, not “the dependencies are healthy”. Monitor the execution logs for whether work is actually flowing.

  • engineering
  • architecture
  • queue