Job queue architecture for scheduled social posts
A practical architecture for a job queue that reliably fires scheduled social posts at the right time, even under load.
By André LugA reliable scheduled-post queue needs durable state instead of in-memory timers: occurrences materialized ahead of a horizon with deterministic keys, atomic claiming (in PostgreSQL, FOR UPDATE SKIP LOCKED), leases with heartbeats so abandoned jobs become claimable again, idempotency keys that make re-claimed jobs safe to retry, and per-platform-target delivery tracking rather than a single per-post success flag.
"Post this at 9am Tuesday" sounds like a one-line feature until you have to guarantee it actually happens — once, at roughly the right second, even if the process that scheduled it has restarted twice since, even if a hundred other posts are due in the same minute, and even if the worker handling it dies halfway through. A single setTimeout or a cron job with an in-memory list doesn't survive any of those. A queue that actually holds up under load needs durable state, atomic claiming, and a recovery path for a job whose worker never checks back in.
PostGate handles OAuth, publishing and delivery evidence for every network through one API. Start free or read the docs.
Why doesn't 'wake up and post' survive a restart?
An in-memory timer per scheduled post has three failure modes that only show up in production. A process restart — a deploy, a crash, an autoscaler cycling an instance — wipes every timer that hasn't fired yet, silently. Running more than one instance for redundancy means every instance holds its own copy of the same timer, so the post fires once per instance instead of once. And a slow synchronous call inside the timer callback blocks whatever else was scheduled to fire in that same window. None of this is a corner case; it's what happens the first time the service needs to scale past a single process. If you are choosing the queue itself, our BullMQ vs RabbitMQ comparison covers how each one handles delayed jobs, cron schedules and retries, and pg-boss vs BullMQ compares the two Node.js queues that can keep their jobs in PostgreSQL.
Materialize occurrences ahead of a horizon, not one at a time
A recurring schedule — a cron expression or a set of daily slots — isn't a thing that fires; it's a template that expands into concrete, individually-tracked posts ahead of time, up to some horizon (the next day or two is typical). Each occurrence gets a deterministic key derived from the schedule ID and its timestamp, inserted with an upsert that's a no-op if that occurrence already exists — so a materialization job that reruns, or runs from two workers at once, never double-creates the same post. PostGate's own schedules resource works this way: a cron or slots template produces individually scheduled posts, and changing the expression or timezone resets how far ahead that template has been materialized — so a mid-flight edit can't leave stale occurrences computed under the old timing.
curl -X POST https://postgate.studio/api/schedules \
-H "Authorization: Bearer $POSTGATE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Weekday update",
"kind": "cron",
"expression": "0 9 * * 1-5",
"timezone": "America/Sao_Paulo",
"profile_group_id": "grp_acme123",
"template": {
"post": { "body": "Good morning from Acme." },
"profiles": ["pro_linkedin456"],
"media": []
}
}'Claim due jobs — don't just SELECT them
Once a post exists as a row with a due timestamp, the naive next step — SELECT everything due, then process it — races the moment you run more than one worker. Two workers can both read the same row before either has marked it as taken, and both publish it. The fix is to make the read and the claim a single atomic step: update the row's status while selecting it, locking out concurrent claimants, so exactly one worker ends up owning each job.
-- Skips rows a concurrent worker has already locked, instead of
-- blocking on them — two workers polling at the same instant split
-- the batch instead of colliding on it.
UPDATE scheduled_jobs
SET status = 'claimed', claimed_by = $1, claimed_until = now() + interval '90 seconds'
WHERE id IN (
SELECT id FROM scheduled_jobs
WHERE status = 'due' AND run_at <= now()
ORDER BY run_at
LIMIT 50
FOR UPDATE SKIP LOCKED
)
RETURNING id, run_at, payload;Why does a claimed job need a lease, not a promise?
Claiming a job isn't the same as finishing it — the worker that claimed it can still crash, get killed by a deploy, or lose its network connection mid-publish. Give every claim an expiry (a lease, or a visibility timeout in queueing terms) instead of trusting the worker to report back. A long-running job should extend its own lease with periodic heartbeats; a job whose lease expires without a heartbeat is assumed abandoned and becomes claimable again automatically. This is the difference between a queue that quietly loses work when a process dies and one that just retries it a little later.
How does the idempotency key make a re-claim safe?
Lease expiry inevitably produces the case where two workers both attempt the same logical job — one whose lease just timed out (but is still slowly finishing) and one that just re-claimed it. That's fine as long as the actual side effect is safe to attempt twice. PostGate's create-post endpoint accepts a stable Idempotency-Key that it honors for 72 hours; reusing the same key on a retried attempt returns the original result instead of creating a second post. Deriving that key from the schedule ID and occurrence timestamp — the same key used at materialization time — means the whole pipeline, from template expansion to final publish, is safe to re-run at any step. See Idempotency & safe retries for the full contract.
curl -X POST "https://postgate.studio/api/posts?profile_group_id=grp_acme123" \
-H "Authorization: Bearer $POSTGATE_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: schedule:sch_01JQ:2026-08-27T12:00:00Z" \
-d '{
"post": { "body": "Good morning from Acme." },
"profiles": ["pro_linkedin456"],
"media": []
}'Track delivery per platform target, not just per post
A single "post fired successfully" boolean hides the case that matters most: a unified post going to four platforms where three publish and one fails. PostGate models this explicitly — the aggregate post is draft, scheduled, processing, or one of the terminal states published, partial, failed, canceled— while each individual target carries its own status through pending, media_ready, publishing, and terminal outcomes of published, failed, or skipped, with retry_scheduled and awaiting_reconnect as intermediate states a retry loop or a customer needs to act on. A queue architecture that only tracks the job-level outcome can't represent "three out of four worked" — see Delivery lifecycle for the full state model, and why this distinction matters when a scheduler reports success but nothing actually appeared.
Frequently asked questions
Why can't I just use setTimeout or cron for scheduled posts?
An in-memory timer has three production failure modes: a process restart (deploy, crash, autoscaler cycling an instance) silently wipes every timer that has not fired yet; running multiple instances for redundancy means each instance holds its own copy of the timer, so the post fires once per instance; and a slow synchronous call inside a timer callback blocks everything else scheduled in the same window.
How do I prevent two workers from publishing the same scheduled post twice?
Make the read and the claim a single atomic step: update the row's status while selecting it, so exactly one worker owns each job. In PostgreSQL that is an UPDATE wrapping a SELECT with FOR UPDATE SKIP LOCKED, which lets two workers polling at the same instant split the batch instead of colliding. An idempotency key on the publish call is the safety net for the remaining edge cases.
What is a lease or visibility timeout in a job queue?
An expiry attached to every claim, instead of trusting the worker to report back. A long-running job extends its own lease with periodic heartbeats; a job whose lease expires without a heartbeat is assumed abandoned and becomes claimable again automatically. That is the difference between a queue that quietly loses work when a process dies and one that retries it a little later.
How do idempotency keys make retried publishes safe?
PostGate's create-post endpoint accepts a stable Idempotency-Key it honors for 72 hours: reusing the same key on a retried attempt returns the original result instead of creating a second post. Deriving the key from the schedule ID and occurrence timestamp — the same key used at materialization time — makes the whole pipeline, from template expansion to final publish, safe to re-run at any step.
How should a recurring schedule be turned into actual posts?
Treat the cron expression or slot set as a template that expands into concrete, individually-tracked posts ahead of a horizon (the next day or two is typical). Each occurrence gets a deterministic key derived from the schedule ID and timestamp, inserted with an upsert that is a no-op if it already exists — so a materialization job that reruns, or runs from two workers at once, never double-creates the same post.
