Run Autumn from a background worker

Autumn tasks keep running in the cloud after your request returns, which makes them a natural fit for a worker: start the task, persist the task_id, and let the worker die and come back without losing anything. This recipe is the wiring that makes that safe.

1. Check credits before a batch

A worker that starts fifty tasks against an empty balance produces fifty 402s. Check once per batch, not per task:

import os, requests

AUTUMN = "https://api.autumn.ai"
HEADERS = {"X-API-Key": os.environ["AUTUMN_API_KEY"]}

credits = requests.get(f"{AUTUMN}/credits", headers=HEADERS).json()
if not credits.get("unlimited") and credits["credits_remaining"] <= 0:
    raise RuntimeError("out of credits, pausing queue")

2. Start the task, then persist the id first

The task_id is your resume point. Write it to your database before doing anything else with the response. If the worker crashes one line later, the task is still running in the cloud and the stored id is how you find it again.

task = requests.post(f"{AUTUMN}/task", headers=HEADERS, json={
    "prompt": job.prompt,
    "clarify": False,
}).json()

db.jobs.update(job.id, autumn_task_id=task["task_id"], state="running")

Decision: if your jobs have a known shape (fixed columns, repeatable brief), start from a spec with /task/start instead. See The task model.

3. Poll with the full terminal check

Workers are exactly where a sloppy exit condition hurts most, because nobody is watching the loop spin. Use the complete check from Task lifecycle, including the deleted and out_of_credits cases, and poll on a lazy interval; a research task does not need sub-second updates.

import time

while True:
    task = requests.get(f"{AUTUMN}/task/{task_id}", headers=HEADERS).json()
    if is_terminal(task):   # the exact function from Task lifecycle
        break
    time.sleep(10)

4. Serialize writes per task

A task runs one turn at a time, so two workers continuing the same task_id produce a 409, not a queue. Route all writes for one task through one worker (or take a per-task lock), and treat 409 as “try again after the current turn”, never as a failure. See Errors.

5. Land the rows

Fetch the rows, flatten the cells into your own schema, and keep _sources if anything downstream needs to audit a value:

rows = requests.get(
    f"{AUTUMN}/task/{task_id}/output",
    headers=HEADERS, params={"limit": 1000},
).json()["rows"]

for row in rows:
    flat = {k: v["value"] for k, v in row.items()
            if not k.startswith("_") and isinstance(v, dict) and "value" in v}
    db.results.insert(job_id=job.id, **flat)

Crash recovery

On restart, reread the stored ids and resume polling; the tasks never stopped. If you lose the ids entirely, GET /task lists recent tasks so you can reconcile against your queue.

Task lifecycle

The terminal check this worker depends on.

Follow-up tasks

Sending refinements to a task the worker already finished.

Start here

Pipelines

Production wiring

Agents

Run Autumn from a background worker

1. Check credits before a batch

2. Start the task, then persist the id first

3. Poll with the full terminal check

4. Serialize writes per task

5. Land the rows

Crash recovery

Task lifecycle

Follow-up tasks

​1. Check credits before a batch

​2. Start the task, then persist the id first

​3. Poll with the full terminal check

​4. Serialize writes per task

​5. Land the rows

​Crash recovery

​Related

Task lifecycle

Follow-up tasks

1. Check credits before a batch

2. Start the task, then persist the id first

3. Poll with the full terminal check

4. Serialize writes per task

5. Land the rows

Crash recovery

Related