Run Autumn from a background worker

Autumn tasks keep running in the cloud after your request returns, which makes them a natural fit for a worker: start the task, persist the task_id, and let the worker die and come back without losing anything. This recipe is the wiring that makes that safe.

1. Check credits before a batch

A worker that starts fifty tasks against an empty balance produces fifty 402s. Check once per batch, not per task:

import os, requests

AUTUMN = "https://api.autumn.ai"
HEADERS = {"X-API-Key": os.environ["AUTUMN_API_KEY"]}

credits = requests.get(f"{AUTUMN}/credits", headers=HEADERS).json()
if not credits.get("unlimited") and credits["credits_remaining"] <= 0:
    raise RuntimeError("out of credits, pausing queue")

2. Start the task, then persist the id first

The task_id is your resume point. Write it to your database before doing anything else with the response. If the worker crashes one line later, the task is still running in the cloud and the stored id is how you find it again.

task = requests.post(f"{AUTUMN}/task", headers=HEADERS, json={
    "prompt": job.prompt,
    "clarify": False,
}).json()

db.jobs.update(job.id, autumn_task_id=task["task_id"], state="running")

Decision: if your jobs have a known shape (fixed columns, repeatable brief), start from a spec with /task/start instead. See The task model.

3. Wait for status changes

Workers can wait efficiently without sleeping or guessing from legacy status fields. Use the bounded wait and stop when done becomes true.

import time

def get_status(task_id):
    while True:
        r = requests.get(
            f"{AUTUMN}/task/{task_id}",
            headers=HEADERS,
            params=dict(wait=50),
        )
        if r.status_code in (429, 503):
            time.sleep(float(r.headers.get("Retry-After", 5)))
            continue
        if not 200 <= r.status_code < 300:
            raise RuntimeError(r.json().get("message") or r.text)
        return r.json()

task = get_status(task_id)
while not task["done"]:
    task = get_status(task_id)

if task["state"] == "input-required":
    raise RuntimeError(task["message"])
elif task["state"] in {"failed", "canceled"}:
    error = task.get("error") or {}
    raise RuntimeError(
        error.get("message") or error.get("code") or task.get("message") or task["state"]
    )

4. Serialize writes per task

A task runs one turn at a time. A continue submitted while it is running queues and returns 202; if queueing is unavailable, the legacy 409 task_executing response is used. Execute is not queued while a turn is running. Route writes for one task through one worker or take a per-task lock. See Errors.

5. Land the rows

Fetch the rows, flatten the cells into your own schema, and keep _sources if anything downstream needs to audit a value:

rows = requests.get(
    f"{AUTUMN}/task/{task_id}/output",
    headers=HEADERS, params={"limit": 1000},
).json()["rows"]

for row in rows:
    flat = {k: v["value"] for k, v in row.items()
            if not k.startswith("_") and isinstance(v, dict) and "value" in v}
    db.results.insert(job_id=job.id, **flat)

Crash recovery

On restart, reread the stored ids and resume polling; the tasks never stopped. If you lose the ids entirely, GET /task lists tasks newest first, 50 per page with an X-Next-Cursor header for the next page, so you can reconcile against your queue.

Task lifecycle

The status fields and bounded wait this worker depends on.

Follow-up tasks

Sending refinements to a task the worker already finished.

Start here

Pipelines

Production wiring

Agents

Run Autumn from a background worker

1. Check credits before a batch

2. Start the task, then persist the id first

3. Wait for status changes

4. Serialize writes per task

5. Land the rows

Crash recovery

Task lifecycle

Follow-up tasks

​1. Check credits before a batch

​2. Start the task, then persist the id first

​3. Wait for status changes

​4. Serialize writes per task

​5. Land the rows

​Crash recovery

​Related

Task lifecycle

Follow-up tasks

1. Check credits before a batch

2. Start the task, then persist the id first

3. Wait for status changes

4. Serialize writes per task

5. Land the rows

Crash recovery

Related