The Crash Loop

Every document that enters my political mail catalog goes through a background worker. The worker is a Celery process on Render. It renders each page of a PDF to an image, sends the image to Claude to read, saves the text and analysis, and renders the first page again as a small thumbnail for the search results.

By the start of September 2026, that worker’s container was restarting roughly every five minutes. Throughput fell to 53 documents an hour, against roughly 250 in theory. Eventually Render suspended the service. A redeploy changed nothing, because nothing about the deploy was wrong.

The logs were the strange part. Each run ended mid-document, with no traceback and no error. A process that crashes usually says why. This one just stopped.

Why there was no error

At first the worker ran four child processes, each handling one document. When a child process runs out of memory, the operating system kills that child, and Celery notices and logs that a worker exited prematurely. That line wasn’t in the logs. What was being killed was the whole container: every child and their parent at once, because together they had gone over the container’s memory limit.

That kind of kill is a SIGKILL. It can’t be caught, so none of the worker’s error handling ran. Nothing marked the document as failed; it stayed in the PROCESSING state. And because the worker acknowledges a queue message only after the document finishes, the message stayed unacknowledged too. The container came back, its children pulled the next documents off the queue, reached the same step, and died again:

flowchart TD
  pull["Each worker child pulls a document"] --> render["Render a page to an image at a fixed zoom"]
  render --> limit{"Container over<br/>its memory limit?"}
  limit -- N --> next["Document finishes"]
  limit -- Y --> kill["The whole container is killed (SIGKILL):<br/>no error handling, no traceback, no acknowledgement"]
  kill --> stuck["Documents stay PROCESSING;<br/>queue messages stay unacknowledged"]
  kill --> restart["Render restarts the container"]
  restart --> pull
  stuck -.->|"redelivered after 46 minutes,<br/>or rescued by the recovery sweep"| pull
  classDef bad stroke:#8fc1f0,stroke-width:2px
  class render,kill bad

Every restart threw away the in-flight work of four documents, and every document big enough to cause the problem came back later to cause it again.

First fix: fewer, shorter-lived workers

The first changes treated the symptom. I cut the worker from four children to two and recorded why next to the setting:

# concurrency=2, not 4: four children each rasterising PDF pages for OCR
# exceeded the container's memory, and the process was being restarted every
# ~5 minutes. Each restart discarded four documents' in-flight work, so the
# lost throughput outweighed the parallelism — 53 documents/hour observed
# against a theoretical ~250. Raise again only alongside a larger plan.
startCommand: "celery -A worker.celery_app worker --loglevel=info --concurrency=2 --prefetch-multiplier=1"

I also had each child replaced after 10 tasks, so memory it accumulated while rendering pages would be returned to the system. The setting’s comment admits the number is mostly theoretical: the container was restarting so often that no child lived long enough to reach a higher one.

Second fix: a pixel budget for reading pages

The real question was why rendering a page took so much memory. The OCR step rendered every page at a fixed 200 DPI. That sounds reasonable for a letter-size mailer, but the archive runs from letter-size mailers to posters and large-format press files, and a fixed resolution scales memory with the page’s physical size.

It was also wasted. Claude’s vision input downscales any image whose long edge is over 1,568 pixels on the model the app uses, so every pixel past that was rendered, held in memory, uploaded, and thrown away. The fix renders each page to that budget instead:

def _ocr_render_matrix(self, page) -> "fitz.Matrix":
    max_edge = settings.ocr_max_image_edge_px  # 1568
    long_edge_pt = max(page.rect.width, page.rect.height)
    if long_edge_pt <= 0:
        return fitz.Matrix(1.0, 1.0)
    zoom = min(max_edge / long_edge_pt, 200 / 72)
    return fitz.Matrix(zoom, zoom)

PDF dimensions are in points, 72 to the inch, so a zoom of 1.0 is 72 DPI. The min keeps small pages at the old 200 DPI and brings only oversized ones down.

The crash loop survived the fix.

The loop came back at the preview step

The thumbnail step had its own copy of the rendering code. Two copies, in fact: the async and sync paths each had a byte-identical version, and neither was touched by the OCR fix. Both rendered the first page at a fixed 2× zoom, to produce a thumbnail no bigger than 300 by 400 pixels.

I measured it on nearly empty PDFs, so the numbers are the rendering alone:

Page size	Pixmap at 2×	Raw pixmap	Peak memory
8.5 × 11 in. mailer	1224 × 1584	5.8 MB	153 MB
24 × 36 in. poster	3456 × 5184	53.7 MB	302 MB
48 × 72 in. banner	6912 × 10368	215.0 MB	815 MB

Peak memory runs far above the pixmap because the pixmap was still alive while Pillow decoded it into a second full-size image. Two children doing that at once was enough to kill the container.

The banner file that produced the last row was 0.8 KB. A PDF describes a page; it doesn’t contain its pixels. The upload size limit I already had bounded nothing here, because a file’s size and the size of its rendered page are unrelated.

The fix gave previews their own budget, 800 pixels on the long edge, twice the thumbnail’s, merged the two copies into one function, and dropped the pixmap before Pillow decodes the image:

pix = page.get_pixmap(matrix=cls._render_matrix(page))
img_data = pix.tobytes("png")
# Drop the pixmap before Pillow decodes the PNG, so the raw
# samples and the decoded image are not both resident.
pix = None

img = Image.open(io.BytesIO(img_data))
img.thumbnail((300, 400), Image.Resampling.LANCZOS)

Peak memory for the banner went from 815 MB to 136 MB, which is the worker’s baseline with its libraries loaded. Page size stopped mattering at all. The thumbnails came out the same size as before.

The deeper lesson was the duplication. The OCR fix didn’t fail; it fixed one of three copies of the same idea. The test that came with the merge asserts that neither entry point renders a page on its own again.

What the loop broke on its way

In June I had built machinery for exactly this situation: a worker dying mid-document. Each document records a heartbeat while it’s processed, and a recovery task runs every two minutes, finds documents whose heartbeat has gone stale, and puts them back in the queue. A lock in Redis stops two workers from processing the same document. And a per-page checkpoint lets a restarted document resume where it left off, instead of paying Claude to read every page again.

That machinery had rarely run before. A crash loop runs it constantly, and it turned out to have four bugs.

The page reader wasn’t lazy. It was written as a generator, but it collected every page’s OCR before yielding the first. The heartbeat is written as each page is consumed, so a long document sent no heartbeat while it was being read, and a healthy document could be rescued as dead while it was still working. The checkpoint also only covered the analysis of each page, not the OCR, so every turn of the loop paid to read every page again.

Resuming truncated documents. A resumed document skipped the pages before its checkpoint, but nothing stored those pages’ results, so there was nothing to restore. A document that died at page 40 of 50 was saved with pages 41 to 50 only. One-page documents lost their analysis instead: the logs show a document resuming from page 1 and then saving with zero keyword mappings. A test asserted that the skipped pages were absent. The bug was the specified behavior.

Checkpoints after the first were never saved. The checkpoint lived in a JSONB column, and the code updated it in place:

def set_metadata(self, **metadata):
    if self.file_metadata is None:
        self.file_metadata = {}
    self.file_metadata.update(metadata)

SQLAlchemy doesn’t see changes made inside a plain JSONB value; it only notices when the attribute itself is assigned. The first checkpoint on a document with no metadata was saved by accident, because assigning {} is a real assignment. Every later one was silently dropped, which is why every resumed document in the logs resumed from “checkpoint page 1”. Clearing the checkpoint when a document finished was dropped the same way. All 17 documents completed during the incident still carried a stale checkpoint, ready for a later retry to resume from and truncate. The fix assigns a new dictionary instead of mutating the old one:

self.file_metadata = {**(self.file_metadata or {}), **metadata}

Reprocessing didn’t clear the checkpoint, so asking for a document to be processed again resumed it instead.

Now the page reader is lazy and can start after a given page, so a resume skips pages before rendering them. Each page’s results are saved with its checkpoint and restored on resume. A checkpoint without saved results restarts the document from page 1, on the principle that paying for a page twice is cheaper than storing a corrupted document. On a six-page test document killed while analyzing page 4, the retry now makes 3 OCR calls instead of 6, and the saved text has all six pages instead of the last three.

A script finds documents the loop damaged: text that begins at a later page, or keywords that have mappings while the analysis has none, a combination a clean run can’t produce.

What’s still open

Image files still have no pixel budget. A large TIFF or PNG is decoded at full size, though Pillow refuses images past about 179 million pixels, which turns the worst case into a missing thumbnail rather than a restart. And I later found one more resume bug: the resume check compares the number of stored pages with the checkpoint’s page number, but blank pages aren’t stored, so one blank page before the checkpoint makes every resume start over from page 1. That one costs money, not data.

The core issue

The trigger was a fixed render resolution meeting a corpus with pages from 8.5 inches to 6 feet. Memory follows the page’s physical size, not the file’s, and a 0.8 KB file was enough to take down the worker.

But the bigger finding was the recovery code. Code that handles crashes runs only when things crash, which in a healthy system is almost never, so it is the least exercised code in the project. Mine had shipped with a test that specified the wrong behavior, a save that never saved, and a page reader that wasn’t lazy. The crash loop was the first real test it ever got.