Parallel document analysis: 282.11s → 73.89s

01 Background

Sequential processing made later documents wait for external AI responses, and more workers also mean more API calls and memory, so I had to find how far to scale. I used Kafka because it can replay messages and keeps order within a key, so long AI jobs run asynchronously while IDs prevent duplicates. 1 → 4 Workers cut analysis from 282.11s to 73.89s. Raising concurrent requests per Worker from 2 to 4 slowed it to 95.92s, and two embedding jobs on one GPU stretched each from about 9.5s to 19–21s, so the two stages got separate scaling rules.

02 Experiments and improvements

03 Improved processing flow

flowchart TD
 A[Document processing request] --> K[Kafka distributes document jobs]
 K --> W[Multiple workers with per-worker request limits]
 W --> L[Lock writes within the same workspace]
 V[Embedding generation] --> G[Check contention on a shared GPU]
 L --> D[Check time, memory and content preservation]
 G --> D
 D --> R[Choose scaling policy per stage]

flowchart TD
 A[Document processing request] --> K[Kafka distributes document jobs]
 K --> W[Multiple workers with per-worker request limits]
 W --> L[Lock writes within the same workspace]
 V[Embedding generation] --> G[Check contention on a shared GPU]
 L --> D[Check time, memory and content preservation]
 G --> D
 D --> R[Choose scaling policy per stage]

04 Results and lessons

73.8% less analysis time for 4 documents, 300 queued jobs with zero loss

With 4 workers and 2 concurrent requests each, total processing time for 4 documents fell by about 73.8%. Combined process memory rose from 351.7MiB to 1,401.0MiB, about fourfold. Of 16 facts fixed in advance, 10, 14 and 11 were fully preserved with 1, 2 and 4 workers, so although each condition ran once on a small sample, I did not treat quality as identical. I excluded raising concurrent requests further and parallelising embedding generation on the same GPU, since both made things slower. External-response waits and device contention are different bottlenecks, so each stage needed its own scaling approach. Replaying saved AI responses for 100 documents under 3 configurations processed all 300 jobs without omissions or failures. Judging that the memory increase could be absorbed by distributing AI functions on EKS as MSA, I used at most 4 workers for parallel document processing.