01 Background
Rewriting about 44,000 drug records (about 1GB) as-is was estimated to cost about $200 in API calls. I needed a way to explain every drug within budget without cutting the number of drugs covered.
I judged that if conversion cost exceeded the budget, we would have to cut the drugs we explain and narrow the service. Lowering the cost per conversion, I thought, would let the same budget cover every drug.
02 Problem analysis and implementation
Removing exact duplicate sentences
Problem Converting about 44,000 records totaling 1GB was estimated to cost about $200 in API usage. I needed to cover all drugs within budget.
Experiment and result Examined 100 records and found about 70% duplicate content. Used this sample and a work plan to secure 7 days for preprocessing. Split text into sentence blocks, merged exact matches and preserved links from each drug to its blocks.
Decision Convert repeated sentences once and reuse them rather than cover fewer drugs.
GPT Batch API conversion and mapping results back to drugs
Problem Deduplication reduced input, but I still needed to minimize model costs.
Experiment and result The work did not require immediate responses during service use, so batch waiting was acceptable. Converted unique sentences with GPT Batch API, then reused the preserved drug-to-block mappings to attach results to each drug. Actual API spending was $11.18 versus about $200 estimated for all original text.
Decision Combined deduplication and batch conversion to deliver the feature without reducing drug coverage.
2026 PySpark reproduction: measuring ending normalization and similar-sentence merging
Problem Part of the 2025 code and the raw files no longer existed, so the effect of stages after exact deduplication could not be shown in numbers.
Experiment and result Re-collected 42,714 products from the MFDS public API, split them into 3,299,602 blocks following the XML structure, and reproduced normalization and exact deduplication in PySpark. Then applied an ending-normalization dictionary and MinHashLSH (character 3-grams, Jaccard distance), comparing reduction and merged samples at thresholds 0.1, 0.2 and 0.3. PySpark split about 3.24 million blocks across cores and can scale to a cluster as data grows.
Decision At 0.3, text shrank 18.78% but sentences about different conditions (Lapp vs. total lactase deficiency) merged, and at 0.2 some precaution sentences were lost. Wrong merges are costly for medical information, so I chose 0.1 (a further 6.69%) with guards that merge only when numbers and applicability conditions match.
03 Improved processing flow
flowchart TD A[Original drug text: about 1GB] --> B[Split into sentence blocks] B --> M[Keep drug ID to block ID mappings] B --> C[Merge exact duplicate sentences] C --> G[Convert unique sentences with GPT Batch API] G --> R[Map converted text back to each drug] M --> R C -.-> S[2026 PySpark reproduction, measured ending normalization and MinHashLSH]
Converted each repeated sentence once and kept per-drug mappings to attach results. In 2026 the same process was reproduced in PySpark and measured.
View Mermaid source
flowchart TD A[Original drug text: about 1GB] --> B[Split into sentence blocks] B --> M[Keep drug ID to block ID mappings] B --> C[Merge exact duplicate sentences] C --> G[Convert unique sentences with GPT Batch API] G --> R[Map converted text back to each drug] M --> R C -.-> S[2026 PySpark reproduction, measured ending normalization and MinHashLSH]
04 Results and lessons
Actual API spend $11.18, PySpark reproduction cut text by 94.1%
Completed conversion for $11.18 in actual API spending versus about $200 estimated for all original text. Reusing converted unique sentences kept the feature within budget without reducing the set of drugs covered. Part of the 2025 preparation code no longer existed, so in 2026 I re-collected 42,714 products (960MB of plain text) from the MFDS public API and reproduced the same preparation in PySpark. 3,236,626 blocks reduced to 176,648 unique blocks, cutting text from 744MB to 44.1MB (−94.1%). An ending-normalization dictionary then removed 0.78%, and MinHashLSH similar-sentence merging a further 6.69%.
About $200 was the upfront estimate; $11.18 was actual spending. Seven days of preprocessing labour are not included. The 2026 figures come from re-collected data, not the 2025 dataset. The reproduction code was written with AI coding tools; I designed the experiment and chose the thresholds.