Fruition
2026.04 — PresentLLM Wiki-based AI workspace · AI SW Maestro, 17th cohort · Deployed on AWS EKS, in QA
Team project (3 + designer) / AI lead
How it started: a teammate’s pain point with managing documents, refined through design thinking — saved documents could not be found when needed. Team: me (all AI), 1 backend (Spring backend), 1 frontend & DevOps (frontend, MSA deployment on AWS), external designer · 3 mentors
GitHub · AI repository ↗Improving retrieval for grounded AI answers
Finds the document evidence AI answers need, and attaches none to questions the documents cannot answer.
- Evidence selection for RAG Problem: In a service that answers from documents, retrieval missed evidence and attached evidence to unanswerable questions, undermining trust. Decision: GPT selection cost about 2.5x and took longer; tuning weights and RRF gave similar scores across settings, so I adopted Jev, which picks only from fixed options and had first been validated on request routing (77 → 81 of 98 correct). Solution: Two stages: BGE-M3, the best Korean semantic model in benchmarks, and BM25 for exact terms gather candidates and Jev picks the final evidence. Compared on identical candidates with blind cross-grading, then cut candidates to 300 and compared 4, 6, 8 and 10 parallel batches, choosing 8 with no accuracy difference. Result: Coverage on 80 answerable questions rose from 66.25% to 91.25%. Median selection time 9.090s → 1.137s (still slower than the original 0.431s). Higher evidence reliability gives users a reason to trust answers and keep using the service.
User-defined agent Skills with approval-based execution
Recurring tasks are saved and reused as Skills, and AI-made changes apply only after the user approves them.
- User-defined task instructions (Skills) Problem: Recurring tasks differ by user, so they cannot all be built as features, and AI edits arrived with parts of the request missing, leaving repeatable work to chance. Decision: AI drafts instructions, code checks permissions and scope, and the user decides publishing and applying. Code format checks missed omitted requests, so I added LLM semantic evaluation. Solution: Authoring, review and publishing of natural-language instructions with checks against bypassing permission or approval; preview and separate approval before changes; semantic evaluation with at most one rewrite. Generation, evaluation and rewrite run as LangGraph nodes for step-level tracing, and published Skills are stored as immutable versions in PostgreSQL. Result: Edit quality 94 → 104 of 114 drafts (82.5% → 91.2%); documents stayed unchanged before approval. Stable results keep prepaid credits from going to rework.
PDF document conversion
Converted uploaded paper PDFs into Markdown that keeps tables and equations, so they can be searched and cited in answers.
- PDF → Markdown conversion Problem: Broken tables and equations in paper PDFs made the figures behind Wiki pages and answers unusable. Decision: Repairing only detected damage missed errors, and full AI conversion lacked table/equation accuracy and was costly, so I split processing by content type. Body text uses fast AnyDoc extraction; repair moved from an sLLM to commercial LLM APIs to avoid GPU server costs. Solution: Extracted body text with AnyDoc and repaired only garbled parts with AI, read table/equation regions separately, and kept original figures. Result: Pass rate on the same 30 pages (445 units, blind) rose from 45.17% to 89.89%. Key figures now enter the Wiki intact, reducing quality-driven churn.
MSA: Splitting only by feature would keep DB load the same while adding hops, so I proposed splitting by data ownership and co-designed boundaries and communication with teammates and mentors.
All figures are from internal evaluations.
See the detailed portfolio for the problem-solving process and experimental results.
RAG evidence selection ↗ Agent execution environment ↗ PDF conversion ↗
