Implementing the LLM Inference Pipeline

The project started from an abstract goal — "use AI to strengthen abuse detection for escrow-payment evasion" — so the concrete rules for automatically sanctioning detected users had to be defined in collaboration with the internal operations team responsible for user sanctions and dispute resolution. Their sanction rules carried a number of exceptions and conditions, and copying them into an LLM prompt as-is risked confusing the model. So we first structured the exception rules by priority before converting them into a prompt.

Reducing False Positives by Splitting the Task

After building a test dataset for prompt evaluation together with the operations team, we ran prompt engineering aimed at maximizing precision on this dataset to minimize false positives in production. We initially designed a single prompt to judge all sanction rules at once, but precision was hard to improve as exception conditions kept conflicting with each other. Analyzing the false-positive cases showed that detection rules were being applied ahead of certain exception rules, so to resolve this interference we redesigned the system by splitting the single task into 4 independent tasks. With each task making its judgment from its own independent prompt and criteria, the interference between sanction rules was resolved, and we were able to raise precision on the test dataset to the target level.

Why We Chose an Open-Source LLM Over an LLM API

This was a decision that traded off higher token usage from calling the LLM multiple times. On top of that, every image sent in a chat had to be reviewed as well, so the number of tokens needed to review a single chat scaled with the number of images sent. As a result, during a short PoC that used an external LLM API for a gradual rollout covering only 10% of total review volume, the average daily cost turned out higher than expected. Once we projected that cost to the full rollout's call volume, we estimated that API-based operating costs would substantially exceed managed-service hosting costs. On that basis, we looked into hosting an open-source LLM through a managed service.

We limited the candidate models to publicly available as of Q2 2026 with size around 30B parameters with Korean-language capability. There was one task that required image processing, but since we also considered running a separate text model and a separate multimodal model if text performance alone proved sufficient, multimodal support wasn't a selection criterion. Switching from an LLM API to a public model couldn't come at the cost of performance, so we compared each candidate's precision on the same test dataset across all 4 tasks.

Some candidates measured higher recall on certain tasks, but since the system's core goal was minimizing false positives, we adopted gemma-4-26B-A4B-it after confirming it maintained consistently high precision across all 4 tasks. This model also performed well enough to handle the image task with the same model, which let us simplify the system down to running just a single model.