In-Chat Abuse Detection System
I helped build a system that detects escrow-payment-evasion abuse in Bungaejangter's in-chat feature using an open-source LLM. It sanctioned 50% more abusive users than the legacy system, driving a shift to secure payment and growing escrow transaction volume — this walks through how it was built.
Situation
Bungaejangter's secure payment (escrow) system charges sellers a 6% fee on the item price. To avoid this fee, we found a number of sellers sharing personal identifying information — phone numbers, bank accounts — directly in chat. Transactions that bypass secure payment expose buyers to potential fraud, so we had already defined this as abuse and were running detection and sanctions for it. But our existing rule-based detection couldn't keep pace with increasingly sophisticated evasion patterns — coded language, images — so the detection logic needed an upgrade. So we introduced an LLM-based detection system that interprets chat context to judge evasion patterns.
Task
- Contextual reasoning: Evasion patterns evolve in real time, so instead of a system built on a model trained on static data, we needed an LLM-based system that could interpret context and make a judgment call at inference time.
- Precision first: Falsely flagging a normal conversation as abuse causes immediate, direct harm to a good-faith user, while a missed detection can be addressed incrementally later — so the decision was made to prioritize minimizing false positives.
- Operational flexibility: We planned to start with hourly batch inference and gradually move to real-time inference once operations stabilized. That meant we needed an event-driven architecture that would work for both batch and real-time inference.
Action
Result
This section summarizes how the in-chat abuse detection system — built together with the internal operations team responsible for user sanctions and dispute resolution — performed once it went live.
The legacy system relied on regex matching against a database of banned phrases and an OCR model. This approach couldn't handle new abuse patterns that hadn't already been catalogued. The new system instead detects abuse patterns based on the LLM's judgment, which let it successfully catch patterns it had never seen before — during the parallel run, it detected 4.6× more abuse attempts than the legacy system.
After the gradual rollout began, users the LLM system flagged as abusive were sanctioned automatically under existing operational rules. If a sanctioned user appeals, the operations team reviews the case and decides whether to approve it. We defined this appeal-approval rate as the system's false-positive rate and monitored it closely. Because the daily false-positive rate stayed consistently low, we decided to fully replace the legacy system — this system now runs as the standard pipeline judging abuse across every chat on Bungaejangter.
After full rollout, the number of users sanctioned without a successful appeal increased 49.5% over the legacy system. Later analysis showed that some of these users, in subsequent sales, stopped evading secure payment and switched to escrow instead — and that shift produced a meaningful increase in transaction volume. Transactions that had been happening outside the secure-payment system were pulled back inside it, which we consider a meaningful business outcome.