Building the LLM Inference Infrastructure

The overall architecture was designed in collaboration with a senior engineer on the team. I owned implementing the Batch Job that collects chat information to review, and the inference pipeline inside the LLM Worker.

Batch Job

sequenceDiagram
    autonumber
    participant Scheduler as Airflow
    participant Batch as Batch Job
    participant Athena
    participant SQS

    Scheduler->>Batch: Triggers hourly
    Batch->>Athena: Request chat info to review
    Athena-->>Batch: Chat metadata to review<br/>(e.g. chat room ID, start/end time)
    Batch->>SQS: Publish event with metadata
          

Chat logs were already landing in our internal data lakehouse, so we queried them through Athena. If we'd stayed with batch calls, the simplest approach would have been to upload the hourly-collected chat information to S3 and trigger the LLM worker off that — but because we ultimately planned to move to real-time inference, we adopted an event-stream-based architecture instead. Building the worker to subscribe to SQS meant that when the time came to switch to real-time inference, we'd only need to swap the event publisher — from this batch job to, say, the chat service's own event-emission logic. We also chose standard SQS over Kinesis Data Streams or SQS FIFO as the broker, since the pipeline's tasks run sequentially within a single worker (so there's only one subscriber), and each event is an independent chat-history unit that doesn't need ordering guarantees.

LLM Worker

sequenceDiagram
    autonumber
    participant SQS
    participant Worker as LLM Worker
    participant Buntalk as Bungaejangter API
    participant LLM as LLM Endpoint
    participant DB as MySQL

    Worker->>SQS: long-poll
    Worker->>Buntalk: GET chat history for event
    Buntalk-->>Worker: Chat history (received, preprocessed)

    loop Inference pipeline (4-stage task)
        Worker->>LLM: Chat history
        LLM-->>Worker: Per-task inference result
    end

    Worker->>DB: Upsert abuse review result
    Worker->>SQS: Delete the event
          

Events published to SQS are picked up by the LLM worker and processed in the order shown in the diagram. This workflow is designed around three axes — reliability, scalability, and fault tolerance:

1. Duplicate-processing prevention: Because processing a single event can take a while (the pipeline calls the LLM multiple times), we set SQS's visibility timeout generously long to prevent an event from being processed twice.
2. Scaling: Event volume can spike up to 20× between the quietest hours and peak traffic, but since a single LLM worker pod needs very few resources, we simply run enough pods to cover peak traffic rather than relying on autoscaling.
3. Fault tolerance: If the open-source LLM inference endpoint stops responding for any reason, we fall back to an external LLM API endpoint we had already benchmarked during the PoC.

LLM Endpoint

When the feature first launched, we ran the LLM endpoint on a managed service to avoid the operational burden of self-hosting an open-source LLM. But when that provider announced it would end support for our current hardware tier and force a migration to a higher spec by the end of the year, we recognized the risk of being passively locked into a provider's roadmap for both pricing and expected performance. To secure a runtime we could control, we benchmarked self-hosting the same open-source LLM in a vLLM container on a GPU instance against the managed service, comparing performance and quality. The self-hosted setup was measured under two configurations depending on quantization — BF16 and FP8 — for three setups total.

On performance, the self-hosted setups measured meaningfully faster than the managed service.

Breaking down the wall-time gap by task in the pipeline showed it was concentrated in the task requiring image processing. That task alone had a p50 TTFT several times higher than the other setups, while the gap for text-only tasks stayed fairly consistent. Since the managed service didn't offer a Korea region, we attributed this gap to the network cost of fetching image URLs from the managed service. We concluded this gap would remain structurally unresolved unless the managed service offered region selection.

On quality, we judged all three setups equivalent within margin of error, and concluded that migrating to the self-hosted, FP8-quantized setup would be effective. Also, unlike when we ran this benchmark, the vLLM image now supports speculative decoding (MTP) for this LLM — and the fact that a managed service's adoption of such optimizations is entirely at the provider's discretion and timeline made migration even more attractive. For now, the managed service still has better price competitiveness, so we haven't rushed the migration — but when the announced forced migration takes effect, we plan to switch to self-hosting on the strength of this benchmark.