Building the CPU Inference Infrastructure

Building on an architecture designed by a senior engineer on the team, I owned implementing the batch job that indexes product vectors and the embedding API. Working through the implementation, I came to understand the reasoning behind each component's design, which I've laid out below.

Product Vector Indexing Batch

sequenceDiagram
    autonumber
    participant Dev as Developer<br/>(CI/CD)
    participant ECR
    participant Airflow as Airflow<br/>(MWAA)
    participant SM as SageMaker
    participant RT as SageMaker<br/>Training Job
    participant PDB as Product DB
    participant VDB as Vector DB

    rect rgb(245, 245, 245)
        Note over Dev,ECR: Once, on batch code changes
        Dev->>ECR: Build & push batch image
    end

    rect rgb(245, 245, 245)
        Note over Airflow,VDB: Repeats hourly
        Airflow->>SM: CreateTrainingJob<br/>(image URI, job code)
        SM->>ECR: Pull image
        ECR-->>SM: Image
        SM->>RT: Run container
        activate RT
        RT->>PDB: Fetch recommendable products
        PDB-->>RT: Product list
        RT->>RT: Convert products to vectors
        RT->>VDB: Upsert & delete vectors
        deactivate RT
    end
          

Most products registered in Bungaejangter's product DB sit unclicked and untouched for a long time — items with real demand sell quickly. So we built internal logic that treats the sellable-products DB as effectively defining the catalog of products that can actually be recommended, and built a batch task that converts newly listed or modified products within it into product vectors and indexes them in the vector DB. We defined an Airflow DAG that triggers this hourly as a SageMaker Training Job. Even though this is a batch job, not model training, the team used Training Jobs for it because they were already a familiar, conventional way to run a container with a given image, parameters, and instance spec.

To choose a vector DB service to store this catalog and support real-time similarity search, the team compared three options: Aurora PostgreSQL (pgvector), Milvus, and S3 Vectors. Since our filtered, recommendable-product catalog was on the order of millions of items, all three options performed essentially the same. With performance a non-factor, the comparison came down to operational burden and cost — and the team decided on Aurora.

Aurora vs. Milvus — When Performance Is Equal, Pick the One That's Easier to Manage

Running Milvus in production would require managing several different node types and storage layers on top of a Kubernetes cluster. Adopting it would have handed the DevOps team a set of new management touchpoints — deployment, monitoring, version upgrades — for each component. Adding the pgvector extension to our already-standard RDS cluster, by contrast, gave us vector search with no new learning curve, and that convenience was the basis for choosing Aurora. Granted, if the catalog keeps growing large enough that the vector index no longer fits in cluster memory, we'll need to revisit alternatives like Milvus — but for any foreseeable future, we judged Aurora was enough.

Aurora vs. S3 Vectors — When Management Difficulty Is Equal, Pick the Cheaper One

Both options are fully managed, so neither required standing up any new infrastructure — management difficulty was equally low. But this recommendation system is called from both the home and product-detail recommendation surfaces, so request volume was high. S3 Vectors' pricing model (request count plus scanned data size per request) risked scaling up quickly under high-QPS workloads compared to Aurora's fixed hourly instance pricing. To validate this risk, we ran a two-week PoC after S3 Vectors launched: building the same vector catalog in S3 Vectors and routing 10% of the recommendation system's traffic to it. Search latency and recommendation CTR were similar between the two services during the PoC, but extrapolating the measured cost at 10% traffic to 100% put S3 Vectors' projected monthly cost at roughly 3× Aurora's. On that basis, we confirmed the original plan of staying on Aurora.

Product Vector Embedding API

sequenceDiagram
    autonumber
    participant Client
    participant RecAPI as Recommendation API
    participant EmbAPI as Embedding API
    participant VectorDB as Vector DB

    Client->>RecAPI: Clicked product info<br/>(e.g. product name, category)
    RecAPI->>EmbAPI: Request embedding
    EmbAPI-->>RecAPI: Return embedding
    RecAPI->>VectorDB: Search for similar vectors
    VectorDB-->>RecAPI: Return search results
    RecAPI-->>Client: Return recommendation results
          

Our internal backend standard stack is Kotlin-based Spring Boot, but the embedding API needed a Python ML stack — PyTorch, transformers — to convert product features into vectors, so we built it as a separate service. We packaged the embedding API, built with FastAPI, running as a single uvicorn worker process managed by gunicorn, into a standard-format image and uploaded it to ECR; the DevOps team that manages our internal EKS cluster then tuned an HPA scaling policy based on this image and the expected peak RPS from the recommendation surfaces. As we developed and improved the lightweight encoder, throughput on the same CPU node pool increased, which in turn reduced the replica count and instance footprint needed for production serving — letting us translate the algorithm improvement directly into lower production operating cost.