Improving the Text Encoder (v2)

Why We Use a Custom Text Encoder

From the very first version (v1), the text encoder that vectorizes product titles wasn't a public Hugging Face model with continued training — it was our own model, pretrained directly on Bungaejangter product text. General public models are trained on data that doesn't sufficiently capture Bungaejangter's domain vocabulary, including abbreviations, slang, brand names, and expressions specific to secondhand trading. There had, of course, been an attempt to solve this by continuing training a public model on Bungaejangter data — in fact, the team was running a text encoder called ib3-base at the time, built by continuing training Microsoft's multilingual-e5-base on Bungaejangter text. But continuing training on a public model carries two structural limitations.

1. Inefficient tokenizer: ib3-base's tokenizer, trained on a multilingual corpus, didn't sufficiently merge Bungaejangter's commerce vocabulary. For example, '스니커즈' (sneakers) gets split as finely as ['▁스', '니', '커', '즈'], degrading semantic preservation and increasing compute, since encoding the same text now requires more tokens.
2. Resource-optimization constraints: Even with continued training, a model still inherits its backbone's size. So ib3-base, like the multilingual-e5-base encoder it started from, carried 278M parameters. Using a public model as the backbone means the backbone's size sets a hard floor on operating cost. Designing a lightweight model from scratch with self-pretraining in mind, on the other hand, lets you take instance size and inference cost much lower.

So we preprocessed titles and descriptions from Bungaejangter products listed over the past 5 years into a training dataset of roughly tens of gigabytes, and pretrained a lightweight encoder (11M parameters) that met our operating-cost constraints, following the RoBERTa paper.

Redesigning the Text Encoder

Later, we discovered a bug where the tokenizer training had omitted an alphabet-size parameter, causing Korean syllables that appear frequently in product titles to be dropped from the alphabet list — which produced a flood of UNK tokens. In practice, the syllables '뗑' and '킴,' needed to spell the brand name '마뗑킴,' were missing from the alphabet list, and the name was being tokenized as ['_마', '<UNK>', '<UNK>']. This need to retrain the tokenizer from scratch became the trigger for a broader rewrite, improving the existing pretraining logic along two axes.

1. Swapping the Pretraining Task (RoBERTa → ELECTRA)

RoBERTa's MLM (Masked Language Model) task only uses 15% of the tokens in a dataset for training, while ELECTRA's RTD (Replaced Token Detection) task can use every token as a training signal. Since our training data was only tens of gigabytes — not especially large — we judged that a task extracting more training signal from the data we had was the better fit.

2. Domain Alignment via Supervised Contrastive Learning (Supervised SimCSE)

After pretraining, we collected (search query, product name) pairs from a year's worth of "searched, then clicked" logs and ran an additional round of supervised SimCSE training. This reinforced the training signal once more, pulling semantically similar text toward nearby embeddings. We called this two-stage-trained text encoder emb6.

Validating the Redesigned Encoder

To validate emb6's performance, we compared its retrieval quality against ib3-base. Once both models had finished training on data up to before August 2025, we collected a day's worth of (search query, product name) pairs from September. After preprocessing, we measured HR@k on a retrieval task — for each search-query embedding, retrieving the clicked product out of several-million-scale unique pairs and their roughly one-million unique product names.

Even though ib3-base was trained with several retrieval-specialized tasks folded into its continued training on Bungaejangter text, it performed on par with emb6 — roughly 1/20th its size — on Bungaejangter's own retrieval task. This let us draw two conclusions.

1. Given the nature of multilingual models, a large share of ib3-base's 278M parameters exist to serve other languages and domains. The backbone model's own Hugging Face page notes that low-resource languages can underperform, and that Korean — trained on far less data than English — may not perform as well.
2. So, for the same retrieval task within the same domain, a small model designed from scratch for that domain and task can be more efficient than a huge general-purpose model.

We also ran a load test on identical infrastructure (AWS Graviton c7g.large) to compare the two models' processing cost. ib3-base hit CPU saturation at a lower concurrent-request count than emb6, and its per-request CPU cost measured consistently about 4–5× higher than emb6's across every load level we tested (1–5 concurrent users). In other words, the two models retrieve at parity, but ib3-base needs roughly 4–5× the CPU resources to do it. On the strength of this conclusion, we adopted emb6 as this recommendation system's product-name text encoder.