Recommendation systems are mostly ranking, not LLMs

At Gravity9, where I work as an AI Lead Consultant, we built a recommendation system for an influencer-marketing platform. It replaced a manual process that could take up to a month with one that produces a creator shortlist in minutes.

What makes the project a useful example of building with AI is not that it uses a large language model everywhere. It is that each part of the problem is handled by the mechanism best suited to it. Language models interpret and explain; vector search retrieves; a model trained on historical outcomes decides the order.

This article walks through that architecture, the choices behind it and the offline results. The central lesson is simple: a recommendation system may look like an LLM feature from the outside, while most of the intelligence inside it comes from retrieval, ranking and measurement.

How this AI recommendation system was designed

An influencer-marketing platform sits between brands and creators. A brand arrives with a campaign, the platform proposes a shortlist of creators, the brand picks some, money changes hands. The shortlist is the product.

The ask was not "add AI to this". It was to reproduce the judgement experienced account managers were already making, quickly enough to matter, and to prove it had been reproduced. That last clause did most of the architectural work.

Proving it required an answer key, and the platform already had one. The most valuable asset was not the creator database but the record of who had been hired before. Roughly nine thousand historical campaign-creator outcomes formed a labelled dataset hidden inside an operational table.

That answer key changes the job. Instead of producing plausible recommendations, the system can be scored against decisions that actually happened. Two hundred campaigns were held out for evaluation; nothing about them reached the trained model.

The resulting recommendation pipeline has five stages. Each stage has one job, making it easier to test and improve independently.

How an LLM turns campaign briefs into structured data

A campaign brief arrives as prose written by a human for a human. Fitness and wellness, audience skewing UK and Ireland, no one who has posted for a competing supplement brand in the last six months, budget in a range, tone described in adjectives rather than numbers.

The first stage turns that prose into a schema: a predictable set of fields for categories, geography, audience-size band, exclusions and budget bounds. It also creates a short intent string used to search creator profiles.

This is schema filling, not comprehension. Described as extraction, it can be tested: collect real briefs, record the expected structured objects and run them on every deploy. Drift then appears as a failing test rather than a vague decline in recommendation quality.

Claude Haiku 4.5 does the extraction; longer or contradictory briefs route to Sonnet 5. This difficulty-based routing requires a difficulty signal, a fallback path and visibility into which tier answered.

This stage will sometimes be wrong, and the damage has a direction. A misread filter narrows the candidate pool, and a candidate excluded here can never be recovered by anything downstream. That single fact sets the rule for the next stage.

Vector search retrieves candidates but does not rank them

Stage two pulls a candidate pool out of the roster using MongoDB Atlas Vector Search over Voyage-4-large embeddings of creator profiles, matched against the intent string from stage one.

One thing about this stage gets turned around constantly. Retrieval is tuned for recall, not precision. Its job is to avoid losing the right candidate, not to put the right candidate first. A pool with a hundred mediocre creators and the correct one buried at position eighty-three is a success at this stage, because two stages downstream exist purely to fix ordering. A pool that missed the correct creator entirely is a failure that nothing can repair, no matter how good the ranker is.

Retrieval is therefore graded on recall at k: how many relevant candidates appear within the first k results. Precision belongs to the next stage.

Learning to rank with LightGBM and LambdaRank

This is the stage that does the matching, and it is the stage that never appears in a demo, because a bar chart of feature importances does not look like the future.

Learning to rank is supervised machine learning where the thing being learned is an order rather than a score or a class. You give the model a set of queries, a set of candidates for each query, features describing each query-candidate pair, and a label saying how good that pairing turned out to be. The model learns a scoring function whose sort order reproduces the labels.

That framing sounds small until you notice what it excludes. It is not similarity search with extra steps, because similarity has no idea what a good outcome looks like on this platform. It is not classification, because classification treats each candidate independently and ranking only exists relative to the rest of the list. The unit of learning is the list, not the item.

How LambdaRank optimises recommendation rankings

A ranking model should care more about mistakes near the top of a list than mistakes near the bottom. NDCG, or normalised discounted cumulative gain, measures exactly that by rewarding relevant results more when they appear in higher positions.

LambdaRank trains by comparing pairs of results and weighting each mistake by how much swapping the pair would change NDCG. LightGBM's lambdarank objective implements this idea inside gradient-boosted trees, so it learns which candidate should appear first rather than predicting an isolated value.

How to structure learning-to-rank training data

The ranking data format differs from classification in one important way.

One row per query-candidate pair. Rows belonging to the same query must sit contiguously. A separate array tells the library how long each query's block is. That array is called group in LightGBM, qid in XGBoost, group_id in CatBoost. It is the only thing telling the model which candidates compete with each other, and getting it wrong produces a model that trains without error and ranks like noise.

# One row per (query, candidate) pair.
# Rows for the same query are contiguous, and `group` gives block lengths.

X = [
    # sim,  rerank, bm25, recency, past_category_fit
    [0.81,  0.94,   12.3, 0.20,    1.0],   # campaign A, creator 1
    [0.77,  0.55,    9.8, 0.90,    0.0],   # campaign A, creator 2
    [0.62,  0.31,    4.1, 0.55,    0.0],   # campaign A, creator 3
    [0.88,  0.71,   15.0, 0.10,    1.0],   # campaign B, creator 4
    [0.59,  0.22,    3.7, 0.75,    0.0],   # campaign B, creator 5
]

y     = [3, 1, 0, 3, 0]   # relevance: 3 = hired, 1 = shortlisted, 0 = passed over
group = [3, 2]            # first 3 rows are campaign A, next 2 are campaign B

ranker = LGBMRanker(objective="lambdarank", metric="ndcg")
ranker.fit(X, y, group=group)

Three things in that snippet do the real work.

The features are numbers, not prose: vector similarity, keyword overlap, recency, historical category fit, engagement rate and audience overlap. That makes the ranker's behaviour inspectable feature by feature and cheap to run over thousands of candidates.

The labels are outcomes, not opinions. A hired creator is 3, shortlisted but not booked is 1, and passed over is 0. Nobody had to annotate them: the model learns from commercial decisions the business already made.

The grades are graded. Binary labels throw away most of the signal in a ranking problem. The gap between hired and shortlisted is real information about how a good match differs from an acceptable one, and NDCG is built to consume exactly that.

Feature engineering without data leakage

Most of the work in a ranking project goes into features, not the model. Some come from earlier stages, such as vector similarity. Others encode business meaning, such as whether the creator was active during the relevant window or how they previously performed in the category.

The trap is leakage. A field populated after a creator is booked is not a feature; it is the label in disguise. Every feature must be computable when the brief arrives, using only what was known then. Reconstructing historical features is tedious, but it separates trustworthy metrics from flattering fiction.

Why use learning to rank instead of an LLM?

An LLM can rank a list, but there are three reasons not to make it the default here.

A language model has a general, internet-shaped notion of a good creator match. The trained ranker is fitted to what this business has actually paid for. The decisive difference is not model size but what each system has been shown.

The second part is that you cannot audit it. When a creator ranks fourth and an account manager wants to know why, a tree ensemble gives you a real answer, decomposed by feature. A language model gives you a fluent post-hoc story about its own reasoning, which is a different kind of object and not one you should put in front of a client.

Third, LLM ranking can be order-sensitive: shuffle the same candidates and the result may change. Ranking needs a deterministic score per candidate.

The claim is narrow: when you have outcome data, prefer a model fitted to it over one that has never seen it.

Why recommendation systems use retrieval and reranking

The pipeline runs a second, different ranker over the top results. Each stage handles fewer candidates and looks harder at each one.

Vector search scores the whole roster cheaply using precomputed embeddings. Its weakness is that each profile was compressed into a fixed vector before the query existed.

LightGBM scores the retrieved pool, because a tree ensemble over a handful of features is cheap. It knows the platform's hiring history and is blind to the actual text.

Voyage rerank-2.5 then scores only the top slice. As a cross-encoder, it reads the query and candidate together, catching matches that a fixed embedding can flatten away. That deeper comparison is expensive, so you feed it tens of candidates, not thousands.

The two rankers are wrong in different directions. The learned ranker knows the business but cannot read. The reranker reads well but knows nothing about the business. That complementary failure is why stacking helps.

A rerank score can instead become a feature in the learned ranker, but then every training pair must be cross-encoded. Applying it afterwards is cheaper, though it can only reorder candidates the first ranker promoted. The available scoring budget decides the trade.

Use LLMs to explain recommendations, not rank them

The second and last language model call takes the finished ranking and writes, for each candidate, why they are on the list. Category history, audience alignment, the specific thing about this creator that fits this brief. An account manager reads it, edits a line, and sends it to the brand.

This is genuinely language-shaped work: prose for a specific audience, where a slightly clumsy sentence costs little. The important detail is the ordering.

Stage five runs after the order is final. A hallucination can change an explanation, but it cannot move a candidate. There is no path from a generated token back into a ranking position.

If instead you ask a model to pick five candidates and explain them, a hallucination becomes a decision. The invented reason and promoted candidate are the same event because one forward pass controls both.

This safety comes from pipeline shape, not prompting. Prompt instructions are requests; pipeline order is a constraint.

The explainer also receives the feature values behind each result, so figures quoted in its prose can be checked programmatically. That catches the fabrication that matters: invented specifics.

How to evaluate recommendation system ranking quality

Both published metrics were computed over two hundred campaigns held out of training and scored against what the business actually did.

The split must be by campaign, not by row. If candidates from one campaign appear in both sets, the model has seen part of the list it is being asked to order and the result is meaningless. The training group is also the unit of evaluation.

Hire recovery, 67.88%. Of the creators genuinely hired in those campaigns, this fraction was surfaced by the pipeline. It answers the central question: would the system have found the people the humans found?

The original decisions also used relationships, phone calls and context absent from the feature vector. Recovering two-thirds of a relationship-driven decision from structured features is the real claim.

Mean reciprocal rank, 0.877. For each campaign, take one divided by the position of the first correct candidate, then average the results. A score of 0.877 means the first genuinely good candidate is usually first and occasionally second.

MRR only sees the first hit. If a campaign hired five creators, the other four are invisible to it. Paired with hire recovery, the metrics say the pipeline finds most of the right people and puts one near the top. Either alone would be a partial truth.

Match the metric to the answer: MRR when one right result is enough, NDCG at k when graded ordering matters, and recall at k for retrieval. Measure retrieval separately because retrieval and ranking failures need different fixes.

No model judges model output in this evaluation. The answer key is a history of commercial decisions and every metric is deterministic. It can run nightly and trace a regression to a commit.

Grade a system like this the way you would grade a search engine, not the way you would grade a chatbot. The discipline already exists, it is decades old, and it has libraries.

How to build a simple recommendation system

The architecture survives being shrunk. Label a small set of queries and correct results, measure vector retrieval, add a local cross-encoder, then train a LightGBM lambdarank model from similarity scores and structured features. Evaluate each stage separately with recall, NDCG and MRR.

If the learned ranker does not beat reranking alone, that is still a useful result: you probably need more outcome data, not a larger model. Measurement tells you which component has earned its place.

Draw one AI feature you have shipped as boxes. Label each box with its actual mechanism, then ask whether it can be scored offline against something that already happened. Where the answer is no, start by finding the outcome data you have not yet treated as labels.

Sources

• gravity9, Influencer AI Matching Engine case study. The published figures used throughout: up to a month of manual shortlisting, an eleven-week build, roughly 9,000 historical hiring outcomes, 200 unseen campaigns, 67.88% hire recovery and MRR of 0.877.
• Burges, C., From RankNet to LambdaRank to LambdaMART: An Overview, Microsoft Research, MSR-TR-2010-82, 2010. Where the lambda gradient trick comes from.
• LightGBM parameters documentation, for the lambdarank objective and the group array.
• XGBoost learning-to-rank tutorial, for the same data shape expressed as qid.
• ranx, for computing NDCG, MRR, MAP and recall without a compile step.
• sentence-transformers, for running open-weight embedding models and cross-encoders locally.
• Voyage AI documentation for embeddings and rerankers, and the pricing page the September 2026 figures came from.