N=1 personalization: Replacing 25 cohorts with user-level ranking
From cohort-based defaults to true user-level personalization at Cars24 UAE: +150% Click Recall@50, +16% buyer conversion, and solving cold-start for 35% of the user base.
Contents
At Cars24 UAE, every user in a demand cohort saw the same ranking of roughly 10,000 used cars. Two people who both liked SUVs but disagreed on everything else (one wanted a 2022 Tucson under AED 80,000, the other a 2019 Pajero under AED 50,000) got identical recommendations. Twenty-five cohorts were not enough to tell them apart, and about 35% of users weren’t in a cohort at all.
This post describes how we replaced cohort-level ranking with N=1 personalization: a two-stage system in which a Two-Tower retrieval model generates candidates for each individual user and a LightGBM re-ranker orders them. It covers the architecture, the five-gate pipeline we used to ship it safely, how we extended it to users with no click history, and what I’d change.
I joined Cars24 as a Data Scientist in January 2024 and rebuilt the UAE recommendation engine end to end, from data sourcing through A/B testing, full rollout, dashboards, and maintenance. The system went live for 100% of UAE users on all platforms (app, mobile web, and desktop) on 12 September 2024. Cars24 later adopted it in Australia, Thailand, and India, where I worked with the local data science teams to port it.
After full rollout, the system delivered the following results:
- Click Recall@50: 12% → 30% (+150%)
- Buyer conversion (U2BI): +16%
- View-to-buy intent (V2Bi): +21%
- Cold-start coverage: 65% → 100% of relevant users
Background: why cohorts stop working#
Cars24’s existing personalization stack was about three years old when I started. It was a two-stage system: unsupervised clustering assigned each user to one of 25 demand cohorts, and a supervised ranking model per cohort ordered cars to maximize booking likelihood. The clustering used clickstream data (impressions, clicks, searches, filters, wishlists, gallery views, and inspection-report views) to segment users.
The system had been iterated on several times and worked reasonably well. It had three structural problems:
Granularity ceiling. Preferences diverged enough within each cohort that the cohort ranking was a compromise for everyone. The “mid-segment SUV” cohort contained users who wanted a budget family car and users who wanted a premium compact SUV. No single ranking satisfies both.
Coverage gap. The cohort system personalized about 65% of users: those with enough clickstream history to be assigned a cohort confidently. The remaining 35% got default sorting. These weren’t marginal users. They included every new user and every low-engagement returner, the people who most needed help finding a car.
No user-level signal fusion. Cohort assignment was a hard partition. A user’s search for “Honda Civic 2021” and their filter for “under AED 60,000” were used for clustering but lost in the aggregation. The ranking model saw the cohort label, not the individual signals.
Adding more cohorts wouldn’t fix any of these. The fix was to remove cohorts entirely and rank for each individual user: N=1 personalization.
Architecture overview#
The system is a two-stage retrieval-and-ranking pipeline. The first stage generates a broad candidate set from learned user and item embeddings. The second stage re-ranks those candidates with a feature-rich gradient-boosted model. Scores are published asynchronously and consumed when a listing page renders.
graph LR
subgraph "Data layer"
A["Clickstream\n(GA4)"] --> FE
B["Search & Filter\nImpressions"] --> FE
C["Car Attributes\n& Inventory"] --> FE
D["Demand Signals\n(I2V, V2BI, STR)"] --> FE
end
FE["Feature\nEngineering"] --> TT["Two-Tower\nRetrieval Model"]
TT --> HNSW["HNSW Index\n(ANN)"]
HNSW -->|"Top-K candidates"| LGB["LightGBM\nRe-ranker"]
LGB --> PS["PubSub\nScore Publishing"]
PS --> LR["Listing\nRender"]
class TT amber
class HNSW violet
class LGB blue
class PS greenWe chose two stages over one for three reasons:
Scale. Cars24 UAE had about 10,000 active listings and more than 10 million monthly sessions. Scoring every user against every car with a full-featured model at request time doesn’t fit in a 100 ms latency budget. The Two-Tower model produces embeddings offline or in near real time, and Hierarchical Navigable Small World (HNSW) retrieval runs in under a millisecond. Only the top-K candidates reach the expensive re-ranker.
Feature richness. The Two-Tower model works on features that encode well into dense embeddings, such as behavior patterns and car attributes. The re-ranker works on features that don’t embed well but matter for ranking: real-time demand, price competitiveness, inventory freshness, and position in the sales funnel. Two stages let each model do what it’s best at.
Decoupled iteration. I could improve the retrieval model and the ranker independently; a better embedding didn’t require retraining the ranker, and vice versa. As the only data scientist on the UAE team, I couldn’t run parallel experiment tracks, so sequential, decoupled iterations were the only practical option.
Two-Tower retrieval: learning user-car affinity#
The Two-Tower architecture, also called a dual encoder, learns separate embeddings for users and items and scores relevance as the similarity between them in a shared latent space. It’s the same architecture behind YouTube’s candidate generation and most large-scale retrieval systems.
Why Two-Tower fits used cars#
The used-car marketplace has a property that makes collaborative filtering and matrix factorization awkward: inventory turns over completely. A car listed today may be sold tomorrow, so there’s no long-lived catalog for a user-item interaction matrix to accumulate signal on. Every car is effectively a new item.
The Two-Tower model handles this because it embeds attributes, not item IDs. A white 2022 Hyundai Tucson with 30,000 km priced at AED 72,000 gets an embedding from its features: make, model, year, body type, color, odometer, price, and condition score. When that car sells and a similar one is listed, the new car lands in the same region of the embedding space without any interaction history.
The two towers#
graph TD
subgraph "User tower"
U1["Click history\n(make/model/price distribution)"] --> UE["User\nEmbedding"]
U2["Search query tokens"] --> UE
U3["Filter selections\n(body type, price range, year)"] --> UE
U4["Session features\n(time on site, pages viewed)"] --> UE
U5["Recency-weighted\ninteraction sequence"] --> UE
end
subgraph "Item tower"
I1["Car attributes\n(make, model, year, body type)"] --> IE["Item\nEmbedding"]
I2["Price & market position\n(vs. segment median)"] --> IE
I3["Condition & inspection\nscore"] --> IE
I4["Demand signals\n(I2V, STR, BI rate)"] --> IE
I5["Listing age &\ninventory freshness"] --> IE
end
UE --> SIM["Cosine Similarity\nin Shared Space"]
IE --> SIM
SIM --> SC["Relevance\nScore"]
class UE amber
class IE blue
class SIM greenThe user tower encodes behavioral signals into a dense vector. For users with click history (at least 3 clicks in the past 30 days), the main features are the distribution of attributes across the cars they clicked: makes, models, body types, price ranges, and age ranges, weighted by recency and depth of engagement. A gallery view plus an inspection-report view counts for more than an impression. For cold-start users, search query tokens and filter selections take the place of click history; the cold-start section covers this in detail.
The item tower encodes car attributes and demand signals. Including demand-side features such as impressions-to-views (I2V), sell-through rate (STR), and booking-initiated (BI) rate is the key design choice. These features capture “cars that similar users showed interest in” without explicit collaborative filtering. A car with a high I2V rate among users whose embeddings resemble the current user’s gets a relevance boost, even if this user has never seen it.
Serving with HNSW#
After training, every car in active inventory gets an item embedding, and those embeddings are indexed in an HNSW graph for approximate nearest neighbor search. At serving time, the system computes the user embedding and HNSW returns the top-K most similar items in under a millisecond.
HNSW suited our scale better than the alternatives (LSH, IVF, or brute force). About 10,000 items is small enough that HNSW’s memory overhead is negligible, but large enough that brute-force cosine similarity across many concurrent sessions would eat into the latency budget. HNSW gave us consistent sub-millisecond retrieval with recall above 95%.
The index is rebuilt on a batch cadence as inventory changes. Because cars are listed and sold every day, freshness matters: stale embeddings for sold cars were the most common failure mode. A car that no longer exists appearing in someone’s top 10 is worse than no personalization at all.
LightGBM re-ranker: precision ranking#
The Two-Tower model retrieves a broad candidate set optimized for relevance. The LightGBM re-ranker re-orders those candidates using features that are too complex or too dynamic for the embedding model.
Feature categories#
The re-ranker uses five categories of features:
Car attributes. Odometer reading, odometer per year, price relative to the segment median (the “proportion RFC of bought price”), and odometer deviation from the make-model median. These capture whether a specific car is a good or bad deal compared with similar inventory.
Demand signals. I2V at 15-day and 90-day windows, view-to-booking-initiated (V2BI) at both windows, 90-day STR, and BI per listing per day. These features encode the platform’s collective behavior. If many users view a car but nobody books it, something is wrong with the listing: the price, the condition, or the photos.
Market science. Active listing count for the same make, model, and year; listing count for competitors with a similar body type and price band; and price deviation from that competitive set. A car priced 10% below its competitive set is a value proposition the ranker should surface.
Car condition. Post-refurbishment inspection score and specific defect flags. Users care about condition but rarely filter for it, so the ranker learns implicitly that users who click newer, lower-mileage cars are condition-sensitive.
Supply context. Mean similar-car listing count at the body type and price level. If there’s only one SUV under AED 50,000 on the platform, it should rank higher for SUV-preferring users regardless of other signals, because scarcity matters.
The ship pipeline#
Many ML projects fail at this stage. A model that improves offline metrics but degrades the production experience is worse than no model. We shipped through a sequence of five gates, each of which had to pass before the next:
graph LR
A["Offline Eval\nNDCG@K / MAP\nmust beat baseline"] --> B["Position-Bias\nCorrected Replay\non logged traffic"]
B --> C["Shadow\nDeployment\n(no user impact)"]
C --> D["Power-Analysed\nA/B Test\n+ SRM checks"]
D --> E["100% Rollout\n+ PSI Drift\nMonitoring"]
class A,B amber
class C violet
class D blue
class E green- Offline evaluation. We measured NDCG@K and mean average precision (MAP) on a held-out test set. The model had to beat the production baseline by a meaningful margin. This gate catches models that don’t learn anything useful.
- Position-bias-corrected replay. Many teams skip this gate, and it’s the most important one. Logged clicks are biased: users click items partly because they were shown at the top. A model trained on raw clicks learns to reproduce the existing ranking rather than improve it. Position-bias correction (inverse propensity scoring, or a position feature that’s zeroed at inference) debiases the training signal. The replay then evaluates the new ranking against what users actually engaged with, correcting for the fact that they only saw the old ranking.
- Shadow deployment. The new model scored every production request alongside the existing model, but only the existing model’s scores were served. This catches infrastructure problems such as latency spikes, missing features, and null scores without affecting users.
- Power-analysed A/B test. We randomized at the user level, not the session level, so each user saw the same model throughout their journey. Daily sample ratio mismatch (SRM) checks detected randomization bugs. The test ran until it had enough statistical power for the primary metric, U2BI. We monitored secondary metrics but didn’t use them for the ship decision.
- Drift monitoring. After launch, the Population Stability Index (PSI) tracked whether production feature distributions were drifting from the training distribution. A PSI spike means the model is scoring data it wasn’t trained on, and triggers a retrain.
Cold start: the 35% problem#
On a typical day, Cars24 UAE’s users broke down as follows:
- About 24,000 unique daily users
- About 12,000 “relevant users,” with at least 10 seconds on site and at least one impression
- Of those, about 5,400 had 3 or more clicks in the past 30 days: the users the system was designed for
- About 2,500 had 1–2 clicks, enough signal to personalize partially
- About 4,080 had zero clicks, 34% of the relevant base
The cohort system couldn’t personalize zero-click users at all; they got default sorting. We designed the N=1 system for users with at least 3 clicks and then experimentally expanded it to everyone. Without click history, though, the user tower had nothing to embed.
Search and filter as proxy signals#
Intentful users, the ones who actually want to buy a car, almost always search or apply filters before they click anything. A user who types “Honda Civic 2021” and then filters to “under AED 60,000” has revealed a lot about their preferences without clicking a single car.
These signals are weaker than clicks. A search for “Honda” doesn’t mean the user will buy a Honda; they might be comparing. But they’re far stronger than nothing, and for cold-start users that’s the bar.
We added search query tokens and filter-selection impressions as user-tower features. For users with click history, these features are present but dominated by the stronger click signals. For zero-click users, they are the signal: the user embedding is built mainly from what the user searched for and filtered by.
What worked and what didn’t#
For users with 3 or more clicks (45% of the relevant base), results were strong across the board:
| Metric | Cohort personalization | N=1 personalization | Uplift |
|---|---|---|---|
| Click Recall@50 | 12% | 30% | +150% |
| BI Recall@50 | n/a | 33% | New metric |
| U2BI | Baseline | +16% | |
| Time on site | Baseline | +6% |
For users with 1–2 clicks (21% of the relevant base), results were mixed:
- Clicks per user improved by 13%, from 1.16 to 1.30.
- Time spent increased by 33%, from 28 to 33 minutes.
- The conversion signal was too sparse to measure reliably.
For zero-click users (34% of the relevant base), engagement improved but clicks didn’t:
- Time spent improved by 13% (22% for brand-new users).
- Clicks per user didn’t improve.
That last result matters. The search and filter proxy puts the right category of cars in front of zero-click users, which is why they spend longer on the site, but it doesn’t make them click. The likely explanation is that these users aren’t failing to find relevant cars. They’re browsing casually, comparing prices, or waiting for a better deal. Personalization solves a discovery problem; these users may have a decision problem that personalization can’t fix.
So the 35% cold-start problem is solved in one sense: 100% of relevant users now get personalized rankings. It’s unsolved in another: personalized rankings don’t move users who aren’t ready to engage.
Results#
After full rollout on 12 September 2024, we let the system run for several weeks until the numbers stabilized. This section explains what each metric measures and what the result means.
Click Recall@50#
Click Recall@50 measures the fraction of cars a user eventually clicked (anywhere in the catalog) that appeared in the model’s top 50 recommendations. It’s the best single measure of whether we understood what the user wanted: a high value means the model surfaced the cars the user would find interesting before they had to scroll past 50 generic listings.
The metric rose from 12% to 30% (+150%). Under the cohort system, only 12% of eventual clicks were on cars in the top 50; under N=1, 30% were. Users found what they wanted 2.5× faster.
Buyer conversion (U2BI)#
User-to-booking-initiated (U2BI) is the fraction of users who start a booking: the first hard commitment, where a user chooses a car, enters their details, and proceeds to payment.
U2BI rose 16% over the cohort baseline. Faster discovery led to faster decisions and more bookings.
View-to-buy intent (V2Bi)#
V2Bi is the fraction of cars with a detail-page view that progressed to booking initiation.
V2Bi rose 21%, driven by the LightGBM re-ranker. This metric isolates the re-ranker’s contribution: the Two-Tower model gets the right cars into the candidate set, but the re-ranker sets the order, and the order determines which cars get viewed.
An intuition for cohort versus N=1 ranking#
Imagine 6 users and 10 cars. Under cohort personalization, users A, B, and D are in cohort C1, so they all see the same ranking: cars 3, 1, 2, 5, then the rest. Under N=1, each user gets their own ranking. A sees 1, 2, 3; B sees 1, 2, 5; D sees 1, 3. The rankings aren’t wildly different, because the cohort wasn’t wrong about the broad category. The value is in the order within that category, which cohorts can’t capture.
Lessons learned and future work#
Data lag limits quality#
The system personalized users on interaction data that was two days old. User A’s recommendations on Tuesday reflected their clicks from Sunday, not Monday. In a marketplace where new inventory lands daily and intent shifts between sessions, that’s a meaningful gap.
Here’s how it fails: a user browses Nissan Sunny and Hyundai Lancerx on day T-2. On day T-1, they switch to Sportage and Creta, perhaps because their budget changed or they’re shopping for someone else. On day T0, the recommendations still push Sunny and Lancerx, and the user has to search manually for what they want now.
The fix is to reduce the data lag to same-day or half-day. The blocker was the batch architecture: GA4 events landed in Snowflake with a processing delay, and feature engineering ran on a daily cron. A streaming architecture (Kafka feeding real-time feature updates) would solve this, but it required infrastructure investment beyond the data science team’s scope.
Weight recent interactions more#
When we did have interaction data, every interaction in the 30-day window carried the same weight; a click from 28 days ago counted as much as one from yesterday. Preferences shift, and recent actions are far more predictive. Exponential time decay on interaction weights, with a half-life of about seven days, would improve embedding quality for active users.
Add quality guardrails for low-click users#
Users with exactly 3 clicks qualified for full personalization. But 3 clicks on variants of the same car, such as three white Honda Civics, may mean the user is comparing prices for one model rather than wanting a white Civic in general. Requiring clicks on at least two distinct make-model combinations would trade some coverage for signal quality. I’d expect a small drop in the number of personalizable users and a meaningful lift in accuracy for the rest.
Zero-click users need a different intervention#
As the cold-start results showed, personalization doesn’t move clicks for users with no click history. These users may need something else entirely: a better default assortment, clearer pricing, or notifications when a car matching their searches drops in price. Personalization solves “I know what I want but can’t find it.” Zero-click users may be stuck on “I’m not sure I want to buy at all.”
Move to online scoring#
The shipped system used batch-computed scores published through PubSub. The natural next step is online scoring: computing rankings at request time from the user’s in-session behavior. That captures intent shifts within a session (a user who started on sedans and moved to SUVs) and removes the data lag entirely. The Two-Tower architecture supports this cleanly: recompute the user embedding with in-session features, query HNSW, and re-rank. Infrastructure cost is the blocker, not the ML.
Conclusion#
Moving from 25 cohorts to N=1 personalization was an architecture change, not a model upgrade. The cohort system found groups of similar users and gave each group one ranking. The N=1 system learns what each user wants and ranks the whole catalog for them.
The Two-Tower model represents user preferences and car attributes in a shared embedding space where relevance is a distance. The LightGBM re-ranker handles what embeddings can’t: demand dynamics, price competitiveness, and inventory freshness, all of which change daily.
Using search and filter signals as proxy features extended coverage from 65% to 100% of users. It doesn’t make zero-click users click, but it shows them the right category of cars, which beats the default sorting they had before.
Most of the engineering effort went into the ship pipeline: offline evaluation, position-bias correction, shadow deployment, a power-analysed A/B test, and drift monitoring. A recommendation model is only as good as the infrastructure that validates it, deploys it, and catches it when it degrades. Without that pipeline, the model is a notebook; with it, it’s a production system serving more than 10 million sessions a month.
If I built this again, the one thing I’d change is data freshness. A two-day batch lag caps recommendation quality. The model, the features, and the cold-start approach can all be iterated on; the data lag is structural.
I built this system at Cars24 UAE.
Responses
Select any passage to highlight it or comment on that line. Highlights stay private, in this browser.