Scaling Scam-Call Data Without Scaling Phone Lines

Scam AI, a cybersecurity platform, employed honeypot AI agents¹ to keep ten thousand scammers engaged in long running phone conversations. This achieved two goals: (1) collect data on scammers’ conversational tactics to learn how scammers work and fight back (2) waste scammers’ time and resources so they don’t target your relatives.

With a surge in social engineering tactics, AI fraud and deepfakes, the elderly in our society are at the mercy of scammers. Older Americans lose up to $81.5 billion a year to financial fraud². The FBI logged $7.75 billion in losses among over-60s in 2025 alone, up 59% in a single year³.

Whitney AI built synthetic data generation pipelines to accelerate Scam AI’s defensive anti-fraud research. We trained a model to simulate scam call transcripts that are near-indistinguishable from real scammers. This generation approach allows the creation of benchmarks and datasets for Scam AI to train detectors.

Fraudulent call detection has a data problem⁴, since almost no one has access to real call data. Scam and spam calls are rarely recorded because true scam victims don’t generally record their conversations. To train a model to recognize a social-engineering script as it unfolds, you need a reliable dataset with long-running calls, grounded in reality.

Scam AI’s approach of running honeypot AI agent lines is not scalable. It is infrastructurally complex and costly to maintain voice AI agents across thousands of calls. Social engineering tactics change weekly, and a dataset like that can grow old quickly. Therefore, we need synthetic data grounded in a model that sounds like a scammer.

A model writing convincing scam dialogue, however, invites an obvious question about dual use and who benefits. Scammers are not script-constrained - they already have working scripts, refined against real targets. The fraud-detection side, on the other hand, lacks volume and coverage of data: labeled examples of tactics as they evolve, enough in quantity to train a detector. As a safeguard against misuse, the ground-truth for this dataset, weights and code will not be released publicly.

Here is one turn from a real scam call:

Neither the AI scammer nor the real caller turn is implausible on their own. The problem is that B doesn't sit in the distribution real scammers actually speak from, and that error compounds: the longer the transcript, the further a one-shot generation drifts from ground truth, leading to severe compounding error.

Over half of the conversations end with the caller punting or the line simply dropping (Figure 1). Around 5% end in a completed scam. These are subtle distributional behaviors. You can’t prompt your way into a target distribution - even frontier models like Gemini fall short (Figure 2) and fail to write convincing enough synthetic transcripts. So we post-train a generator to create such transcripts.

The setup is as follows:

We aim to post-train a model that fools the discriminator, while ensuring its outputs match the distribution of real scam calls on properties such as turn length, information elicitation, and speaking style.

Here is the main metric we track:

We also track various distributional metrics:

For all of the above distributional metrics, we aim for them to be as close to the ground truth rates.

First, we benchmark OSS models + one frontier model against three frontier discriminators. We see that Gemini-3.1-Pro, our frontier benchmark, performs the best at a 28.8% deception rate. We choose Inkling-Small, a MoE model, to post-train since it is open source and at the Pareto frontier of cost / quality with a 15.5% deception rate and 12B active parameters.

We run the following post-training recipe:

• Supervised Fine-tuning (SFT)⁵: First, we run SFT on our training dataset to fit the target distribution directly
• Direct Preference Optimization (DPO)⁶: We follow up SFT with DPO to close the residual distributional gaps SFT left, using ground-truth preference pairs
• On-Policy Self Distillation (OPSD)⁷: We follow up with OPSD to further improve the model.

Post-training lifted Inkling-Small's mean deception rate from 15.5% to 41.3% - 25.8 points, or roughly three-quarters of the distance to the 50% ceiling, where a judge is guessing between real and synthetic turns.

Because we trained against a gpt-5.6-luna judge, the obvious worry is that we fit that judge's idiosyncrasies rather than the real distribution. We therefore evaluated against two judges never used during development, gpt-5.6-terra and Sonnet-5. The gains replicate on both, and they are in fact larger on the held-out judges.

We start with the observation that the model needs to “sound like” our scammers – syntactically, lexically, and stylistically. We leverage SFT to do this, fitting the model to our empirical distribution of how scam callers actually talk on the recorded line. SFT yields the largest portion of our gains by removing most, but not all, of the easy stylistic tells that our discriminators rely on.

The usual critique of SFT is that it fits the training distribution tightly at the expense of out-of-distribution task generalization (i.e., catastrophic forgetting). Here, that trade is fine since the model’s responsibility is narrowly scoped to our synthetic data task of interest.

SFT is also known to be weak against distributional shift, which requires nuance across two dimensions:

On tactics turnover, the solution is simple: continual learning. Refresh the ground-truth periodically as new categories show up and retrain to embed them into the model. On cascading errors, a common mitigation is applying a round of on-policy RL or self-distillation to facilitate off-distribution recovery: the student learns on the states that it creates itself.

SFT did not remove all stylistic quirks; we noticed discrepancies in distributional metrics such as self-repair and disfluency:

This inspires us to apply a round of DPO first to close the gap. For e.g. disfluency measures the occurrence of filler words in a conversational turn. SFT is at 48.3% while the real transcripts are at 43.9%. DPO moves us closer to the target.

The last stage is OPSD, where the model teaches itself with privileged information. The teacher sees an abstracted specification of the real caller's turn – its shape, but not its words, meant to improve generalization for new pretexts and unseen turn positions. Concretely, a turn that reads “Hello” reaches the teacher as “greeting, short, opens its own agenda”. The student only sees the transcript so far and learns to infer the shape rather than be told.

OPSD trades marginal accuracy on self-repair, disfluency and turn length for gains on unterminated turns and information elicitation, while lifting mean deception rate from 40.4% to 41.3%.

Scam AI has access to a reproducible synthetic data generation pipeline with a post-trained model at its core. Our model outperforms the frontier baseline by 12.5 percentage points on fooling frontier discriminator LLMs and generating real-sounding transcripts.

At Whitney AI, our mission is to empower enterprises to own their intelligence, build a compounding advantage and make post-training accessible. We share our learnings from our post-training experiments and case studies to further democratize this knowledge in the hope that it helps the next researcher who is tackling a similar problem.