Data is the New Frontier

Edited for clarity. Timestamps are approximate.

1. Nick Mecklenburg00:06

   We’ve been interested in the data angle lately, so we have Brendan Foody, Mercor’s CEO, here to chat with us. Please give him a round of applause.

2. Brendan Foody00:15

   Thank you for having me.

3. Nick Mecklenburg00:18

   Why don’t we begin by talking about your origin story for Mercor?

4. Brendan Foody00:24

   I grew up in the Bay Area. I met my co-founders, Adarsh and Surya, when we were 14 in high school in San Jose. We started Mercor when we were 19, initially hiring people internationally, matching them with our friends, and automating all the processes of how we did that.

   Similar to how a human would review a résumé, conduct an interview, and decide who to hire, we automated all of those steps, hiring engineers from India and matching them with our friends. Then we dropped out of college and met OpenAI shortly thereafter, where we saw this enormous transition in the data market.

   It was moving away from the crowdsourcing paradigm that Scale and Surge pioneered: how do you get low- and medium-skilled people writing barely grammatically correct sentences for early versions of ChatGPT? It was moving toward the agentic data paradigm: how do you find the top software engineers, ML engineers, lawyers, and doctors who can work collaboratively in teams to build frontier evals and RL environments?

   We scaled up extremely quickly from there, becoming the largest data vendor to almost all of the frontier labs and the top application-layer startups, ranging from Harvey and Sierra to Cognition, Cursor, and many others. We scaled from $1 million in revenue run rate to over $2 billion in about 24 months. Now it’s dramatically higher than that, with accelerating growth over the past year.

   My top focus now, and what’s pertinent to this conversation, is how we take all this technology that we’ve built for the AI labs in San Francisco and apply it to the broader set of enterprises and startups now exploring how to post-train their own models. I believe this is a fascinating market because it’s a very small part of how tokens get consumed today, but it’s going to be the lion’s share of how tokens get consumed in five years. I’m excited to discuss it and share our insights with everyone.

5. Nick Mecklenburg02:27

   That’s great. You mentioned this previous industry-shift moment, and I feel like we’re almost at the cusp of another shift, specifically recursive self-improvement, or RSI. What does RSI mean for Mercor’s business? Let’s say you can hook up a model and its agent traces and have it improve itself.

6. Brendan Foody02:49

   I would say a few things. First, models really struggle to learn on their own in non-verifiable domains, for the same reason a student struggles to grade their own homework. If you have a math problem with a calculator as a super-clean simulation, it’s easy. But when models build a slide deck, they aren’t able to identify the mistakes in their own slide deck.

   The market for humans teaching models how to improve is bounded by the number of things a human can do in the economy that a model can’t do. Of course, we’ll automate the post-training and hill-climbing process along the way. But I think we’ll move toward a paradigm where, once you have an eval for something, RL is so effective that you can saturate that eval immediately.

   The bottleneck for automating everything in the economy is how you build evals for everything. Evals are definitionally beyond the frontier of what models are able to do.

7. Nick Mecklenburg03:50

   You’ve spoken publicly about how 80% of model differentiation comes from data. Why is that the case, and where does the other 20% come from?

8. Brendan Foody03:58

   Someone asked me this, and upon further reflection, I would more so say it’s 100%, with the caveat that there are algorithmic shifts, like reasoning models, where there are giant breakthroughs.

   But for the most part today, everyone’s using the same algorithms. So long as people are using the same algorithms, the model is just a reflection of the data that goes into it. There might be different tricks for augmenting expert data and getting the most out of the data they have, but ultimately most of those tend to come down to data curation. That tends to be where labs invest a lot of their compute: how they generate more data and ultimately differentiate their models.

9. Nick Mecklenburg04:51

   On the topic of data, South Park Commons is full of very ambitious people. I get a question along these lines almost every forum session: what do we do if we’re a founder and we just have access to data that we think is valuable?

10. Brendan Foody05:06

    It’s an interesting question. It depends a lot on what kind of data. There are so many ways you could go about it. In the extreme case, if you have an extremely valuable dataset where you could train a frontier model for a specific capability, you could build a lab, serve a model, and monetize that.

    If you just have a company dataset from a startup that shut down, you could sell that. We buy hundreds of company datasets. If you have RL tasks, we would buy those as well. It’s very contextual to the type of data you have. You could use it to figure out how to train a model for the use case you care about, or figure out a way to monetize that data. We’d love to be a buyer of that, and I know many others would as well.

11. Nick Mecklenburg05:58

    As a follow-up, how do you determine the scientific value, and perhaps the correlated monetary value, of an eval or that data?

12. Brendan Foody06:08

    There are a couple of things that matter. The first is realism: how much is it in the distribution of the economy and what the broader set of users of AI models care about?

    The second, especially if it’s an eval where you have tasks, is the pass rate or difficulty. If you have a task that every model is able to complete, it’s not valuable at all. If you have a task that no model is able to complete, it becomes dramatically more valuable.

    The final thing, and maybe the largest misconception in how people think about the value of data, ties to the RSI question around traces. Models aren’t able to learn just by reading your Slack or all the company data, for the same reason we aren’t able to learn from that stuff. If I just read a bunch of examples of slide decks, I don’t necessarily know how to build the most immaculate slide deck.

    What I need is feedback: here’s the thing I did well in producing the slide deck; here’s the thing I didn’t do well. How do I measure that in a way that lets me make more attempts, iterate, and improve? That’s why RL tasks are orders of magnitude more valuable than strictly having a bunch of data about what happened. That’s why experts play a giant role in creating these. We have about a million hours a month now of people working to build out these tasks.

13. Nick Mecklenburg07:46

    This is a great segue, because I was going to ask you about experts next. You mentioned that creating data to train AI will be the last human job. That implies there’s a shelf life for human data collection. What happens when the models are so good at the subject that there aren’t any, or enough, human experts to label it or judge it?

14. Brendan Foody08:08

    As I was getting at earlier, the market for humans improving models is bounded by the number of things a human can do that a model can’t do. I think eventually we will get superintelligence, where there’s nothing in the digital or physical world that a human can do that models can’t do.

    But along the way, that entire road is paved with translating human intelligence into the models through building the evals and training datasets that we can hill-climb. I think that’s going to take an extremely long time to do effectively.

    The reason that’s how humans will contribute to the economy is that we’ll move away from people doing repetitive, redundant work: a customer support representative responding to a hundred tickets, a lawyer repetitively redlining contracts, or consultants following their McKinsey template. They’ll move toward training an agent how to do that thing.

    That will manifest itself in a variety of forms. Sometimes it will be building an eval; sometimes it will be updating skills files. I think the entire way humans contribute to the economy is going to move toward this paradigm of training agents how to do things, because that is a structurally higher-leverage way to complete work.

15. Nick Mecklenburg09:33

    What about math? There are exceptions in math with clean verifiers.

16. Brendan Foody09:38

    Yeah.

17. Nick Mecklenburg09:40

    Would you say human experts are losing utility in a place like math?

18. Brendan Foody09:46

    I would say that. A really good heuristic is how we learn about things. If we want to get better at writing really efficient code, we can look at how efficient that code was after compiling it. We have a clean verification loop where we don’t need to get feedback from others.

    There are problems like math, code efficiency, and even post-training, where you can look at the score on the eval. It feels like large parts of those problems aren’t going to require a lot of human expertise because there’s a well-defined environment.

    But if I think about the vast majority of things in the economy, like having a conversation with a candidate in a way that gets them excited, or having a sales conversation, there are so many things that are intrinsically human. Even so many legal questions, or coming up with a business strategy: there are ten ways you could come up with a world-class business strategy for a given topic and a hundred ways you could make a mistake. That’s the opposite of a clean verification loop.

    It would be really hard for someone to learn if you just said, “Go implement all those business strategies over the next ten years and figure it out.” There needs to be intermittent reward: mentors who give us feedback about what’s going well and what’s not. If I reflect on how we learn, the vast majority of it is feedback from others. But there is some element where we have super-clean feedback loops, like a calculator.

19. Nick Mecklenburg11:34

    Has Mercor been shifting its focus on evals toward some of these regimes that are less verifiable or less clean?

20. Brendan Foody11:42

    From the beginning, that’s always been our focus, because the market in any domain is bounded by what a human can do that a model can’t do. We started out hiring IMO medalists in the early days of reasoning models. Of course, we don’t really need the IMO medalists anymore. But at the same time, we were hiring a lot of consultants and bankers, and we still really need those people.

    So long as there is anything one of those experts can do that the models can’t do in the digital or physical world, we need to teach the models how to do that. Some of it might feel intrinsically human, like having a really compelling conversation with someone or expressing empathy.

21. Nick Mecklenburg12:29

    This is another great segue, because I wanted to talk to you about APEX, where you’re starting to do some of this work. A year ago on APEX, the best model was scoring in the low single digits. Where is the frontier now, and what’s left in the gap between that number and a professional?

22. Brendan Foody12:44

    We built this benchmark called APEX-Agents, where we hired tons of experts across the top professional domains that we see in finance, law, and consulting. They created data rooms that mirror real data rooms from jobs they were completing in the economy, rendered those into mocks of Google Workspace, and created prompts and rubrics to verify how well models can complete responses.

    The fascinating thing is that o1, which I guess is about 18 months old at this point, scored less than 1% on APEX-Agents. When we launched APEX-Agents in February of this year, frontier models were at about 23%. Now they’re at 65%.

    It’s been an insane demonstration that, once you can build an eval for something, the labs can hill-climb it ridiculously quickly, even for diverse evals across a broad problem space. Naturally, the next evolutions are: what things in the economy is that not capturing? How do we increase the realism, and how do we measure what comes next?

23. Nick Mecklenburg13:57

    You mentioned APEX-Agents, but you have more, right? You have APEX-Accounting.

24. Brendan Foody14:04

    Yeah. We built that with Ramp.

25. Nick Mecklenburg14:05

    Exactly. And you have another partnership with Harvey on another eval.

26. Brendan Foody14:08

    We built Legal Agent Bench with Harvey. We built tau-bench with Sierra, and APEX-SWE with Cognition as well.

27. Nick Mecklenburg14:17

    How does that come together? What do those companies give you that you can’t just do in-house?

28. Brendan Foody14:23

    They understand the realistic and economically valuable usage patterns in their user base. Their motivation to contribute is that an application-layer company’s reason for existence, or its defensibility against frontier models, is framed by how much it can commoditize the frontier models.

    If Claude were way better than any other model at coding, to an extreme extent, there’d be no market for an application-layer company competing against Claude Code. If they put a public eval out in the world, all the Chinese labs will distill against it, and every other lab will hill-climb it. There starts to be less leverage from just the frontier model, so they’re able to improve the entire ecosystem in the distribution they care about, which is incredibly important for their businesses.

    I think we’ll see this trend increase dramatically across every startup and enterprise: those companies wanting to build public evals. You can build a public eval for a million dollars, and the labs will spend hundreds of millions climbing it. It’s a very high-leverage way to improve model capabilities in the distribution you care about.

29. Nick Mecklenburg15:51

    Could you argue that a company like that could use its unique insight and edge to hill-climb a model and post-train internally?

30. Brendan Foody16:00

    In some cases, that makes sense. Each of them also has other evals that are private, and a set of evals that are public. It’s very contextual to where you want to build competitive advantages in your business versus the areas you want to commoditize. That’s an interesting strategic question every app-layer company is working through, and sometimes post-training, as you’re getting at.

31. Nick Mecklenburg16:31

    Let’s talk a little more about training. What happens to Mercor tomorrow if a super sample-efficient learning or modeling technique is released and takes the world by storm?

32. Brendan Foody16:40

    The thing this would most likely look like is having much better processes for creating synthetic training data to hill-climb an eval. Then I think all data production will converge on building evals. Even if you can automate the entire training process, so that once you have an eval you can saturate it overnight, you still need some sense of what good looks like. What’s beyond the frontier, and how do you measure that?

    One thing a lot of people don’t realize about our business is that the majority of what we do is actually building evaluations, not training data. I think that will increasingly be the trend as post-training gets automated throughout every domain in the economy.

33. Nick Mecklenburg17:33

    Is there a rough percentage you have there, like 80% of the work is eval creation?

34. Brendan Foody17:39

    There’s always a blurry line because it’s the same task shape, so it’s not always perfectly clear. But I would say a slight majority is eval. Often it’s eval for that use case of creating synthetic variants and then having super sample-efficient hill-climbing at those evals.

35. Nick Mecklenburg17:57

    We were talking about this earlier, but I want to ask about another tweet of yours. You predicted that a frontier lab will launch or relaunch a fine-tuning product in the next three months to compete with the RL-as-a-service companies. As I observe it, the trend seems to be going in the opposite direction. OpenAI had a fine-tuning product, but then they deprecated it. What do you think?

36. Brendan Foody18:20

    It’s certainly right that they had a fine-tuning product and deprecated it, where SFT didn’t really work and RFT didn’t really work. They didn’t get adoption. People who tried to adopt it couldn’t use it because the infrastructure didn’t work well, and the labs didn’t invest in fixing it because the models were improving too fast for there to be market demand.

    The reason I think the labs will start having more post-training products is that, for a certain set of workflows in the economy, we’re reaching a sufficiency threshold where you don’t need the ultra-frontier model for every use case. It’s structurally more efficient to have a smaller model customized using a distilled version of their large frontier model for a specific workflow.

    There’s starting to be this startup ecosystem of companies doing post-training, RL as a service, and all the products around that. Maybe three months is a little bit of an exaggeration. But if this continues to become a very large market over the next three months, in the way things look to be trending, it’s inevitable that the labs are best positioned to distill and train smaller models, especially where their customers have hit the sufficiency threshold.

37. Nick Mecklenburg19:50

    Is that not shooting themselves in the foot a bit?

38. Brendan Foody19:54

    Not necessarily. In some ways, it drives down their total revenue because they might have a smaller model or charge less per token. But that’s better than having a competitor with a comparative advantage on cost, where they lose that customer relationship and use case altogether.

39. Nick Mecklenburg20:47

    On data, how do you differentiate between data that’s suitable for post-training and data that isn’t?

40. Brendan Foody20:57

    The most important thing, outside of realism and difficulty, is the quality of the verifiers. Take the problem of building a great slide deck. It’s incredibly challenging to build a rubric because there are a dozen ways to create a great slide deck and a hundred ways to make a mistake. The best rubric needs to capture the full potential of the problem space: all the paths you could go down and all the mistakes you could make.

    It needs to avoid false positives and false negatives when you roll out hundreds of trajectories during training and score all of them. The most valuable data is these RL tasks where you can build spectacular verifiers. A lot of our technology goes into doing that as efficiently as possible.

    If we subscribe to the idea that humans will contribute to the economy by training agents rather than doing repetitive work, how do we optimize the ratio of converting an hour of human time into as much model improvement as possible across the entire distribution of the economy?

41. Nick Mecklenburg22:09

    What’s hard about making a verifier?

42. Brendan Foody22:14

    It’s all the false positives and false negatives when the problem space is so large. It’s easy in math, where you just check the right answer, but most domains aren’t like that. Even in code, it’s somewhat easy with unit tests, yet we still see a lot of reward hacking where a model deletes the entire codebase, reproduces it, and passes all the unit tests, or proliferates tons of comments.

    In other domains, where you don’t have unit tests or quantitative answers, it’s unbelievably challenging to build verifiers that avoid false positives and false negatives.

43. Nick Mecklenburg22:54

    What happens when you have user signals? Take your PowerPoint example. I’m a company like Gamma generating PowerPoint. Maybe the user accepts these ones and doesn’t accept those ones. Can that be the training or evaluation signal?

44. Brendan Foody23:07

    It’s a helpful flag for which prompts you should create rubrics over. But it’s not a helpful training signal because it doesn’t answer what all the mistakes were and what all the important criteria are. When you roll out a hundred trajectories from the model you’re training, you need a score on all of those trajectories that corresponds to how good or bad each one is.

    You can’t derive the full problem space of all the good and bad in that slide deck from a simple thumbs-up or thumbs-down. But you can understand the failure patterns, where a model makes mistakes, based on the thumbs-down, so you can target rubric or task production at the highest-leverage places.

45. Nick Mecklenburg23:57

    Do you think you could automate looking through the errors and finding the commonalities and weaknesses?

46. Brendan Foody24:04

    You can automate it in domains with very clean environments, like math and code efficiency. But for how aesthetic a slide deck is, I don’t think you can, because it’s definitionally beyond the frontier of model capabilities. How would the model reliably measure something it doesn’t know how to do?

    Many parts of this are about understanding the human condition: what slide decks do humans find aesthetic? Even if you have superintelligence, there’s this problem of building a movie humans love to watch, a slide deck humans love to read, or having a conversation humans find persuasive. That’s often some of the hardest stuff to build verifiers for.

47. Nick Mecklenburg24:55

    Given it’s so complex, how do you build verifiers for that kind of thing? You’ve talked about why it’s hard, but Mercor is taking it on and selling it, right?

48. Brendan Foody25:03

    Exactly. You need to roll out a ton of trajectories, score all of them, and validate how they’re scored to look for false positives and false negatives. You do that through some combination of humans and agents. That’s generally what things have trended toward.

    Oftentimes you don’t even catch everything. Say you roll out a dozen trajectories while building a great verifier. There might be reward hacking that only appears when you start rolling out hundreds of trajectories. Reward hacking is almost certainly the most challenging problem in RL training.

49. Nick Mecklenburg25:49

    How often do you run into that?

50. Brendan Foody25:51

    All the time. We’re producing hundreds of thousands of tasks a month across those millions of hours. The largest thing we think about is the noise in those verifiers and how we structure a really good task shape and eval signal to hill-climb.

51. Nick Mecklenburg26:16

    We have an ongoing debate in the forum: do RL environments have a shelf life? You mentioned that creating data is the last human job, so you probably have some opinions on this. What are your thoughts?

52. Brendan Foody26:29

    I know I’m being a little repetitive, but it depends on the specific market and how many things a human can do that a model can’t do. If you build RL environments in coding, and in 12 months we get to a point where there are very few things a human software engineer can do that a model can’t, it doesn’t feel like there’s a super-large market for continuing to build more environments there. At the very least, that market will be brought in-house by the labs, because they generally want to do purely synthetic things in-house.

    But for other things that are taste-based, or the extreme example of everything in the physical world, like models running scientific experiments and exciting problems in robotics, it feels like it’ll be a really long time before models are better at absolutely everything humans can do, with no task where humans have a comparative advantage.

    I think it’s the last human job, but on a many-decade time horizon. I’m sort of “AGI long, ASI short.” Maybe we’ll automate 50% of tasks in the economy in the next three to five years, but there’s a long tail. A lot of things are intrinsically human in how the economy is wired, and the physical world is going to take another few decades.

53. Nick Mecklenburg28:07

    Tell me more about how Mercor thinks about the physical world.

54. Brendan Foody28:11

    We’re very excited about it. I think it will be the next frontier of data collection, because ultimately we need to measure the frontier of what models succeed at and fail at in the physical world, not just in the low-skilled domains people talk about, but also in high-skilled domains like electricians, mechanics, and scientists.

    We’re very excited about how we can use our products and capabilities to improve models there.

55. Nick Mecklenburg28:52

    What’s unique about preparing data or evals for being an electrician?

56. Brendan Foody28:59

    I’m not an expert in that one.

57. Nick Mecklenburg29:03

    Pick a physical-world task, let’s say.

58. Brendan Foody29:05

    I’ll give you one we’ve been thinking about a lot: wet labs. Part of the reason our customers use us is that I don’t really know how to run wet-lab experiments, and neither do the AI researchers.

    We need people with expertise in chemistry, biology, or whatever is necessary to run those experiments. We need the equipment relevant to that experimentation, and we need to make sure those people are available at the right times. It’s a really hard logistical problem to coordinate this.

    I’ve talked with some of the leading companies automating things in the physical world, and many of them have raised nine or ten figures and plan to spend more than half of that on data, because it’s the primary bottleneck to improving model capabilities.

59. Nick Mecklenburg30:12

    What about iteration speed? You can run and test code pretty quickly, but you can’t patch a circuit as quickly.

60. Brendan Foody30:18

    That’s certainly the case. You can’t do RL with it easily, where you automate the process of attempts and grading. That makes it a lot harder, and it’s structural to the physical world.

    The other structural thing is that the failure tolerance is often a lot lower. With coding, you can build something, and if it doesn’t work, toss it out and redo it, or have an A/B test for how well your experiment works. Driverless cars took so long, with so many brilliant people focused on them, because a 99.99% reliability threshold wasn’t enough.

    If you think about electricians, mechanics, or even doing dishes, you can’t mess up 40% of the time, because you might break all the dishes or mess up the kitchen. The reliability threshold needed for the physical world is one reason I suspect it will take decades before we have superintelligence in the physical world.

61. Nick Mecklenburg31:26

    Would you say synthetic data in environments is much more important in this kind of regime?

62. Brendan Foody31:34

    Not exactly. Simulations are what I would say is really important. We’ve done a lot of this. I’m not sure if you’re familiar with a benchmark called SciCode, but it’s the top science benchmark, built by Minyang Tian, who was a physics PhD at UIUC. He went to his friends who are PhDs and got all their hardest science problems.

    We hired Minyang and built out future versions of SciCode. One of the premises is building simulations that can predict how well a wet-lab experiment is going to run, so we can use that for RL. If we can simulate it in code, even as a rough approximation, that makes it a tractable problem for labs to hill-climb. There’s lots of cool stuff you can do in how the digital and physical worlds blend together.

63. Nick Mecklenburg32:26

    There are a bunch of people at SPC working on this kind of thing, the sim-to-real gap and so on. How much of a problem is that for you as you approach these problems?

64. Brendan Foody32:38

    It’s super critical. That’s why, when you asked about monetizing people’s data, the first thing I said was realism. The data only matters insofar as it’s predictive of a certain percentage of the economy and the value people would place in it.

    The sim-to-real gap is critical, and a lot of what we do is figure out contrarian ideas for gaps people might be missing. One fascinating example: I often ask people what percentage of tasks in their job require interacting with other people. For most jobs, it’s actually over 50%. Yet what’s the representation of that in benchmarks? It’s 0%. There’s some multi-turn with tau-bench, but not interacting with different co-workers.

    There are all sorts of creative things we research to measure the sim-to-real gap across every dimension in the physical and digital worlds, so we can bridge it.

65. Nick Mecklenburg33:44

    On synthetic data, frontier labs are investing significantly in synthetic pipelines and keeping them in-house. Do you think there’s a world where much of the data ends up being synthetic?

66. Brendan Foody33:58

    I think a lot of training data will be synthetic. I think eval data definitely needs to be human. Our thesis has always been that labs will bring synthetic, or almost entirely synthetic, pipelines in-house, because that’s how they build a competitive advantage. If 100% of the model is a manifestation of the data, they need alpha in how they curate their data pipelines.

    But what none of the labs want to do in-house, and some tried and failed, is build a giant talent network of five million people, where we’re paying out $6 million a day and have done seven million interviews to mobilize people across every category in the economy. Humans will play a giant role in evals, and labs will bring their synthetic pipelines in-house. That’s how the world will trend.

67. Nick Mecklenburg34:53

    Some companies are betting that foundation models contain all the preferences and tastes of being human, like Simile, for example. Do you think this goes against the narrative here?

68. Brendan Foody35:11

    I think they contain part of it. I love Simile. I invested early on, and I think Joon and Percy are awesome. There are certainly capabilities embedded in pre-training that could be valuable, especially for their use case of market discovery for consumer products.

    But it feels like there’s always going to be this structural problem: a model can’t reliably understand where it made a mistake in the slide deck. Those evals are created by humans because they’re beyond the frontier.

69. Nick Mecklenburg35:51

    The frontier is a moving target. How do you ensure data you collect today isn’t obsolete tomorrow? Even with NeurIPS or other conferences, people take a slice across a set of models, but by the time the conference is live, it might be obsolete. Do you deal with this?

70. Brendan Foody36:09

    We do, a lot. I would say two things. One is that if you make the problems really hard, they’ll have a longer shelf life. If you create an RL task where the pass rate is 60%, you might launch it and, in two months or less, the models are at 80% or 90%. That didn’t have much shelf life.

    But if you find really hard problems, things models are structurally terrible at, with under 10% or under 25% pass rates at launch, they might have a longer shelf life.

    The second is that frontier data is definitionally bespoke. We’re always pushing the frontier, and that frontier is definitionally something new. We think a lot about how to compound the engine for producing frontier data. We know we’ll always need a larger talent network, infrastructure for spinning up tooling templates and agent quality control on datasets, and a great research culture to put out new evals.

    We can compound all of those things to have a spectacular engine for producing new frontier data, even if it’s constantly evolving.

71. Nick Mecklenburg37:40

    You’re building this engine, you have the data and the network of experts. Would Mercor ever move into the modeling space directly? You said frontier labs tried to set up talent networks and failed, but you have that. Can’t you just build a model?

72. Brendan Foody37:58

    We do a lot of post-training. As you know, we hired Edward Hu, the first author on LoRA. He invented the most common fine-tuning technique, was a core contributor on o1 at OpenAI, and now leads our post-training team, where we do an enormous amount of custom post-training.

    We partner closely with all the AI startups and enterprises that post-train models. But we want to be tactful about who we choose to compete against and who we don’t, long term.

73. Nick Mecklenburg38:39

    You have Edward Hu, and you’re looking at post-training. What does that mean for the roadmap?

74. Brendan Foody38:43

    For us to know that a dataset is useful, ultimately we need to be able to train on it. Post-training is a giant muscle we’re going to continue investing in. We’re also the largest customer of all the RL-as-a-service companies. We were the first customer of most of these.

    There will be cases where customers want us to do the end post-training of the model for them, and we have powerful capabilities to do that. We also have this marketplace of service vendors, so we can outsource the workflows. We will post-train models for customers in some form or another, but ultimately it’s viewed through the lens of how we best allow them to use the data and competitive advantage of all the IP we build with them.

75. Nick Mecklenburg39:38

    Has your mindset on that shifted in the last year or two? You recently hired Edward Hu. What happened in the pre-Edward days?

76. Brendan Foody39:45

    It’s always been very hard to hire top researchers; that hasn’t changed. The company now has a lot more capital to invest in frontier post-training efforts, with billions in cash and hundreds of millions in monthly revenue. That’s one thing that’s changed.

    The other is that custom post-training is now starting to emerge as a market in a way it wasn’t 12 months ago, because of the sufficiency threshold we talked about, the scale of inference spend, and open-source models approaching frontier models. Those create good conditions for a market where it feels like every startup and enterprise needs to post-train models to stay relevant over the next five years.

77. Nick Mecklenburg40:47

    Do you have to de-risk pre-training data in any way, or would you say you have more of a focus on post-training these days?

78. Brendan Foody40:54

    Much more on post-training. We do a little bit on pre-training, and there’s always this blend of mid-training as well. But the vast majority of the focus is on data for RL.

79. Nick Mecklenburg41:06

    Has that changed over time?

80. Brendan Foody41:09

    Not much, no. Maybe it’s expanded away from purely data for RL over time. We’ve done an enormous amount of company acquisition, where we have exclusive licenses to company datasets from former Fortune 500 companies that are brand names, and have acquired hundreds of company datasets.

    We’re not the most vocal about our work in that market, but it can be used for pre-training, mid-training, and post-training.

81. Nick Mecklenburg41:39

    Do you think the story changes for the physical world? We aren’t necessarily, at least to my knowledge, in a place where everybody’s trying to post-train for robotics or the physical world.

82. Brendan Foody41:49

    I’m less of an expert on robotics. I really love this lecture Jim Fan gave about the parallel between the development of post-training for LLMs and robotics. That’s always resonated with me.

    We had all the behavior-cloning data initially collected in the LLM world, and SFT and RLHF. There’s a lot of that happening now with egocentric data in robotics. Then we got to the RL paradigm: doing a bunch of rollouts and scoring them. That’s what he calls the endgame of what we’ll get to in robotics as well.

83. Nick Mecklenburg42:35

    What about world models?

84. Brendan Foody42:38

    We do a lot of work on world models because the digital and physical worlds blend together so much. A lot of the datasets we build for evals also tend to be applicable to world models, so we work with many of those companies too.

85. Nick Mecklenburg43:05

    That’s all my prepared questions, but there’s one more: are you hiring?

86. Brendan Foody43:08

    We’re always hiring, and incredibly quickly. It’s ironic because two years ago everyone was saying, “We’re not going to need anyone else in the economy.” But as we make our team more productive, we’re hiring more than ever.

    Ranging from our AI research team working with Edward, Minyang, and many other exceptional people, to our engineering organization, to operators who help manage these human data projects, we’d love to work with anyone.

87. Nick Mecklenburg43:43

    Why should people work for Mercor?

88. Brendan Foody43:45

    We’re at a moment where I believe the most important problem is what role humans play in the economy over the next decade, when that’s more uncertain than ever. I believe humans will continue to play a really important role, and Mercor is the most important and impactful company in defining that over the next decade.

    Part of that manifests itself in how we help build all the frontier models that billions of people use. Part of it manifests itself in how we’re the fastest-growing creator of jobs in history.

89. Nick Mecklenburg44:23

    I think we’re at time. I wanted to thank everybody for coming, and one more round of applause for Brendan.

90. Brendan Foody44:29

    Thank you for having me.