Determinism Is King: A Game Engine Built for Robot Training Data | Devrim Yasar, Lucky Robots
Devrim Yasar is the founder and CEO of Lucky Robots, which built a game engine from scratch for one job: generating training data for robot AI. You describe a robot, a task and a room, the engine builds the scene, and it runs the episode on real physics while recording every joint and every camera frame, so a policy can fail a million times in simulation before it touches a real robot. Devrim is a repeat founder: he built Koding.com to over a million developers and sold Superpeer to Skillshare. His CTO and co-founder is Yan Chernikov, known to a million YouTube subscribers as The Cherno, whose Hazel engine is the foundation Lucky Engine is built on. Disclosure: Optim VC is an investor in Lucky Robots.
Summary
The name comes from the Boston Dynamics and Unitree videos where engineers kick the robots with sticks. Devrim's robots would be lucky, because in simulation nobody needs to kick them. It also helped that he was eating sushi at Lucky Robot, his favorite restaurant in Austin, when the idea came together.
The argument that runs through the conversation is that robot training data is a spreadsheet that cannot drift: a camera frame and six joint positions per row, millions of rows per task, and if the states and the pictures slip out of sync on the four millionth row, the policy is bad and nobody knows why. Teleoperation gets that right about ninety percent of the time. Game engines, built to drop frames so the gameplay continues, get it wrong by design. Lucky Engine is a game engine rebuilt so that it never drops a frame, records at up to 10,000 Hz, and runs MuJoCo as a first-class citizen. "Determinism is king," Devrim told me, and he applies the test to everyone: world models have never produced a deterministic dataset, teleop cannot rerun the episode where the robot spilled the coffee, and Isaac Sim cost him three months. "If Isaac Sim is BlackBerry, I think we built the iPhone."
A $65,000 robot and five toys
Devrim did not set out to build a simulation company. His last company was being acquired, which meant lawyers every day and no new features, his son was two, and the house was chaos. He wanted a Roomba with an arm that would pick up the Legos. He hired a roboticist from a respectable robotics company, who told him to buy a $50,000 Franka Panda, because that was reliable, and a $15,000 base to put it on. Then sprinkle some toys on the living-room floor, photograph them, upload the photos to Roboflow, and train a policy. "Who trains an AI model with five toys?" he said. "Am I spending sixty thousand dollars to have a clumsy robot that only works in my living room?"
At university in the Netherlands he had built architectural visualizations in game engines, the kind where you take the elevator up a skyscraper that does not exist yet and swap out the kitchen. The connection was immediate. To generalize, a robot needs thousands of living rooms and millions of toys, lights that change, a building you can set on fire if that is what you are training for. "We haven't built the Matrix for robots." And if nobody built it, he said, it was not going to get done.
A spreadsheet that cannot drift
Open a robot dataset, in the LeRobot format Hugging Face standardized, and you find a spreadsheet. Each row is a camera frame and the position of every actuator at that instant: the robot looks down, there are pliers on the table, the joints turn a little, and a little more, across six degrees of freedom. A task needs 10,000 to a million of those episodes. The picture and the joint states on the fifth frame have to agree exactly. "If it drifts, then your AI policy is just going to be bad."
Teleop data, recorded from a person in a VR headset, is above ninety percent correct. Devrim's target for simulation is a hundred, at a thousand or ten thousand hertz, for millions of hours. The comparison he reached for was a car that reads 68 miles an hour when you are doing 68.3: if you need the decimal and the instrument cannot give it to you, you are shooting blanks, and when the policy fails you go back to teleop.
Why Unreal could not do it
Lucky Robots spent its first seven or eight months in Unreal. A game engine is built so that when your CPU spikes in a multiplayer match, the frames you missed are dropped and the game continues from where you are. That is the right call for a player and the wrong one for a dataset. Unreal hands you a tick function, one sixtieth of a second, and expects you to render to a monitor, not to write terabytes of lidar, camera and joint data to disk. Its physics teleports an object into your hand because nobody playing a game cares about contact. Making MuJoCo a first-class citizen inside 3.2 gigabytes of Unreal source, and making the engine wait while the data was saved, did not work. Every one-line change meant a fifteen-minute compile.
The Cherno
Devrim found Yan Chernikov the way most C++ programmers do: on YouTube, around episode 54 of a hundred-episode series. When he told his AI researchers, their reaction was "The Cherno? We all learned C++ from him." One of Yan's videos argued that everybody should give small game engines a chance, and Devrim, who had just concluded that Unreal would never give him five terabytes, emailed him: would you want to build a game engine for robotics? No answer, so he wrote to Jim Waite, Yan's main contributor, and eventually Yan picked up the phone.
Devrim, who had spent the Superpeer years with creators and influencers, expected a $20,000-an-hour consulting pitch. Instead Yan flew to a company offsite in Austin, never mentioned money, and in a couple of days put MuJoCo inside Hazel, the engine he had been building for seven or eight years, and moved the first pieces of Lucky Robots into it. Eight months of Unreal work showed up as a prototype during the offsite. Yan had been looking for something other than games and did not want to go back to Electronic Arts. He started as CTO of simulation and is now a co-founder. The 3.2 gigabytes of Unreal became about 20 megabytes of Hazel, 240 with packages, and a click to see what you just made. "I thought I knew software engineering," Devrim said, "and then I met Yan."
First-class physics
Game physics is accurate to your eyes. It is an optical illusion, Devrim said: a slammed door that does not slam, a bird that flaps. Starting from scratch let Lucky Robots declare MuJoCo the first-class physics engine, with friction values and contact points correct to the millimeter, and keep Jolt, the game physics engine Hazel already had, as a second-class citizen for the things that only need to look right. The engine draws a diameter around the robot and adds only what the robot can reach to the MuJoCo scene, so no CPU or GPU is spent simulating the bird.
Around that sit a Vulkan renderer, C# scripts inside the engine, and gRPC and Python outside. Export the dataset, train an ACT policy, and the ONNX file comes back in over gRPC to drive the robot in simulation, so you can test what you trained. C# scripts the robot from the inside without a teleoperator, and a policy someone else trained on Hugging Face can be dropped onto the coffee machine. Load eight policies, from T-shirt folding to the fridge, write a script to orchestrate them, and "out comes your robot that does everything in your house." One customer's reaction: "I am so glad software engineering has come to our field."
BlackBerry and the iPhone
Isaac Sim, Devrim said, wasted three months of his life. It was built in 2016 and is being bent toward the transformer era, the documentation was out of date, and he had to buy a 4090 to render a couple of frames. He went to GTC and asked Nvidia's developers, who he called amazing people, to update the docs. He does not fear a re-architecture. What would worry him is five excellent people in a garage doing what Lucky Robots is doing. "If Isaac Sim is BlackBerry, I think we built the iPhone."
Determinism is king
I gave him the three ways to get robot data today: teleop fleets, world models that generate video, and physics simulation. He is not a believer in world models for this job. Five million episodes on one fridge, randomized across five hundred fridges, is enough to generalize, and he has never seen a world model produce a deterministic dataset. Even if one did, it would compute every frame, where a game engine produces five million episodes for almost free in a week.
Teleop is not apples to apples, he said. If the bench behind him was recorded by teleoperators, making it five centimeters wider or a different color means calling the people back. When a policy pours the coffee on you, you cannot revert to that frame and fine-tune on it, because the episode never existed as a thing that can be rerun. That is why there are many controlled demos and few robots serving coffee in diners. And when you call the teleop vendor back, they are fully booked for six months. "We need to build the Matrix to really fix the problem." People who say a model like Astra will close the cross-embodiment gap may be right one day, he said, but a game engine does it today.
Who buys, and why they don't build
A frontier lab told Lucky Robots in a meeting that this was exactly what it had wanted to build in-house. Devrim's answer is that no architectural firm ever wanted to build Autodesk, and no model company wants to own its teleop vendor, which he called the most boring kind of sweatshop: ten thousand people, hard to manage, able to send twenty hours of junk where nothing was placed in the right spot. I told him about the policy, trained on data bought from India, that paused mid-task to lift its arm toward a mouth it did not have, because the operators smoked. In a game engine, he said, the labels come for free. The engine knows every object, every contact point and the color temperature of the light, so it can deliver five billion episodes and guarantee which five percent failed.
Today the customers are mostly the labs building robot foundation models, because they have the researchers to use the engine, plus one company with no AI staff that just wants a robot to do something. Lucky Robots wants to stay on the data layer and leave the oil-rig deployments to someone else.
$800 an hour
Teleop data gets quoted at $20 to $50 an hour and up to $200, after a sales process Devrim compared to buying servers from Dell before AWS: three months of questions about the kitchen, the countertop and the robot. One large humanoid company told him it only uses in-house operators, on $500,000 robots, at $200,000 a year, which works out to about $800 an hour, and an operator in a VR headset can only work two hours a day before getting dizzy. A Lucky Engine scene takes a few hours to two weeks to build, then domain randomization is free, a simple task runs at about a hundred episodes a second, and the customer pays per hour of exported simulation. "Not even a tenth" of what teleop costs in time and money, he said, and cheaper on every subsequent order. The December release adds a reinforcement learning stack, with a target of 400,000 frames per second.
Game developers will solve robotics
The engine's first version shipped in early August after a year and a half heads down. The second is "a hundred percent agentic": you open Codex, Claude or Hermes, tell it to use Lucky Engine, upload a photo, and it builds the replica scene. Devrim described himself as DHH-pilled on this: people do not want tools, they want an agent that uses tools. There is always a sim-to-real gap, he said, and the engine may be too perfect, so noise goes back in. Yan and the most senior developers are now on the sim-to-real demos rather than new features.
His contrarian bet for the next twelve months is that game developers, not just AI researchers, will solve robotics. Most researchers come from Jupyter notebooks; game developers know how to ship software, and whoever can build a great level can build a great training level. He expects a Fab-store-style marketplace of tasks, backgrounds and domain randomizations, and the thirteen-person company already has forward-deployed game engineers who build scenes for customers. The ask: customers who want success stories, model companies willing to give the engine a shot, game engineers who want to join, and one or two researchers a cohort for a fellowship that pays them to break the engine and publish on it.
The hot takes
A humanoid folding laundry in a home with nobody in a headset: a year to a year and a half if the Astras and Claudes keep improving at this pace, and he hopes for robots doing things near us by the end of 2027, with the caveat that the G1 behind him once nearly took his finger off. The most wrong belief about simulation is not wrong, just skeptical, because the simulators have been bad. "If you let your cousin fix your computer and the computer doesn't work, is it the computer or is it the cousin?" And in five years, the share of robot training data that comes from simulation: 99 percent. "This problem requires simulation. There's no other way."
Full episode available on:
YouTube
Spotify
Apple Podcasts
Learn more about Lucky Robots: luckyrobots.com. Devrim is on LinkedIn and X.