Sign in Subscribe
INSIGHTS

I Built a Robot Arm and Trained It to Think. Here's What I Learned.

I Built a Robot Arm and Trained It to Think. Here's What I Learned.

I invest in robotics companies. I talk to founders building humanoids, manipulation systems, and autonomous machines every week. Now I've been trying to build a few robots myself.

I've been learning by ordering parts, assembling a robot arm from scratch, and training an AI policy to control it autonomously.

Here's what I did, what I learned, and why it matters.

The Build

I started with the SO-101, an open-source 6-DOF robot arm designed by The Robot Studio in collaboration with Hugging Face. It's the most popular arm in the LeRobot ecosystem right now — the same framework that a growing number of researchers and startups are building on.

The setup is a leader-follower system: two identical arms, one that I move by hand (the "leader") and one that mirrors my movements in real time (the "follower"). This isn't just a cool demo — it's the actual data collection method used by most cutting-edge robot learning systems today.

I ordered a full kit from PartaBot, sorted 12 servo motors by voltage and gear ratio, configured each motor's ID one at a time, assembled both arms from the base up, routed daisy-chain cables through each joint, and calibrated the whole system. Along the way I learned things you simply can't learn from a pitch deck: why motor horn alignment matters, why bus servos are cheap but imprecise, why a loose 3-pin cable at joint 3 kills everything downstream of it.

The Data

Once both arms were assembled and teleoperating, I added a Logitech C920x webcam and started recording demonstrations. Here's where it gets interesting.

A robotics "dataset" is fundamentally simple: it's synchronized camera images paired with motor positions at every timestep. Each frame is a snapshot that says "given what the robot sees right now, here's what the human operator did next." That's it. That's the training data.

I recorded 5 episodes of a simple pick-and-place task — grab a battery, move it to the right, put it down. Each episode was 30 seconds, with 15 seconds in between to reset the object. Total recording time: about 4 minutes.

The Training

I trained an ACT policy — Action Chunking with Transformers — which is the same architecture behind a lot of the cutting-edge imitation learning research right now. It uses a ResNet18 vision encoder to process camera images and a transformer to predict chunks of 100 future motor actions at once, making movements smooth and coherent.

52 million parameters. 100,000 training steps. Ran overnight on a MacBook M5 Pro using Apple's MPS acceleration. No cloud GPU needed.

The loss dropped from 0.070 early in training to 0.028 at the end. The network was learning. But learning what, exactly, from only 5 demonstrations?

The Result

I put the battery on the table, started the policy, and watched.

The robot moved toward the battery. It knew roughly where to go and what it was trying to do. But it kept hesitating — approaching, pulling back, approaching again. It never committed to the grasp.

And honestly? That's exactly the right result. With 5 demonstrations, the policy learned intent but not execution. It saw too few examples to confidently decide when to close the gripper, what approach angle to use, or how fast to move. The uncertainty shows up as hesitation.

What 50 Demonstrations Did

I recorded 50 episodes of the same pick-and-place task and retrained the ACT policy from scratch — same architecture, same 100K steps, same MacBook M5 Pro.

The difference was immediate and visible.

The arm moved with clear, committed intent toward the object. No hesitation, no approach-and-retreat loop. It reached, it oriented, it attempted the grasp. It didn't complete a clean pick every time — but the behavioral signature had fundamentally changed. The policy had enough signal to commit.

This is data scaling in its simplest form, running on consumer hardware, on a $300 robot arm, in a home office. The same principle that justifies the billion-dollar data collection operations being built by the major robotics labs — it just works at every scale.

Noise is less damaging than you'd think. A few of my 50 episodes were messy — I forgot to reset the environment, my hand drifted into frame, I fumbled the grasp. The policy still improved substantially. With enough good demonstrations, the bad ones get averaged out.

Why This Matters for Physical AI

This experience crystallized something I now think about constantly:

Data is the bottleneck, not the algorithm. ACT, Diffusion Policy, VLAs — these architectures all work. The question is always: how efficiently can you collect high-quality demonstration data? The jump from 5 demos (hesitation) to 50 demos (reliable grasping) to 500 demos (generalization across objects) is where the real engineering challenge lives. Companies that have solved data collection at scale have a moat. Companies still hand-teleoperating every demo do not.

The "last inch" is brutally hard. Getting an arm to move in the right general direction is easy. Getting it to close the gripper at exactly the right moment, with exactly the right force, on an object that's 2mm from where it was during training — that's where billions of dollars of R&D are being spent. I felt this viscerally when my robot hesitated. It knew where the battery was. It just couldn't commit.

Hardware matters more than people think. A loose cable, a miscalibrated joint, a motor horn mounted one tooth off — any of these can make a perfectly trained policy fail. The software gets the headlines, but the mechanical reliability and sensor quality of the physical system is often what separates a demo from a product.

Simulation-to-real is the scaling play. Recording physical demonstrations takes a lot of time. Recording 50,000 in simulation takes an afternoon. But the gap between simulated physics and real-world physics is still large enough to matter. Closing that gap — through better simulators, domain randomization, and sim-to-real transfer techniques — is one of the most important open problems in the field.

SmolVLA Rollout

Added after running the fine-tuned model on the physical arm.
July 2026

The training run finished overnight: 20,000 steps in 3 hours 18 minutes on the RTX 4080, with training loss settling around 0.015. The result is a 450 million parameter SmolVLA checkpoint, of which roughly 100 million (the action expert) were trained on our data - 25 episodes, 11,250 frames, one task: pick up the orange block and place it to the right.

Then I ran it on the arm. It reaches for the block and misses.

That sentence is the honest summary, and it is more interesting than it sounds. The arm moves with visible intent toward the block, gets close, and fails to land the grasp. The placing half of the task - move right, open the gripper - looked noticeably more confident than the pick. That asymmetry is not random, and understanding it explains a lot about how these systems learn.

The model, and the lineage behind it

SmolVLA belongs to a family called vision-language-action models, or VLAs. The inputs are camera frames, the robot's joint positions, and a text instruction. The output is a chunk of future actions - in our case, 50 joint-position targets at a time. One network, pixels to motion.

The idea is young. Until mid-2023, robot learning ran on two separate tracks. Robot policies like Google's RT-1 were trained purely on robot demonstrations - 130,000 of them, collected by a fleet of 13 robots - and still struggled outside their training distribution. Meanwhile, vision-language models understood images and text at web scale but could not act. Google DeepMind's RT-2, announced in July 2023, fused the tracks and coined the VLA term: take a web-scale VLM and fine-tune it to emit robot actions as if they were text tokens (Google DeepMind, 2023). Their announcement described it as a model that can "speak robot." The payoff was emergent generalization: RT-2 could throw away a piece of trash without ever being trained on the concept, because the web pretraining already contained it.

RT-2 had two problems for anyone outside Google: backbones running to tens of billions of parameters, and closed weights. The field spent the next two years shrinking and opening the recipe. Physical Intelligence's pi0, released in October 2024, replaced discrete action tokens with flow matching, which generates continuous action chunks in a single pass - fast enough for real-time control on physical hardware (Black et al., 2024). SmolVLA, released by Hugging Face in June 2025, is the community-scale endpoint of that line: 450 million parameters, pretrained on fewer than 30,000 episodes contributed by hobbyists under LeRobot's community tag, most of it recorded on the same class of SO-100/SO-101 arms sitting on my table (Hugging Face, 2025). The design is economical throughout. It keeps only the first 16 layers of its SmolVLM-2 backbone, compresses each camera frame to 64 visual tokens, and attaches a roughly 100 million parameter flow-matching action expert. During fine-tuning the VLM stays frozen; only the expert trains. That is why a task-specific fine-tune fits on one consumer GPU overnight.

ACT, which the earlier phase of this project used, comes from the same year as RT-2 but the opposite philosophy. It emerged from Tony Zhao's ALOHA project at Stanford: no pretraining, no language, a compact model trained from zero on your demonstrations alone (Zhao et al., 2023). Its lasting contribution was action chunking - predicting the next 50 to 100 actions at once instead of one step at a time, which cuts compounding errors and made fine manipulation learnable from tens of demos. On their hardware, ACT hit 80 to 90 percent success from 10 minutes of demonstrations. The chunk survived into every modern VLA, including SmolVLA. What changed is what the network knows before your first demonstration arrives. ACT starts blind. SmolVLA has already watched thousands of hobbyist episodes.

Why placing works before picking

Placing is close to an open-loop motion. The drop zone barely varied across the 25 training episodes, the tolerance for success is wide, and the model can nearly memorize the trajectory. Picking is the opposite problem. It requires closed-loop visual servoing: converting camera pixels into a gripper pose accurate to about a centimeter, at whatever position the block happens to occupy. The forgiving half of a task gets learned first. The precise half is what scales with data.

This is also why robot demo videos mislead. A clip of a successful place looks identical to competence, while the hard part of manipulation is concentrated in the few hundred milliseconds before contact.

How this compares to ACT

The earlier phase of this project trained ACT from scratch: 50 episodes of a battery pick-and-place task, 100,000 training steps, and the result was directional intent without task completion. This run used half the demonstrations and a quarter of the training steps, on top of a pretrained model, and produced similar or better reaching behavior.

I want to be careful with that comparison. Different task, different hardware setup, different data quality - it is suggestive, not controlled. But the direction matches what the SmolVLA authors report: their 450M model outperforms ACT on real-world SO-100 and SO-101 tasks despite training only on community datasets (Shukor et al., 2025). The pretrained base has already seen thousands of manipulation episodes; our fine-tune only has to teach it this arm, these cameras, this block. That is the cheap-data-plus-small-fine-tune thesis this whole project exists to test on real hardware, and the next run turns it into a controlled experiment.

One deflating detail worth knowing: at single-task scale, the language instruction is decorative. The model saw "pick up the orange block" paired with every training episode, so it learned "this is what I do" rather than how to parse a command. Ask it for a sandwich and it would still go for the block. Language becomes a real task selector only with multi-task data, which is where this is headed.

Lessons from the environment war

Getting to the training run took longer than the training run. The transferable lessons, compressed:

Python 3.13 silently kills processes when compiled ML extensions misbehave - no traceback, no error, the process just ends. LeRobot requires 3.12 as a floor, and for now 3.12 is also the stability ceiling. Second, pip extras silently replace CUDA PyTorch with the CPU build. It happened twice in one evening. The working order on Windows: install CUDA torch, install the package, then force-reinstall CUDA torch. Always check torch.cuda.is_available() before a long run - the identical job ran at 28.7 seconds per step on CPU versus 0.6 on GPU, roughly a 45x difference. Third, the LeRobot CLI is moving fast enough that --help is the documentation. Flags get renamed between versions, and policy rollout now lives in its own command with its own config rules.

For calibration on how normal our result is: an independent writeup of fine-tuning SmolVLA on comparable hardware reports months of iteration to reach a 40 percent grasp success rate (O'Keefe, 2026). Erratic-to-working is the expected curve, and we are on it.

Next steps, stated as predictions

Three changes, each with a testable claim attached.

The wrist camera comes back. It was dropped during recording because three cameras saturated the single active USB cable; a second cable is on the way. Prediction: this disproportionately improves the grasp phase, because front and side views resized to 512 pixels cannot resolve gripper-to-block alignment at contact distance.

The next dataset grows to about 50 episodes with deliberate structure: block positions covering the workspace in a rough grid rather than random placement, and a slower approach during the grasp segment so the model gets denser supervision exactly where precision matters.

Before retraining, the current checkpoint gets a measured baseline: ten rollouts with the block in a fixed position, successes counted. The same protocol runs after retraining. That gives the next update a single chart - success rate against training episodes and camera count - instead of an anecdote.

The arm misses today. What changed is that the miss now has a specific explanation and a dataset designed against it.

References

Update: Losing a fight to a cube that doesn't exist

July 10 2026

I wired the leader arm into a physics simulator. 140 lines of Python read six joint angles over serial at 60 Hz and mirror them onto a simulated SO-101 in MuJoCo, the open source physics engine DeepMind maintains. The virtual arm costs nothing, runs faster than real time, and never strips a gear.

The bridge took an evening, most of it spent reading tracebacks. LeRobot moves fast enough that its internals had shifted between releases, so the session doubled as API archaeology. Living at the head of an open source robotics stack is a tax paid in tracebacks.

Then I added a cube. Three centimeters, twenty grams, orange, resting on a checkered floor with full rigid-body physics. The task: drive the virtual gripper with the real leader arm and pick the cube up. The same grab I have practiced for weeks on the physical arm with a foam block.

Pushing the cube is easy. I can slide it, spin it, park it on the target pad across the floor. Ten minutes in, pushing felt natural.

Picking defeated me. The jaws close and the cube squirts out sideways like a watermelon seed. On good attempts it rises a centimeter, hangs, and slips out mid-lift. A clean pick has beaten me so far.

The asymmetry has a clean explanation. A push borrows physics from the environment: the floor carries the cube's weight, and any contact at all moves it. A pick gets no help. Two rigid surfaces must beat gravity through friction alone, which requires aligned contact, sustained force, and stable grasp geometry. Roboticists call it force closure.

Reality assists every grasp you make. Foam dents. Fingertips flex. Skin drapes over edges and multiplies the contact area. Default simulation offers none of that. MuJoCo resolves contact as a convex optimization over a handful of idealized contact points, recomputed every millisecond, per its documentation. My virtual cube and virtual jaws are ideal rigid shapes. Squeeze harder and the solver finds a mathematically valid outcome: eject the cube.

My first instinct came from the real world: add grip. I raised the friction coefficient from 1.5 to 3 to 4.2. Nothing changed. The cube kept slipping, and under grip pressure it started sinking into the floor. The coefficient was never the lever. MuJoCo's default solver configuration allows a little tangential creep at every contact, every step, regardless of the friction number. The fix was one line: an elliptic friction cone with a raised frictional-to-normal impedance ratio, plus stiffer contact parameters on the cube. Friction went back down to 2 and the pick holds.

That changes the lesson. Sim physics is downstream of configuration choices that never appear in a demo video. Tune them one way and grasps hold; leave defaults and no coefficient saves you. A policy trained under either setting learns habits calibrated to that world, and only hardware reveals which habits transfer. Domain randomization, training across thousands of perturbed physics variants, amounts to an industry-wide admission that no single configuration earns trust.

The result rhymes with last week's SmolVLA rollout on the real arm: placing worked, grasping failed. Different mechanism, same address. Difficulty in manipulation concentrates in the last inch, where contact lives.

For the investor notebook: when a founder says "we train in simulation," the follow-up is "how do you model contact, and how do you measure transfer?" Teams that answer with a number deserve the longer meeting.

References

Read more