Robot Learning Infrastructure · Updated Nov 2026

Building a Franka FR3 Robot Learning Cell from Scratch

I built this Franka Research 3 manipulation cell at TASL almost entirely from scratch: the 80/20 bench, the real-time control stack, the cameras and gripper, a GELLO leader arm, and the data-collection and rollout pipeline on top. This page walks through the build in the order it happened, what the finished cell has produced since — 60 zero-shot rollouts of a generalist policy and the lab's 10-task demonstration dataset — and what I learned about where capable policies quietly break.

Franka Research 3libfranka 0.18 · FCI 1 kHzPolymetis / DROIDRT-kernel NUCZED 2i · ZED MiniRobotiq 2F-85GELLO teleopRLinf · LeRobotπ₀.₅-DROID / openpi
bread → plate
tabletop
pen → cup

Autonomous rollouts of a π₀.₅-DROID vision-language-action policy, deployed zero-shot on the finished cell — language in, tabletop manipulation out. No teleoperation, no scripting.

At a glance

HardwareFranka Research 3 (firmware 5.9.2) with a Robotiq 2F-85, two ZED 2i exterior cameras and a ZED Mini on the wrist, on a hand-built 80/20 bench.
ControlA real-time-kernel NUC owns the FCI link at 1 kHz through libfranka 0.18.1 inside DROID's Polymetis container; policies and teleop run on top at 15 Hz.
ComputeAn RTX 4090 workstation runs the cameras, the RLinf real-world environment, the dashboards, and openpi inference; a laptop only orchestrates over ssh.
Output60 zero-shot π₀.₅-DROID rollouts across six tasks, and the lab's 10-task, 250-episode GELLO demonstration dataset in LeRobot v2.1.
InsightInfrastructure failures look like intelligence failures. A camera frame that disagrees with the robot, a soft controller, or a 224×224 crop that erases a pen all show up as “the policy can't grasp”. Make the cell agree with itself before judging the model.
On this page1Bench2Arm & control plane3Control stack4Perception5Teleop & data6Architecture7Zero-shot results8Lessons

01 · The bench

It started as a pile of aluminum extrusion

Before any robot, there was a frame to build. The cell is an 80/20 aluminum-extrusion bench on casters — squared up, leveled, and bolted together by hand so the arm would have a rigid, repeatable base to mount to and the cameras would stay put between calibrations.

Aluminum extrusion bench frame on casters
The base frame squared up on casters, with the remaining rails still on the floor.
Assembling the extrusion frame by hand
Bolting the extrusion together — the unglamorous half of robot learning.
Franka FR3 mounted on the bench next to control laptop

02 · Arm & control plane

Mounting the FR3 and giving it a real-time brain

With the bench up, the Franka Research 3 (firmware 5.9.2) went on top. The arm is driven by a separate real-time-kernel NUC that owns the FCI link at 1 kHz and exposes a small robot server to the GPU workstation over the robot network — first through franky, later through Polymetis (§3). The NUC has no GPU and does nothing else: it is the robot's real-time hands.

Getting here meant working through the parts of robotics that never make it into papers: firmware compatibility, the right libfranka version, motion-generation gotchas, and a controller that recovers cleanly instead of dropping the FCI session mid-run.

03 · The control stack

Why I started on franky, then moved back to Polymetis

Exactly one process may speak libfranka to the arm, so choosing that layer is a real decision. The legacy option — DROID's Polymetis fork — was frozen on libfranka 0.13, out of spec for the FR3's 5.9 firmware (protocol 10 needs 0.18+), and would not even compile against our setup. So I first built the controller on franky: it tracks libfranka through 0.18, supports FR3 5.9.x officially, and installs with plain pip. Within a day the bench was running real 1 kHz control behind a small HTTP server.

Two things later pulled me back to Polymetis. The policy I wanted to deploy, π₀.₅-DROID, was trained on the DROID stack and expects its exact action and observation conventions, so running it faithfully — and comparably to published numbers — was easiest inside DROID's own container, which I rebuilt with libfranka swapped up to 0.18.1. And the lab's direction moved toward RLinf for real-world data collection and online RL, whose environment I gave a Polymetis controller backend. The result is a Docker image on the NUC that owns FCI and serves the robot over zerorpc; everything above it — teleop, data collection, policy inference — runs at 15 Hz on the workstation.

NUC1 (RT kernel, no GPU)                       FR3 (fw 5.9.2)
  container droid-nuc-fr3:0.18.1                   ▲
    polymetis driver + libfranka 0.18.1 ── FCI 1 kHz ┘
    zerorpc server :4242  ◄── workstation: RLinf env / openpi / dashboards @ 15 Hz
    Robotiq 2F-85 driver (RS-485)

rule: exactly one libfranka client. The franky robot_server must be stopped
      before the container takes FCI, or both hang. The launchers enforce it.

So it wasn't “franky wrong, Polymetis right.” franky made the hardware controllable quickly when the legacy stack couldn't even build; Polymetis came back once the goal shifted from make the arm move to deploy this specific DROID-trained policy and plug into this specific RL framework. A control stack is chosen by what the rest of the pipeline already speaks, not by which library is best in the abstract.

Robotiq gripper and wrist-mounted camera on the FR3 end-effector

04 · Perception & end-effector

Cameras, gripper, and the calibration layer

The end-effector carries a Robotiq 2F-85 parallel gripper and a wrist-mounted ZED Mini; two ZED 2i cameras observe the workspace from fixed viewpoints. That mirrors DROID's observation layout — one exterior view plus one wrist view, each resized to 224×224 for the policy — so a DROID-trained checkpoint sees roughly what it was trained on.

This part was less about adding sensors and more about making all of them agree with the robot. Each camera had to be calibrated into a consistent frame so that image observations, gripper poses, and tabletop geometry referred to the same physical world. The residuals came out around 3 mm, limited more by marker detection than by the arm's repeatability. The recurring nuisance was physical: the wrist camera's USB link dropped under motion until it was reseated and the launchers learned to probe camera readiness before starting a run.

This layer quietly decides whether everything downstream is possible. If the cameras and the arm disagree about where the table, the objects, or the gripper are, even a strong policy fails for reasons that look like “bad intelligence” but are infrastructure errors.

05 · Teleoperation & data

A 3D-printed leader arm, and the pipeline behind it

To collect demonstrations, I built a GELLO leader arm — a low-cost, 3D-printed kinematic twin whose joints map directly onto the FR3, so a human can puppet the real arm one-to-one. It is the bridge between “the hardware works” and “the hardware can learn”: every teleoperated episode becomes training data.

Collection runs through RLinf's real-world environment with the Polymetis backend from §3. A joint-space wrapper turns the leader's joints into the robot's 8-D joint-velocity action at 15 Hz — the same vector π₀.₅-DROID predicts — and each episode is written straight into a LeRobot v2.1 dataset: two 224×224 views, joint state, per-step intervention flags, plus full-HD stereo recordings on the side so frames can be re-cropped later without re-collecting.

Around it I built what a one-person collection session actually needs: a browser dashboard that starts and stops runs, marks an episode as a success with one key, shows the live camera feeds, and browses datasets with per-episode playback; and a pair of launch scripts that bring the NUC controller up, gate on FCI activation, home the arm, and tear everything down in the right order afterwards.

Deep dive: building & calibrating the GELLO arm →

GELLO teleop
Leader (left) and follower (right) moving together.
GELLO teleop
Reaching across the tabletop workspace.
GELLO teleop
Collecting another demonstration.

250

episodes across 10 tabletop tasks in the lab's FR3 dataset, 66,463 frames, collected on this cell with the GELLO arm

15 Hz

8-D joint-velocity actions, recorded directly as the dataset action

2 × 224²

camera views per frame, plus full-HD SVO recordings for re-cropping

Pilot datasets (stack a cube, assemble a lamp) came off the pipeline on June 22, 2026; the 10-task dataset was collected by lab members over the following months.

06 · The cell, end to end

Three machines, one robot

The finished cell is three machines with strict roles. The NUC is the real-time hands, the workstation is the brain and eyes, and the laptop only orchestrates. Teleop collection and policy evaluation share the same robot and camera infrastructure; only the process at the top changes.

LAPTOP ── ssh ──┬──────────────────────────────────┬────────────────────────────────┐
                ▼                                  ▼                                │
 DESKTOP  TASL-1 (RTX 4090)                 NUC1 (RT kernel, no GPU)               │
   2× ZED 2i + ZED Mini (USB 3)               container droid-nuc-fr3               │
   RLinf real-world env ── zerorpc :4242 ──▶   polymetis + libfranka 0.18.1          │
     ├ GelloJointIntervention ◄── GELLO         Robotiq driver                       │
     ├ CollectEpisode ──▶ LeRobot v2.1 + SVO        │ FCI 1 kHz                      │
     └ PolymetisController                          ▼                                │
   openpi serve_policy :8000 (eval)               FR3 arm  ◄── Robotiq 2F-85         │
   collect dashboard :8004 / eval dashboard :8003                                    │
   teleop.sh / teleop-stop.sh  (bring-up, FCI gate, home, teardown) ◄────────────────┘

Two constraints shape everything: FCI is exclusive, so exactly one controller process may exist and the launchers stop the other before starting; and the cameras and the robot socket are shared, so the collection and evaluation dashboards never run at the same time. Training never touches the bench — datasets go to the GPU cluster, checkpoints come back for evaluation.

07 · Zero-shot results

Where a capable policy quietly breaks

With the cell finished, I used it for a small in-the-wild study: deploying the π₀.₅-DROID generalist policy (openpi) zero-shot — no fine-tuning on our robot — across six tasks and 60 rollouts, 10 trials each, in June 2026. I went in expecting zero-shot transfer to real hardware to be brittle. The surprising part was how much did work: through a compliant control stack the arm moved smoothly and solved common pick-and-place tasks it had never seen on our exact setup.

The more interesting part was where it broke. Object grounding was usually correct — the arm went to the right thing — but precise grasping was the bottleneck: the policy hovered or froze above small objects, and a gentle push downward by a human was often enough for it to finish. Spoken ordering was not always respected, and a simple human handover failed every single time.

TaskSuccessPartialWhat happened
Bread → plate10/100Reliable large-object pick-and-place.
Pen → plate10/100Reliable despite the pen being thinner than the bread.
Pen → plate, then bread → plate8/102Often completes both, but sometimes grabs the bread first.
All objects → basket10/100Robust when the target is large and the order is free.
Black pen → red cup3/107Grounds the pen, then hovers or freezes before a stable grasp.
Bread → outstretched hand0/100Does not treat a human hand as a valid target.

Six tasks, 10 trials each. “Partial” means the policy grounded the right object and made progress but did not finish without help. The pattern that stuck with me: the control stack matters as much as the policy, capable models fail silently at the last few centimeters, and “it understood the instruction” is not the same as “it can do it.”

Recurring failure modes

  • Grounding is usually right; the last centimeters are not. Hovering and micro-adjusting above small objects without closing the gripper was the most common failure.
  • Temporal order is treated as a suggestion. After a failed pen grasp the policy switched to the easier bread, even when the instruction said pen first.
  • Human interaction does not transfer. The same bread the policy placed 10/10 on a plate was never handed to a hand.
  • Resolution and viewpoint cap fine manipulation. After padding and resizing to 224×224 a pen is a few pixels wide in the exterior view; the dashboard shows the policy's view next to the full feed for exactly this reason.
  • Out-of-distribution objects expose the priors. A 3D-printed light bulb and “push the toy car” produced no coherent strategy at all.
Read the full report (PDF) ↗

08 · Lessons

What I'd tell myself before starting

  • Pick the control stack by what the pipeline speaks. The best library in the abstract lost to the one the policy and the RL framework already expected.
  • Treat FCI exclusivity as a design constraint, not a bug. Most controller “hangs” were two processes fighting for the arm; the fix was launch and teardown scripts that enforce the order.
  • Verify the cell before judging the model. Calibration residuals, camera frames, gripper timing, and the policy's actual 224×224 view all masquerade as policy failures.
  • Design collection for one operator. A success key, live previews, a dataset browser with playback, and readiness probes turned a two-person job into a solo one.
  • Record more than the policy sees. Keeping full-HD stereo beside the 224² dataset frames made re-cropping possible without re-collecting a single episode.
  • USB cameras will drop. Plan for reseats and probe readiness at launch instead of discovering it mid-episode.

Timeline

  • May 2026 — bench assembled, FR3 mounted, first ZED captures, franky controller running at 1 kHz.
  • Jun 1 — Polymetis container rebuilt on libfranka 0.18.1 and validated end to end over zerorpc.
  • Jun 11 — 60-rollout zero-shot π₀.₅-DROID study written up; Polymetis backend merged into RLinf's real-world environment.
  • Jun 16–17 — GELLO leader calibrated, first teleop rollouts.
  • Jun 22 — joint-velocity collection pipeline, dashboard, and launchers live; first pilot datasets.
  • Since — the cell serves as the lab's FR3 bench for demonstration collection and policy evaluation.