Introduction
What if a robot could learn to walk without anyone writing a single line of locomotion code?
We built a complete reinforcement learning (RL) pipeline for the Boston Dynamics Spot quadruped, starting from a robot that can’t even stand, and ending with one that navigates obstacle courses using only a depth camera. Every behavior emerged purely from reward shaping and trial-and-error at GPU scale.
We used NVIDIA Isaac™ Lab to train thousands of virtual Spots in parallel on a single laptop GPU. This post covers the four training stages, the reward engineering lessons we learned the hard way, and the moments the robot outsmarted us.
Why Teach a Robot Dog to Walk?
RL for robotics is no longer academic; it is what clients actively ask about. At Ekumen, we wanted hands-on expertise with Isaac Lab before real-world deadlines demanded it. We chose the Spot because quadruped locomotion exercises every part of the RL pipeline: contact dynamics, gait generation, terrain interaction, and goal-directed behavior.
The full stack runs on an NVIDIA RTX 3070 Laptop (8 GB VRAM) with RSL-RL’s PPO implementation, proof that you don’t need a server rack to train advanced locomotion policies.
Stage 1: Standing Up
We spawn Spot with scrambled joints and ask it to get upright. There is no locomotion and no navigation goal, just a simple objective: don’t fall.
The key insight was implementing a push force curriculum. Once the robot survives 70% of each episode, external pushes grow from 50N to 500N. Without this, the robot learned to lie flat on purpose, the safest strategy when you can’t resist external forces. A structured curriculum allows the policy to build confidence before facing harder challenges.
Stage 2: Walking
Now Spot follows velocity commands on cobblestone terrain. Our first attempts produced “the shuffle”, feet dragging along the ground, technically tracking the command, technically earning maximum reward, but looking absolutely ridiculous.
The fix required a gait reward enforcing a trotting pattern. Diagonal foot pairs must move in sync, while opposite pairs must alternate. This single term (weight +10) transformed the awkward shuffle into a natural, dynamic trot.
We then added rough terrain featuring seven procedural sub-terrains and a terrain curriculum that promotes robots to harder rows as their locomotion improves.
Stage 3: Waypoint Navigation
Instead of velocity commands, the robot now receives 5 waypoints and must reach them in sequence, discovering its own gait and heading purely from goal-relative observations (distance and direction to the current and next two waypoints in the body frame).
This stage taught us the most about reward engineering through two memorable failures:
- The robot parks at waypoint 1 forever: The approach reward peaked right next to the goal. Hovering earned 3.0 per step indefinitely, far more than the +10 crossing bonus. The fix: Zero out the approach reward inside the arrival radius, boost the crossing bonus to +50, and add a time penalty for loitering.
- The robot does a backflip at the last waypoint: The policy discovered that one explosive jump generated enormous forward velocity, earning a massive reward for a single timestep. The existing torque penalty was too small to discourage it. The fix: Forward reward only counts when feet are in contact with the ground, paired with a direct vertical velocity penalty (-5) and a increase to the torque penalty.
Every time we thought the reward was right, the robot found a loophole. RL agents are optimization machines, they find the highest-reward strategy, not the one you intended. Reward engineering is adversarial debugging.
Stage 4: Obstacle Avoidance with Depth Camera
The final challenge: navigate waypoints while dodging randomly placed cylinders using a forward-facing depth camera ( pixels, flattened to 1,200 values). Combined with proprioception and waypoint goals, the policy receives a 1,253-D observation vector.
Between five and ten red kinematic cylinders are scattered in a 2–8m ring around each robot on every reset. The reward structure adds a depth penalty (-3, based on the closest object in the image) and a geometric proximity penalty (-2, acting as a backup for obstacles outside the camera FOV). Entering an obstacle’s radius immediately terminates the episode.
An interesting side effect emerged: once the camera was introduced, the robot finally learned to walk forward toward goals instead of approaching them sideways or backward, as the depth image gave it a practical reason to face its direction of travel.
Results and Takeaways
All five tasks were trained on a single NVIDIA RTX 3070 Laptop GPU (8 GB VRAM), progressing from basic standing balance through camera-guided obstacle avoidance.
- Start simple, then stack: Each stage built directly on the previous one. Standing taught balance, walking taught gait, waypoints taught goal-seeking, and obstacles taught perception. Skipping stages means debugging four complex behaviors at once.
- Budget for reward debugging: The code to define a reward function is five lines long; getting the weights right takes a full training run. We spent ~70% of the project on reward engineering, not on simulation setup, infrastructure, or framework compilation.
- You don’t need a data center: Isaac Lab’s parallel simulation makes robot RL accessible on standard consumer hardware. While vision and camera tasks push the limits of an 8 GB VRAM buffer, they still execute reliably.
Now it is your turn! Train a Spot, break the reward function, and discover your own backflip moment.
Have Your Own Reward-Hacking Story?
If you’re evaluating Isaac Lab or reinforcement learning for your own robotic fleet, or if you’ve hit a humorous reward-hacking moment of your own, we want to hear about it.
This is the kind of hands-on, exploratory engineering we do at Ekumen before a client’s deadline demands it. Talk to our robotics engineers to learn how we can accelerate your physical AI stack.