Essay · Reinforcement learning
What the environment teaches
An RL environment specifies the learning problem. What an agent can observe, try, and receive feedback on shapes the behavior it can learn. Environment design deserves as much attention as the choice of optimizer.
Start with the interface
A policy acts on the information it receives. Privileged observations during training can create dependencies that deployment cannot satisfy. Partial observations may make memory or information-gathering actions necessary. Selecting an entire plan and choosing the next motor command are different control problems. Resetting after every failure also changes which consequences the agent encounters. These choices establish its opportunities before optimization begins.
Choose the feedback
Reward supplies the optimization target. Reaching a goal, minimizing elapsed time, and accumulating intermediate progress can favor different behavior. Delayed feedback makes it harder to connect outcomes to earlier decisions. If identical visible histories require different actions, more optimization cannot supply the missing distinction. Ask what information the intended task provides and what behavior the reward favors.
Test what carries over
CoinRun makes the evaluation problem concrete. Cobbe and colleagues trained agents on procedurally generated levels and tested them on unseen levels from the same distribution. They found substantial overfitting, even with many training levels.
That test addresses one kind of generalization. Transfer to different dynamics, interfaces, or operating conditions needs separate evidence. A high training score leaves those questions open.
My practical starting point: design training and evaluation environments together, keep test cases out of the learning loop, and state exactly what successful transfer demonstrates. The environment is part of the claim about capability.