Training a coding model to paint watercolours with TRL and OpenEnv

A coding model was trained to paint watercolors by writing JavaScript code that uses drawing tools to create images. The training used reinforcement learning guided by aesthetic preferences rather than a single correct answer, combining scores from a compiled code checker, code length measure, a vision model comparing paintings to hand-rated reference images, and a preference model trained on human image choices. The approach treats the hand-curated pool of reference images as the definition of what is beautiful, making the work about building that set rather than tuning technical parameters. All components of this system, including the training scripts, environment, models, and datasets, were published openly on Hugging Face.