Teaching a Local LLM to Play Pokémon Crystal on a DGX Spark

The Architecture and Final Product #

To avoid being like those cooking blogs I so desperately despise, I'm going to give you the TL;DR on what I built right up front:

I've spent the last month working on a harness that lets a local LLM play Pokémon Crystal on its own, in real time, with ways for me to monitor the progress and interfere if needed. The game runs in an emulator called PyBoy on my PC, and the brain, Qwen3.5-122B, runs on my Nvidia DGX Spark. Everything runs at home, except for the occasional question to Claude when it's completely hit a wall.

One rule I set early on was that the harness isn't allowed to know it's playing Pokémon; the harness had to be completely agnostic of the game. Everything game-specific lives in one piece called the adapter. It reads the game's memory and ROM directly (where the player is standing, what's on screen, who's talking, what's in the party and the bag, where every door goes) and hands the rest of the harness a clean snapshot. Everything else only sees things by their role, so a Poké Ball is just "something you can throw at a wild creature" and badges are "milestones." The idea is that the next game only needs a new adapter and a new knowledge graph.

The main loop makes one decision at a time, and before it asks the model anything, it figures out what's going on. A battle goes to a separate battle agent. A question on screen goes to its own step that reads the whole question and picks an answer. If someone's talking, the harness reads the conversation all the way to the end, then has the model boil it down into leads and facts. Only when things are quiet does the planner get asked what to do next. It answers with a tool to use (walk somewhere, talk to someone, use an item, teach a move, etc) and a one-sentence "why" that shows up on my console as its thoughts. That's where most of the gallery at the bottom comes from.

The model never touches the D-pad. It says something like "travel to Violet City" and the navigator does the rest. It works out which maps to go through using the game's own warp tables, then paths each floor with A* on the game's actual collision data. It knows about ledges, surfing, trees you can cut, holes you're supposed to fall through, and, after a lot of painful iterations and thinking, slippery ice.

Battles get their own agent. Code handles things that must never go wrong, like not picking a move with no PP left or not knocking out the thing you're trying to catch, and the model picks a strategy from whatever moves are left.

For memory, every run keeps a journal, and there's a knowledge graph that sticks around between runs: around 4,000 nodes of places, people, items and facts. Most of it the agent learned while playing, and the backbone was mapped out ahead of time (more on that below). The graph also holds "recipes" (not sure why I didn't name them skills but alas I'm unique), which are little how-to guides for things like healing at a Pokémon Center or weakening something before you try to catch it.

When it gets stuck, there's a ladder for it to climb through to prevent cycles/issues. It remembers what already failed so it doesn't keep trying the same thing, it gets nudged when it's going in circles, it can stop and think really hard, it can ask Claude for advice (threw $20 into API calls I still haven't used up), and if none of that works, it asks me (or Claude moonlighting as me while I'm afk).

All of this shows up on a console I built, mainly, the live game, its thoughts, the logs, its leads and party, and a chat box where I can nudge it or just type the exact command I want it to run. I've even sent some interested friends a read-only version through Cloudflare.

All in all, it's about 13,900 lines of game-agnostic harness and 5,500 lines of Crystal-specific code in the adapter.

If you're interested in the design process and the many, MANY iterations it took to get here, keep reading. It was probably the furthest thing from functional when I started, and here we have it closing in on the 8th gym. There's also a gallery at the bottom with some of my favorite silly "thoughts" from the AI, plus a few other mechanics I thought were neat. It's the journey, not the destination, after all.

The Story #

The First Night #

I dragged my husband to Micro Center. After some (but honestly not nearly enough) thought, I had decided what I needed in my life was an Nvidia DGX Spark. I told my husband all the cool ways it could support my career, my education, and my personal growth. Of course, the first thing I was actually going to do was make it play the best game of all time: Pokémon Crystal.

I spent a few minutes admiring the lovely rose gold coloring of the Spark (it matches my office perfectly), walked through a nice, easy setup, and got Qwen3.6-35B up and running. Plenty of room for it to play, with some left over for me to experiment on the side. I put together a neat little harness with PyBoy and some basic prompting and set it up to run overnight. If a goldfish could play Pokémon, a 35-billion-parameter model certainly could. I watched it get started, name itself Chris, and then stand in the starting room, which gave me all sorts of nostalgic feelings. I tucked myself into bed and excitedly told my husband all my plans for the Spark after it beat Pokémon.

The Next Morning #

I woke up like a kid on Christmas morning and scurried to my office to see what I'd find. Would it be at the first gym? Would it have chosen its first Pokémon? Maybe it had caught a few extras too. I don't know why I expected all that from the bare-minimum setup I had, but I was immediately let down when I saw that the model hadn't even left the starting room. It was stuck, frustrated, and bouncing around the same spots (although it did figure out how to interact with objects in the room). It couldn't find the exit and had done literally nothing overnight. It had now appeared to me this might be a little harder than I had thought.

I started brainstorming what would make a good harness. I decided to give it a nice list to keep track of what it was doing and any important quests, because SURELY it wouldn't need that much context... After a day of struggling, it eventually made it downstairs to the conversation with Mom and the neighbor (two characters it just could not seem to recognize). After hours of tweaking the prompt and the list, and a strange fixation on daylight savings time, I realized that maybe this model was a little too small for my big dreams.

I shut down the game, deployed Qwen3.5-122B, and let it rip, noticing that this time the reasoning was much better. Maybe I could just keep going bigger? To my dismay, a quick Claude chat told me 122B was about the max I could get out of one Spark. I either had to make this size work or cough up another $4.6K for a second Spark to run something bigger. As tempting as it was to go right back out and spend more money, I decided to work on the harness a bit more and see what I could improve. After all, with the smarter model, we had finally made it out of the house, so it MUST be only up from here?

Downsizing the Goal and Improving Movement #

While ambition is great, I immediately realized telling the model to "beat the game" was maybe a little too vague for a goal. Instead, I scoped the goal down to just the second gym badge. There are a couple of caves and challenges along the way that would require the model to do some smarter things, things that took me weeks to figure out as a kid playing Crystal. I'd call it a success if it could operate at the level of young child Amanda. I also stopped letting the model decide when it was done. At one point it proudly announced that it had reached Cherrygrove City and completed the goal. It was nowhere near Cherrygrove City. This led to the harness fact checking the game itself instead of trusting the model.

With a more manageable goal in place, I realized that watching the model bumble around like it was playing with its eyes closed was kind of annoying. I'd watch it miss doors, overshoot the characters it was trying to talk to, and get lost in its own house because of poor pathfinding. I had it sending screenshots to the model to figure out where to go, but that wasn't helping much. I ended up finding an A* implementation online that did wonders for movement. Pathfinding isn't an "AI problem" in the sense of the best course of action is to throw a model at it. It was solved a long time ago by folks who wanted software to play video games in a pre-LLM era. I created a movement agent, because asking the AI about every single step was slow and tedious and wasn't getting me anywhere quickly.

To my joy, after many, MANY attempts, the model walked "Chris" over to Professor Elm's lab and actually picked a starter: Cyndaquil (it was the closest one). Then it proceeded to promptly get stuck on the nickname screen, because the picture of the Pokémon that flashes up first apparently freaked it out a bit.

A First Pokémon... and Another... and Another #

This was my first dance with in-game choices, so there was a lot of fine-tuning here. I had it think through every decision box. I had it click through every decision box. I had it summarize conversations. I had it press B when it recognized the nickname screen. I had it think of a nickname. So many iterations to see how the model would react to these situations. So many restarts, stumbling to the same place.

I ended up keeping tabs on which Pokémon it would choose and what it would name it for my own amusement (metrics at the end for yours). In one of my favorite runs (before I was keeping track, unfortunately), it named its Cyndaquil "Chickorita" because it really wanted a Chikorita but couldn't figure out how to pick the Poké Ball with Chikorita inside.

It finally made it out of the lab, only to immediately turn around and head back home to adjust the clock for daylight savings. Eventually it stumbled into the tall grass, where I got to watch it (fail miserably at) battle (it was a new menu), so I built a simple battle agent to start things off.

It picked the best move for the situation, but before anything else it always checked for a move that would drop the opponent's HP to 0, and if it found one, it used it immediately (I'm sure you can see why that's going to be a problem eventually). But for now, we had our starter and were on our way.

Back to Work #

The weekend is not endless, and I eventually had to go back to work. Since so much of this was waiting for it to run, I wanted a way to keep an eye on things. I built a console for myself with a bunch of stats, the game logs, and a few other metrics so I always knew what was going on. I made a watch page and exposed it through Cloudflare so I could check in on the go (and share it with a few curious friends). I also added a way to nudge it from the console, and set Claude up to keep an eye on the gameplay and act as an advisor while I was away. This was great, except for the feeling of doom I'd get watching it get stuck right away, knowing all I could do was watch it struggle until I got home and fixed the harness, or when Claude just didn't give great advice (it was kind of mean to it too :( ).

On top of that, I kept every log the harness produced, down to every prompt and every model call. Partly for auditing, partly in case I ever want to do some training or something fun with the results, and partly so I could have Opus go through a run and tell me about the interesting parts when I couldn't watch myself. The logs came in especially handy for the Bug-Catching Contest (we'll get to that) and for auditing 'stuck' loops, which I ended up writing a whole separate skill for, since the early runs were VERY prone to them.

Slowly Getting Smarter #

After watching it struggle over and over to get a starter and find its way to Mr. Pokémon's house, I realized my harness needed to be a little smarter. I'd done some work with knowledge graphs before and thought this might be the perfect opportunity to integrate one. The model could store anything it learned in the graph and use the relationships to look things up later.

This seemed to be exactly what was needed to push it out of the very beginning of the game. The agent started getting much further, and I almost shed a tear when I saw it catch its first Pokémon. It took a lot longer than I'd like to admit to get it to reliably follow a chain of connections (for example, it needed to take the Mystery Egg from Mr. Pokémon back to Professor Elm to unlock the Poké Balls it needed to catch a second Pokémon). It was taking baby steps through the first part of the game, but was still making progress and boy, I was so proud.

I also started adding skills to the graph, (or "recipes" as I decided to call them for some reason) for doing things in the game, usually prompted by an observed struggle or epic failure.

One was for catching: get the wild Pokémon's health as low as possible before throwing a ball, so the catch rate goes up. This is right about where catching fell apart. The moment a wild Pokémon got low enough, my wonderful knock-it-out-first battle logic took over and finished it off instead of letting the catch happen. I also had the graph track in-game data, like which Pokémon it had already caught (we ended up with a couple of Caterpie, since it had no way of knowing who was already sitting in the PC). Battles were (mostly) seamless, and it seemed to be making fair progress. This appeared to hold steady up until the first gym.

On the way to the first gym, I started to see the flaws of letting it build the graph itself. It made a LOT of incorrect connections, confusing Mom and the neighbor, Professor Elm and Mr. Pokémon, and later even accusing a guard of moonlighting for Team Rocket (see gallery). This caused a lot of confusion, and the agent would constantly retrace its steps and redo tasks it had already finished because of bad information in the graph.

My favorite example of this comes from one run in particular; it spent almost 40 hours trying to heal its team and never managed it once. Part of that was the game; it only loads the people who are near you, so from the doorway of the Pokémon Center, the nurse technically didn't exist, and it kept talking to the guy standing by the PC instead. The rest was the graph; the old man who gives you the tour of Cherrygrove says the Pokémon Center will "heal Pokémon in no time," and the graph filed that under Cherrygrove City itself. So after every failed attempt, it would walk right back out of the Center to go heal in the town. Or it would head back to Mr. Pokémon's house, where it had been healed exactly once, through a route full of tall grass. 516 tries, zero heals, and a Chikorita over leveled and barely hanging on at the end. Maybe letting it dictate the entire graph wasn't the best idea.

This led me to bring in what I call constant nodes. My good buddy Claude Fable took a trip through the game and mapped out the important things, people and places, and how they're all related. I also had it audit the existing graph (the agent had been super busy mapping everything it could), and it found a LOT of issues. From then on, the agent could still write notes and lessons on the nodes, but it couldn't change the relationships between any of the major nodes in the game.

With that change, it took far fewer ticks to figure things out, and one magnificent run made it all the way to Azalea Town and the second badge. Technically that was the whole goal, but by then I'd already ever so slightly moved the goalpost to beating the game, as I knew this was a project I was going to stick to. At this point I stopped restarting and kept that run going. It had taken a couple of weeks to get here, but I was ready to see it all the way through.

Steady Progress #

A lot of the changes from this point on were to the graph. I had Fable 5.1 do a harness audit when it came out, mostly to see what the new model was capable of, and it optimized a few features for me, but otherwise mostly added skills and tweaked the knowledge graph.

Loop sensitivity needed a lot of adjusting too. Sometimes it would get into intricate loops that never got detected, and other times it would flag loops that weren't really loops (say you actually needed to go back and talk to someone, but the agent had already chatted with them a few too many times earlier, so technically it counted as a loop). There was a lot of smoothing out there, including a "changed state" indicator: whenever something meaningful changes in the game, the loop history resets, so it's allowed to go re-talk to NPCs.

Interesting Happenings #

There were a few super interesting things that happened outside of the silly dialogue. The first was when it entered the Bug-Catching Contest (of its own free will) with just a level 6 Sentret. It stood there for a bit. For those unfamiliar with the game, the contest holds the rest of your team at the gate, so its party went from six Pokémon to one in a single step, and the harness decided that couldn't possibly be real data and ignored everything for about 900 ticks.

I nudged Claude to see what the holdup was, and it assured me it got things moving. Then I stepped away from my computer, and by the time I got back the contest was over, so I asked Opus how it had done. Here's what it said:

How the contest went. Not well: it entered with Sentret alone, met a Scyther, threw two Park Balls that both failed, then fled when it was about to be knocked out, and caught nothing. When the timer ran out, Cooltrainer Nick was declared the winner and the run got a Berry as a consolation prize. Most of the contest was played on the old code, where the fight menu was not even recognized, so it was never going to go well.

I'd say the most interesting battle was right after it caught a Voltorb, which it appropriately named ZapBall (after randomly deciding to deposit Pidgeotto in Box 2 before challenging Team Rocket in Mahogany). Voltorb knew a fantastic move called Self-Destruct. Anyone familiar with the series knows where I'm going with this; it does a LOT of damage, but it also makes the Pokémon using it faint. This created a soft loop where my knock-it-out-first logic kept picking Self-Destruct, which knocked out the opponent, and Voltorb right along with it. Recognizing this as a one-off, I decided I wasn't feeling a complicated fix and just nudged the agent to teach it Rollout instead. It ignored me for about a dozen ticks and wandered off toward the Lake of Rage. So now, if I type a message that's formatted as an actual command, like teach {"move": "TM04 ROLLOUT", "to": "ZapBall", "forget": "SELFDESTRUCT"}, it skips the model entirely and just does it. ZapBall has since evolved into an Electrode. This gave a much needed split between nudges to work towards and things you need to do NOW.

It also got VERY lost in the Olivine lighthouse and walked up and down its floors for days. The funny part is that it made it to the top in seven ticks. Getting back down took 3,170, because the floors are split into sections and the way down isn't the way you came up. I used that for some further refinement of pathfinding and dungeon navigation, including stopping it from resetting its goal on every floor.

There are six floors in that lighthouse, and asking a non-deterministic system to re-evaluate its goal every time it walks down the stairs turns into a LOT of up and down. A little while after it finally got out, it walked up to the Lake of Rage, where the game asked if it wanted to Surf. It said no, because Surfing was "a travel mechanic, not a battle or training opportunity." It needed to cross the lake to progress.

That work definitely helped later on with other complicated dungeons, although the Team Rocket hideout under Mahogany Town still managed to give me the longest loop of the whole project. The agent had already heard both passwords it needed, and then it spent 39 hours walking in circles across three floors. The door to the boss's room is drawn exactly like a wall, and the only way to reach it is to climb up a ladder to the floor above and come back down a different one. Once the harness learned that some walls might actually have a little more to offer than blocking, it walked the ladders by itself and the door opened on the first try.

The Ice Path was VERY painful to watch as well. It took a while for the harness to understand what the ice was actually doing, and why the agent thought it had stepped one tile but ended up somewhere completely different. The second floor of the dungeon is a puzzle that requires boulders pushed into holes so there'd be something on the ice below to stop the slides, which meant building a whole new push tool for the strength ability.

The Latest Updates #

I excitedly write this minutes after it's reported it's officially out! It pushed all four boulders, slid its way through, and walked into Blackthorn City, where it found out the gym is closed until Team Rocket gets kicked out of the Radio Tower back in Goldenrod. So close. Its next move was to walk all the way back down to Union Cave, where it crashed the harness nine times in a row (but that's fixed now).

So it headed back to Goldenrod, and now it's apparently trying to rescue Team Rocket?? From the Radio Tower?? (I checked the logs. It's trying to rescue the Director from Team Rocket, which is at least the right idea.) So far it has tried to take the radio station's quiz in the middle of a hostile takeover, fought its way up to the top floor, gotten a key off the fake Director, and gone looking for the real one in the Underground.

It never found the warehouse he's locked in. Instead it decided he had "already been freed" and hiked all the way back through the Ice Path to Blackthorn anyway. The gym was still closed. Last I checked, it was on Route 31 looking for a trainer who had called it for a rematch, and its notebook proudly reported that it had "executed 50 go_to commands between Route 31 and Violet Gate; confirmed stable map connectivity." Good to know.

Next Steps #

All in all, this was honestly one of the funnest projects I've ever done. There were so many times I wanted to just quit and pay for some tokens to see what a more intelligent model would do, but I persisted, and it's apparently paid off. I found myself watching and thinking "this plays just like I did as a little kid," so I guess mission accomplished?

I'm thinking next would be to have it replay the game to see if the graph holds up on a second run, and how it handles the early game with a complete graph. I'd also like to build a graph for an easier game (Pokémon Red) and a harder one (Pokémon Ruby) and see how the harness adapts. It's pretty slow, so I'm locked into turn-based games for now, and I'll probably stick with Game Boy games since I've got a delightful adapter pattern I can reuse, but I'm sure I'll eventually venture out to other titles/platforms.

I'd also like to try a smarter model (Nvidia, if you're reading this and you've made it this far, imagine how much cooler all of this would be with a few more Sparks) and see how many of the annoyances I'm running into come from the model versus the harness. I might also try a smaller model, but I think getting the run this far has worn my patience down quite a bit.

Big Takeaways #

Now that it's closing in on the last badge, here's what I'd tell anyone thinking about trying something like this (or past me, standing in Micro Center).

Most of its "dumb" moments were really about what I was showing it. When the agent did something strange, the model was usually reasoning just fine about the wrong picture of the world it was provided; The nurse it couldn't find didn't actually exist from the doorway, the "keyboard" it once tried to type a nickname on was the professor's bookshelves, the door it walked past for 39 hours looked exactly like a wall, and so much more. Every time I fixed what it understood about its environment, it would get much better at the game.

Don't make the model do what code already does well. Pathfinding, menus, making sure a move actually has PP left; none of that needed a 122-billion-parameter opinion. Every time I shifted some responsibility from the model to some sort of coded component, things got faster and more reliable, and the model got to spend its effort on the stuff that does actually need judgment, like who to talk to and what to do next.

More memory isn't automatically better. I assumed a knowledge graph could only help and forgot I was dealing with an LLM that hallucinates. This was quickly remembered when it decided a guard was moonlighting for Team Rocket and that all of Cherrygrove City was a Pokémon Center. Letting it build its own picture of the world from scratch meant it believed a lot of its own confident nonsense, and it would act on such. What worked best was a solid backbone it isn't allowed to change, with room for it to add notes on top. The same goes for conclusions/lead completions. Early on, one run's end-of-run summary wrote down that the game was "unplayable," and the next run believed it resulting in a pretty interesting start to the run. Now it only records what happened, never what it thinks it means.

It's good at the next move and bad at the long game. A lot of the harness ended up being about helping it stick with a plan: it has to commit to an intention, each one gets a limited number of ticks, and it has to give an actual reason before it's allowed to drop one so we're not just patting ourselves on the back for no job well done.

Getting stuck is normal, so plan for it. In a game this big it gets stuck CONSTANTLY, so instead of treating that like an error and restarting every time, I built a nice plan to execute around. The biggest lessons there were to measure "stuck" by what's changing in the game instead of how long it's been, and to know when to stop asking for advice and maybe look at the harness/kg. At one point the harness asked for help 61 times about a loop that turned out to be a bug in my battle code, and you can't exactly reason your way around a bug.

Save everything. EVERYTHING. Every prompt and every model call is saved to disk. Almost every fix in this project started with me (or Opus) reading exactly what the model saw right before it did something dumb, and more often than not, the answer was right there.

The harness mattered more than the model. Going from 35B to 122B was a major upgrade to quality of life and reasoning, and I don't want to undersell it. I didn't want to forget to acknowledge, however, that the biggest jumps in performance came from fixing and upgrading the harness. The first badge came two days after I fixed healing, and the seventh came in the first run after I fixed the hideout door. I still want to see what a bigger model can do (hi again, Nvidia), but I'd be kidding myself if I didn't recognize that the bulk of the work lives in the harness.

Thank you so much for reading this far, and hopefully some of the gallery images bring you the same joy they brought me when I first saw them!

By the Numbers #

As promised, the metrics I kept for my own amusement:

Starters: across 39 fresh games, it picked Cyndaquil 15 times, Chikorita 12 times and Totodile 12 times. The fastest it ever got one was 6 ticks (a Totodile named Finn).
Favorite names: Leaf (Chikorita, 5 times), Spark (Cyndaquil, 4), Toto (Totodile, 4), Ember (Cyndaquil, 3) and Splash (Totodile, 3).
Named its starter after itself: 3 times. Chris the Cyndaquil (twice) and Chris the Chikorita.
Named things after exactly what they are: five Pidgeys named PIDGEY, four Sentrets named SENTRET, two Togepis named TOGEPI, one Chikorita named CHIKORITA and one Cyndaquil named CYNDAQUIL. Honorable mentions go to a Raticate named Nibbler, a Cyndaquil named ABURNER, a Hoothoot named RATTY, and a Zubat named SPINNER.
The current team: Leaf the Meganium (Lv 57), Sentret the Furret (Lv 48), Sudowoodo (Lv 47), ZapBall the Electrode (Lv 42), Togepi the Togetic (Lv 42) and Zippy the Crobat (Lv 42). I was tempted to bring Pidgeotto back into the mix until I saw the name of ZapBall.
Overall: 187 runs, over 62,000 ticks, about 5,600 battles, 75,000 calls to the model, 434 "deep thinks," and 18 times it gave up and asked me for help.
What the model would have cost through an API: all those calls add up to about 320 million input tokens and 10.5 million output tokens (the first few days are estimated, since I wasn't logging input tokens yet). At today's cheapest rates for Qwen3.5-122B ($0.26 per million tokens in, $2.08 per million out), that comes to about $105, and even the priciest provider I found would have been around $300. The Spark cost me roughly 40 times that. So if you're considering a Spark just to save on tokens for a project like this, this is not the article you should link to justify it.