ForeSight: calibrated opponent modeling in Pokémon

ForeSight fine-tunes Laya (421M parameters) to forecast opponent actions in Pokémon and uses the probabilities in depth-two expectimax search.

Type: Research
Timeline: Fall 2026
Status: AAAI-27 student abstract

Saiakhil Chilaka built ForeSight to test a small learned opponent model as an alternative to hand-written heuristics and per-turn LLM queries. The aim was to guide lookahead search without the latency and API cost of those queries.

ForeSight fine-tunes Laya, a 421M-parameter "System One" decision model that answers multiple-choice questions in a single pass rather than generating text. It assigns probabilities to the opponent's available actions, which a depth-two expectimax search uses to weight each branch.

01
Battle state
both sides' Pokémon, HP, moves, the last few turns
02
Forecast
Laya gives a probability for each opponent action in 36 to 84 ms
03
Look ahead
depth-2 expectimax weights each branch by those probabilities
04
Move
about 3 s per turn on a laptop

One turn of ForeSight.

Saiakhil generated the training data at no cost by logging decisions from a few thousand battles between simple bots on a local Pokémon Showdown server. Fine-tuning only the top four of Laya's 28 layers took about 35 minutes on an Apple laptop.

72–80%

of opponent moves predicted correctly, up from 43% zero-shot

75%

accuracy on Abyssal, a benchmark bot it never saw in training

56.5%

wins in 200 games against Abyssal, with no LLM

in API calls

Calibration matters because the search weights branches by their forecast probabilities. Predictions assigned 70% confidence are correct about 70% of the time.

Reliability diagram: forecast probability against observed frequency, close to the diagonal for both held-out bots and Abyssal. — Calibration of the fine-tuned forecaster. Points near the dashed diagonal mean the stated confidence matches how often it's right.

Forecast accuracy and win rate

ForeSight's 56.5% win rate against Abyssal is comparable to PokéLLMon (56%), an LLM-based agent, and below PokéChamp (64 to 70%). Looking one additional turn ahead improves performance by 13 to 20 points over one-step play, the clearest effect in the experiments.

Better forecasts did not produce more wins with the search we used. A simple damage heuristic played at least as well, and a uniform opponent-action model, which treats every available move as equally likely, won 55.5% of its games compared with ForeSight's 56.5%. The rough battle simulator and limited search depth were the bottlenecks in these experiments. Improving forecast accuracy alone would not address them, so the next round of work focuses on those two components.

Bar charts of top-1 accuracy by turn number and by number of options, well above chance throughout. — Forecast accuracy by stage of the game and by how many options the opponent had. The black ticks mark chance.

Earlier pilots

ForeSight grew out of a series of small, cheap pilots, each testing one question. In one, a small model reranked an LLM agent's moves in the text game ALFWorld, and success went from 34.6% to 48.4%. Others tried to predict poker actions and human Pokémon players zero-shot, which only partly worked. The repo has a write-up of every pilot's result, good or bad, along with the cost of every API call the pilots made. None of them cost more than a few dollars.