Today's frontier models are seriously impressive. Claude Fable 5 is capable of fully autonomous software engineering tasks spanning days on end. If you've used it recently, you know how big a step up it really is over the previous generation of Claude Opus. The same can be said of other models like OpenAI's GPT-5.6 Sol and Moonshot's Kimi K3, posting very impressive benchmarks.

But there's a question worth asking: does everything have to cost so much money? Fable 5 costs $50 per million output tokens. Can we get away with a cheaper model and still get a useful result? To answer that question I took a relatively cheap model — DeepSeek V4 Flash, at $0.17 per million output tokens — and built a harness around it to see if it could tackle one of the harder problems around: earning a medal on a difficult Kaggle competition.

The competition is Spooky Author Identification, which closed in 2017, so there's no live leaderboard to enter; the run was scored offline against the private answer key using MLE-bench, OpenAI's methodology for exactly this kind of offline evaluation. This month, an agent running on the DGX Spark on my desk produced a silver-medal-grade result: 0.242 log-loss against a 0.270 silver threshold — around the top five percent of the roughly 1,250 teams on the original leaderboard. The agent proposed the experiments, wrote the code, and built an ensemble that grew to thirteen members, seven of them fine-tuned transformers, completely autonomously — and it spent $3.72 in tokens doing it.

Ten days earlier the same flash-tier model — running a newer snapshot of the weights, even — burned six days and $180 of cloud GPU time and finished with no medal-grade anything.  Same model family, same competition, same objective, and the failed run actually had the newer weights. The only thing that changed was the machinery around the model. I now think that machinery is where most of the performance lives, and that teams weighing frontier API prices for agentic ML work are pricing the wrong layer.

The harness is a fixed graph: code decides what happens next, and the model only writes the content of each step. Almost every step is an actor–critic pair inside a bounded retry loop — one agent does the work, then a critic agent or a deterministic check (pytest, a format gate, the no-op probe) judges it, and a failure bounces the work back with feedback for another attempt. 

Before I ran the medal-winning run on my DGX Spark, I ran it on a cloud GPU instance on the Lambda cloud. After nearly $180 of spend, I finally realized that each hillclimb iteration was taking over 12 hours, and it wasn't using the GPU. On its face, that might be ok — not every model needs a GPU. But the long runtimes said something deeper was wrong.

The agent had written its feature extractors to run single-threaded on the CPU! Doh! Training that should have taken minutes took twenty hours per experiment, and nothing in the loop could see it, because the only signal flowing back was the validation score — and the score was fine. Slow code that scores well is invisible to a harness that measures nothing but the number. By the time I understood what was happening, the run's three most promising experiments had been killed by timeouts and impatience, mine included. I later re-ran them on a GPU with proper batching: two of the three were real improvements, including the best single idea of the whole run. The harness hadn't just wasted money. It had discarded the best answers!

I fixed this by adding instrumentation. Every step now records wall-clock time and GPU utilization alongside the score, a review step checks each experiment's code against the machine's actual hardware before any training budget is spent — now we catch it when the agent codes something lame — and a hung process gets its whole process tree killed at a deadline, with the killed step marked failed.

There was another subtle bug. The harness always trains with the same fixed command — the model doesn't get to change how its code is invoked. But the cheap model kept wiring new behavior behind a command-line flag of its own invention, defaulted to off — think "--use-char-ngrams", which nothing ever passes. Doh! The tests passed, training ran the same path as before, and the champion's score reproduced to fifteen decimal places — the experiment had tested nothing. I patched the prompts twice, naming the failure explicitly. It kept doing it. What worked was thirty lines of code: run a one-epoch training pass and byte-compare the predictions against the champion's. Identical bytes mean the change never ran, and the gate bounces it back. That failure mode never survived another iteration.

Prompts influence behavior, but there is no guarantee an LLM will do what you ask. This is where the retry and critic loops earn their keep — we don't have to believe the LLM, we can check it!

Others have shown the harness matters independent of the model. On the full MLE-bench, the best configuration at launch — o1-preview inside the AIDE scaffold — medaled on 16.9% of competitions. Google's MLE-STAR held the model fixed and changed the method: retrieval to ground the initial approach, ablation studies to pick refinement targets, a leakage checker, an ensembling stage. With the identical Gemini-2.0-Flash, the medal rate on MLE-bench Lite went from AIDE's 25.8% to 43.9% (Table 1). The scaffold was worth more than a model-generation upgrade. (A stronger model inside the better scaffold reached 63.6%, but the same-model jump is the number that matters here.)

In my own runs the scaffold components that moved the score were mundane. A research step that writes a ranked menu of approaches before any experiments start, so a cheap model stops spending its first five iterations rediscovering TF-IDF. A rule that forces every third experiment to try a different model family. Archiving every experiment's predictions for ensembling later. None of this is clever, but it is the difference between the model that failed and the model that medaled.

With the fixed harness, it was time to run it again — on my DGX Spark this time, because I was more than a little sad about the $180. Around iteration 29 the agent discovered stacking: train a second-stage model on the first-stage models' predictions. Legitimate technique, very powerful and standard among Kaggle teams. But its implementation fit the meta-learner on the same 3,524 validation rows the selection score is computed from. The validation number collapsed from 0.23 to 0.14 while the true held-out score sat near 0.25. The agent was memorizing the answer key it was graded on. Doh!

This wasn't the model being stupid; it's Goodhart's law in action! A frontier model would have found the same exploit, probably faster. The vulnerability was mine: I gave the loop a target it could game with no backstop — and it gamed it. The run still produced a real result because the held-out test data never leaked into the workflow (it couldn't); the harness just believed the model was better than it actually was. MLE-STAR ships a leakage checker as a core module for the same reason. Run these loops long enough and you will meet this failure.

First, pretraining memory may have aided the model since this 2017 competition's solutions are public. While my harness blocks competition pages during web searches, absorbed training data remains a factor across all such evaluations. However, both of my runs had access to the same memory, yet only one medaled.

Second, cheaper models require more iterations due to errors. Mine ignored written prompt instructions until mechanically gated and relied on a structured research menu to avoid wasted initial moves. Stronger models waste fewer cycles, making model quality critical for single hard reasoning steps. Additionally, wall-clock time is significant: the winning run took four days on my DGX Spark.

The four-day run spanned 102 hours and 41 experiments. Training scaled from one minute for early scikit-learn models to over three hours for fine-tuned transformers (median 86 minutes; max 6 hours). Across roughly twenty model families—ranging from TF-IDF logistic regression, LightGBM, and SVMs to nine fine-tuned transformers and four stacking meta-learners—the total OpenRouter bill (billing export and experiment ledger available on request) came to:

The comparison isn't perfectly fair to the frontier model — it would cache prompts aggressively and would likely need fewer iterations. Grant it a three-fold iteration advantage and generous caching and it still lands well over a hundred dollars for the same climb.

Non-frontier models can be a very cost-effective way to achieve impressive performance on hard tasks. It may take more iterations than a frontier model would, but each iteration is relatively cheap. The harness matters a lot — I spent most of the effort on this project fixing workflow bugs that the agent was smart enough to exploit. My plan is to keep improving the harness and keep taking on hard problems. We are living in an incredible time where models can build models, and I plan to be deep in the weeds on it!

The AI Collective is built by volunteers across 180+ chapters in 40 countries.

Thank you to the thousands of volunteers around the world who make this work possible. We truly could not do this without you.