How LLM Agents Explore on Hour-Long MLE Tasks

13 min read

How an agent explores matters if we care about its strategy in a complex space over a long horizon, such as the strategies that lead to scientific breakthroughs. This behavior is usually studied in expensive settings, such as day-long runs on challenging research tasks, and we ask whether it can also be studied in a cheap one. We run three agents (GPT-6 Astra, GPT-6.1 Terra and Opus 5.5) on 39 tabular prediction tasks, each solved with scikit-learn on CPUs within 2 active hours and under 100 model submissions.

We find that the three agents explore in clearly different ways that the final score does not reveal. Astra roams widely and keeps returning to earlier ideas, Terra climbs from its current choice, and Opus settles exploration early. Compared with Bespoke Labs’ study of 24-hour ML research tasks, our agents change direction far more often, in a narrower space. However, what a change of direction leads to holds the same at both scales, and so do the traits that set the agents apart, such as Astra’s frequent returns and Opus’s low trial volume. Cheap tasks can thus already reveal many aspects of agent differences in exploration.

Every run can be browsed, submission by submission, in the interactive dashboard, which also has a map of all agents for each task.

Visualize exploration

Final outcome alone says very little about how an agent makes its decisions, especially on easier tasks where agents end close together. The curve of test loss over a run is more telling with signs of where progress speeds up or stalls. But it still compresses an attempt into one number, measured on a test set that is chosen somewhat arbitrarily, so it is noisy and leaves out lots of agent behaviors.

Figure 1. Trajectories from three agents on the task of predicting how many user reviews a Steam game will get. Each agent’s first 45 distinct model designs are shown in the order it tries them, and each design is placed on the map by embedding the configuration extracted from its fitted model object. Drag the slider or hover a panel to move through the run, and hover a dot on the map to see which model families that design fits (HGB: histogram gradient boosting, ET: extra trees; “own” is a custom model class the agent writes). The agents never see the test losses.

We focus on the model designs of agents to study their exploration behaviors. One unique benefit of MLE tasks compared to frontier research tasks is that here we have more control over the space the agent operates in, and each design can be represented by a relatively compact text description, which avoids the need for complicated approaches to map designs into a similarity space. By contrast, Fork and Flush summarize each submission’s code with a language model, ask a judge model which of two submissions is closer to a third, and fit a map to these triplet judgments.

We record every model when it is fit and saved, and match the saved model to the submitted file. From these fitted model objects we read the class of every estimator, the arguments it is constructed with, and how the pieces are composed:

{"definitions": {
  "m1": {"class": "sklearn.ensemble.ExtraTreesRegressor",
         "config": {"n_estimators": 180, "max_features": 0.8, …}},
  "m2": {"class": "sklearn.linear_model.Ridge",
         "config": {"alpha": 20, …}},
  "m3": {"class": "sklearn.ensemble.HistGradientBoostingRegressor",
         "config": {"learning_rate": 0.04, "max_iter": 350, …}},
  "m5": {"class": "modeling.Model",
         "config": {"kind": "blend_extra", "alpha": 100},
         "fitted_components": {"extra": "m1", "linear": "m2",
                               "model": "m3"}}},
 "roots": {"model.joblib": "m5"}}

This configuration captures most of the model design, except some of the code around it, such as data preprocessing.

A text embedding model (EmbeddingGemma 2) turns each configuration into a dense vector. Changing a small hyperparameter such as the learning rate moves a design only slightly in the embedding space, whereas replacing a blend of trees with a hand-written smoother moves it a lot.

For each task, we pool the designs of all three agents and jointly cluster them with affinity propagation (Frey and Dueck, 2007, as implemented in scikit-learn’s AffinityPropagation) on the cosine similarity of their embeddings. The method picks the number of clusters itself (15 on the median task), with its preference set to the median similarity among the pooled designs.

The clusters in the embedding space play a role similar to the idea basins of Fork and Flush, which are groups of submissions that share an approach. They differ, however, in granularity, since a basin is often a broad method, judged by a language model and shared by every model that runs the task. A cluster here is finer, comes from the fitted model’s configuration alone, and is discovered separately for each task, which makes sense given the narrower space of MLE tasks.

At any point of the trajectory, the next submission of an agent can either stay in the current cluster, open a new one, or return to one the run has visited earlier. Following Bespoke Labs, we refer to opening and returning as switches.

How agents explore

On the median task Astra Terra Opus
Steps spent in the most-used cluster 30% 50% 50%
Designs closest to the currently selected model 12% 29% 19%
Different models selected over the run 6 11 5

Table 1. Three traits of each agent’s style.

We find that agents are diverse in exploration behavior. Across the 39 tasks, each agent has a style of its own. Astra spends the smallest share of its steps in any one cluster. Terra climbs from its current choice, as more of its designs are variations of the model it has currently selected, and it replaces its selection most often. Opus quickly settles. Like Terra it spends half of its steps in one cluster, but it changes its selected model the least.

Figure 2. How many shared clusters a run reaches. For each task the designs of all three agents are clustered together, and each dot counts the clusters one agent reaches within its first K designs, K being the smallest design count among the three agents on that task. The black tick is the median task.

Astra searches wider. Within the same number of designs, Astra reaches more of a task’s shared clusters than Terra and Opus in most tasks, 6.5 on average against 4.6 and 4.8. Opus reaches slightly more than Terra, but the difference is within noise. Most shared clusters (84%) are reached by one agent only.

Figure 3. How new each design is. Each faint line is one trajectory. The solid line is the mean over tasks and the dashed line is the median task. We stop at design 30 because fewer tasks reach later designs.

Opus and Terra narrow down earlier than Astra. For each design, we compute its distance to the nearest design the agent has already evaluated. Opus opens widest and then drops almost as early as Terra. In contrast, Astra constantly produces new designs remain far from all previous ones until late in the run (Figure 3).

Figure 4. How often each agent changes cluster, split into steps that open a new cluster and steps that return to an earlier one. The rest of the steps stay in the current cluster.

They move between clusters differently. Astra returns to earlier clusters in 1/3 of its steps, compared with 18% for Terra and 21% for Opus, and Terra stays in its current cluster most often (Figure 4).

None of these differences is reflected in the final performance. For each pair of agents, the tasks split almost evenly on which one ends with the lower test loss, and most gaps are within the sampling noise of the test set, so even clearly different search strategies achieve similar results. This is one of the main limitations of studying exploration on simpler and somewhat synthetic tasks.

Comparing with 24-hour tasks

Bespoke Labs’ How models explore ideas measures the same metrics in a large-scale setting on ML research tasks, with Astra and Opus 5.5 among the models. They also follow runs through groups of similar submissions (i.e., idea basins), and investigate agent switches. As we are curious which exploration behaviors are shared across tasks of drastically different difficulty and horizon, we conduct the following comparison.

Bespoke Labs This post
Tasks ML R&D (AutoResearchExam) Tabular prediction (OpenMLE)
Run length 24 hours 2 active hours at most, about 1 at the median
Runs 348, from 12 agents 117, from 3 agents
Steps analyzed 9,179 submissions with new code 6,799 submissions with a new design
What a submission is compared by an LLM’s summary of its code, judged in triplets by a second LLM the fitted model’s configuration, embedded as text
Groups of similar submissions Coarser, 3 to 8 basins per task, shared by all models More fine-grained, 15 clusters per task at the median, shared by the three agents

Table 2. Comparing the two settings on agent exploration.

Our groups are more fine-grained than theirs because a model configuration records every estimator and its settings, and the affinity propagation clustering keeps as many clusters as the embedding entails, whereas their basins are broad methods, manually limited to 3 to 8 per task. The research space is also larger and more open-ended, and even a single direction can take extensive investigation, which makes it hard for agents to explore broadly even within 24 hours. In the MLE space, by contrast, agents readily move on to the next estimator family after a few trials.

24-hour research tasks (Bespoke Labs) 2-hour tabular tasks (Ours)
How often agents change direction
Runs that never switch 47% 0% (0% by estimator family)
Groups visited per run 1.5 to 2 6.1 (10.5 estimator families)
Share of steps that switch 4% to 13% 33% (39% by estimator family)
What a change of direction does
Switches that return to an earlier group about half 73%
New best, new group compared with staying about 2 times as likely 2.6 times as likely
New best, return compared with staying about the same 0.83 times as likely
When switching pays off first quarter of the run (6 of 24 hours) first quarter of the run (by step count)
What distinguishes agents
GPT-6 Astra switches the most, about two thirds are returns switches the most, 75% are returns
Opus 5.5 among the lowest-volume models fewest designs (45 per run, against 70 and 60)

Table 3. What both studies measure.

As Table 3 shows, how often an agent changes direction does not carry over. What a change of direction does generalizes, and so do the agents’ characteristics.

Shared at both scales

Small tasks allow more extensive exploration within a shorter horizon, and our 3 × 39 runs already yield 2,218 switches, enough to measure step-level effects.

Most switches are returns. Bespoke report that half of all switches return to an idea the run has already tried, compared with 73% in our runs.

New best scores come from new directions, not from returns. They find that a switch to a new idea is about twice as likely to set a new best score as a refinement step, while a return gains about as much as staying. On our tasks a step that opens a new cluster sets a new best test loss 24% of the time, a step that stays 9%, a return 8%, and for every agent opening a new cluster is the step most likely to set a new best (Figure 5). This does not contradict the earlier finding that exploration style is not reflected in final performance, since all agents eventually saturate the MLE space.

Figure 5. How often a step sets a new best test loss for its run, by what the step does. All steps of 39 runs per agent; click a name to hide that agent.

The gain from a new direction comes early. They find that switches gain about 3.5 times as much as refinement in the first 6 (out of 24) hours and no more than refinement afterwards. Splitting runs by step count instead of wall-clock time, we find that in the first quarter of a run a step into a new cluster improves the normalized test loss about 6 times as much as a step that stays, whereas in the rest of the run the difference is indistinguishable.

Figure 6. How often a step sets a new best test loss, by where in the run it happens. Each run’s steps are cut into fifths by step count. Hover for the number of steps behind each point, which falls to 27 for new clusters in the last fifth.

Figure 6 shows that new best scores (reasonably) become rare for every kind of step as a run goes on. Opening a new cluster keeps a modest edge, about 1.5 times as likely as staying in the first quarter, though the edge vanishes in the middle fifth. The whole-run ratio of 2.6 is larger because new clusters are opened mostly early, when new best scores are easy to achieve. The last fifth shows the edge again with a small sample size. After the first quarter, a new direction still produces the occasional record, but on average it does not improve the loss more than staying.

Astra’s explorative signature. Bespoke describe GPT-6 Astra as one of the models that switch the most, with about two thirds of its switches returning to an earlier idea. We also find that Astra switches the most, in 43% of its steps against 25% for Terra and 30% for Opus, and three quarters of its switches are returns (Figure 4).

Opus submits less. Opus 5.5 is among the low-volume models on AutoResearchExam, and on ours it similarly makes the fewest distinct designs (45 per run against 70 for Astra and 60 for Terra), fewer than Astra in most tasks. On some tasks, however, Opus 5.5 is more suspicious of the test environment and refuses to continue without further clarification, which our continuous runner does not provide.

Expected differences

How often an agent leaves. Our agents change direction far more often than theirs (Table 3 and Figure 7). Part of this difference comes from how groups are defined from clustering, but the rates stay far apart even with estimator families (such as ridge, boosted trees, or blends of both), so the difference is significant. One possible reason might be the cost of a switch. Replacing ridge regression with boosted trees requires changing only one line of code, whereas in their tasks replacing a BLAS autotuner with a JIT compiler requires a complete rewrite.

Figure 7. Share of steps that change cluster. Each dot is one task and the black tick is the median task.

Connecting exploration with test-set performance. They find that the top-scoring models send more of their switches to new basins (ρ = 0.70 across 12 models). We cannot test this, since our agents finish within test-set noise of each other on most tasks, and with only three agents a correlation across models is not meaningful.

Differences worth a closer look

Is switching consistent over time? On AutoResearchExam, the number of switches barely changes with the number of submissions, so models that submit more have lower switch rates. In our data, the number of switches grows with the number of designs (ρ = 0.57 between runs of the same task). Their agents seem to have a fixed budget of direction changes per run, whereas ours switch at a steady rate per step. The cost of a switch could potentially explain this too.

Does a late switch help? They find no link between switching in the second half of a run and late gains. In our setup, agents still switch often in the second half, and they continue to set new bests there. This might suggest that agents keep more control over exploration in a smaller space and lose direction on harder tasks, which deserves a deeper look. Another confounder is the small number of late switches in their study.

Did the run reach the best region? They report that 58% of runs never reach the basin of their task’s best solution. Most of the clusters in our setup are reached by one agent only, because our representation often places each agent’s submissions in a region of their own, partly for reasons of agent-specific coding style. A shared map on methodological semantics is the main thing in which their LLM-judged similarity outperforms our embedding-based clustering approach.

Takeaway

A 2-hour tabular task is not a small copy of a 24-hour research task. But once an agent does change direction, what follows looks the same at both scales. Returns are common and gain little, a new direction is where new best scores come from, and it stops improving the loss on average after the early part of a run. The trait that sets Astra apart also shows up in both.

Understanding how these findings generalize is crucial for scaling the evaluation of exploration to long-horizon agentic tasks, which needs both broad coverage and scalable oversight over ever longer horizons. There might also be a useful division of labor. Questions about what a step does, and about how agents differ in it, can be studied in low-cost settings, whereas questions about how capably agents find a useful new direction require tasks where a new direction is hard to find.

Citation

If you found this useful, please cite it as:

@misc{chen2026explore,
  author = {Chen, Sanxing},
  title  = {How LLM Agents Explore on Hour-Long MLE Tasks},
  year   = {2026},
  month  = nov,
  url    = {https://sanxing.ai/mle-explore-small-scale/},
  note   = {Blog post}
}