4 Million Views on the Hierarchical Reasoning Model: Why Didn’t the ‘Hierarchical Architecture’ Work? Performance Improvements Have Other Secrets?

Excerpt from ARC PRIZE

Author:ARC PRIZE TEAM

Translated by Machine Heart

Do you remember the Hierarchical Reasoning Model (HRM)?

This work was published in June and caused quite a stir—discussions related to it on X/Twitter garnered over 4 million views and tens of thousands of likes, and YouTube videos analyzing this work exceeded 475,000 views.

In the paper, the authors state that they proposed a brain-inspired HRM architecture, which achieved a score of 41% on the ARC-AGI-1 benchmark with only 1,000 training tasks and a relatively small model of 27 million parameters.

The paper also points out that this architecture has the potential to be a transformative advancement towards general computation and general reasoning systems.

Given the popularity and novelty of the method, researchers from the ARC PRIZE team set out to validate the performance of HRM on the ARC-AGI-1 semi-private dataset (a hidden holdout task set used to verify whether solutions overfit).

Their findings were somewhat surprising.

First: they were able to closely replicate the numbers claimed in the paper. HRM’s performance on the ARC-AGI semi-private set was impressive for its model size:

• ARC-AGI-1：32%—although not at SOTA level, this is already outstanding for such a small model.

• ARC-AGI-2：2%—while the score being greater than 0% shows some positive signals, the ARC PRIZE team does not consider this a substantial advancement on ARC-AGI-2.

Meanwhile, by running a series of ablation analyses, they obtained some surprising findings that challenge the mainstream narrative surrounding HRM:

1. Compared to a similarly sized Transformer, the impact of its “hierarchical” architecture on performance is minimal.

2. However, the relatively under-discussed “outer loop” optimization process, especially during training, significantly enhanced performance.

3. The benefits of cross-task transfer learning are limited; most performance comes from memory of the specific task solutions used during evaluation.

4. Pre-training task augmentation is crucial, although only 300 augmentations are sufficient (rather than the 1,000 reported in the paper). The impact of task augmentation during reasoning is limited.

Findings 2 and 3 indicate that the method in the paper is fundamentally similar to the “no pre-training ARC-AGI” method proposed by Liao and Gu.

Some believe that these analyses from the ARC PRIZE team suggest that HRM is a failed exploration.

However, many others counter that compared to models performing similarly on the same benchmark, HRM is much smaller. While the role of the hierarchical architecture has not been validated, the innovations in other aspects of the paper are still worth studying, as the model’s performance is still quite good.

In response to this analysis, the ARC PRIZE team specifically wrote a blog to elaborate. Here is the content of the blog.

• Blog address:https://arcprize.org/blog/hrm-analysis

• Github:https://github.com/arcprize/hierarchical-reasoning-model-analysis

Review of the “Hierarchical Reasoning Model”

Figure 1: The HRM method draws inspiration from the processing of signals in the brain at different frequencies.

The Hierarchical Reasoning Model was published by the Singapore AI research lab Sapient, claiming its inspiration comes from the brain’s hierarchical and multi-time-scale processing mechanisms.

HRM is a 27 million parameter model that iteratively optimizes through several brief “thinking” pulses.

Each pulse produces:

• A prediction output grid— this is the model’s “in-process” prediction for the ARC-AGI task.

• A “stop or continue” score—this score determines whether to continue optimizing the prediction or submit it as the final result.

If the model chooses to continue optimizing, this “in-process” prediction goes through another “thinking” pulse. This is the outer loop: predict, ask “Am I done?”, and then stop or continue optimizing.

Within the outer loop, HRM runs two coupled loop modules: “H” (slow planner) and “L” (fast executor). These two modules work together to update a shared hidden state, rather than producing independent outputs. The final effect is that the model alternates between “planning” (H) and “details” (L) until the internal state “self-consents” and produces an answer.

The model uses a learned “stop” signal, which is an adaptive computation mechanism to control the number of optimizations.

A key part of this process is task augmentation. This is a process of applying transformations (such as object rotation, flipping, etc.) to each task, aimed at uncovering the underlying rules of the task rather than overfitting to specific shapes or colors.

During testing, the model runs the same augmentations to generate predictions. These predictions are then “de-augmented” (to restore to the original task format), and the final prediction result is decided through simple majority voting.

Predictions are made through transduction (the direct output of deep learning) in the embedding space, rather than through induction (generating a program that can apply transformations). For more information on transduction and induction in ARC-AGI, see the 2024 ARC Prize-winning paper “Combining Induction and Transduction for Abstract Reasoning” by Wen-Ding Li et al.

ARC-AGI Validation Process

The ARC-AGI benchmark has 3 main datasets for testing:

• Public Training Set – Public data used to introduce the ARC-AGI data format. Researchers train and iterate models on this data.

• Public Evaluation Set – Public data for researchers to self-evaluate model performance after training.

• Semi-Private Evaluation Set – A holdout dataset used to validate claims on ARC-AGI. This dataset cannot be accessed online for training, thus increasing its credibility in providing a pure signal of model performance. It is referred to as “semi-private” because validating third-party services (such as models from OpenAI, xAI) means we cannot guarantee the data will always remain completely confidential, and plans will eventually replace it.

In addition to these types of datasets, ARC-AGI currently has 2 released versions:

• ARC-AGI-1— 2019, designed to challenge deep learning systems.

• ARC-AGI-2— 2025, designed to challenge reasoning systems.

We test specific custom solutions like HRM based on testing policies. To qualify for validation, solutions must be open-source, have a running cost of less than $10,000, and complete within 12 hours.

Officially Validated HRM ARC-AGI Scores

Figure 2: ARC-AGI-1 leaderboard, HRM performance versus cost per task.

ARC-AGI-1 (100 tasks)

• Score: 32%,Runtime: 9 hours 16 minutes,Total Cost: $148.50 ($1.48 / task)

Achieving a score of 32% on ARC-AGI-1 is impressive for such a small model. The slight drop from the publicly claimed evaluation set score of 41% to the semi-private set score was expected. The public and semi-private sets of ARC-AGI-1 were not difficulty-calibrated. The observed 9 percentage point drop is on the higher end of normal fluctuations. If the model overfits the public set, its performance on the semi-private set could collapse (e.g., dropping to 10% or lower). However, we did not observe this. This result indicates that HRM indeed has some noteworthy highlights.

Note: The cost of running HRM is relatively high because training and inference are coupled in a single run. The paper’s authors mentioned they are working to decouple this process to submit their solution to the ARC Prize 2025 Kaggle competition.

ARC-AGI-2 (120 tasks)

• Score: 2%,Runtime: 12 hours 35 minutes,Total Cost: $201 ($1.68 / task)

ARC-AGI-2 is significantly more difficult than ARC-AGI-1, so the substantial drop in performance was expected. Unlike ARC-AGI-1, the public and semi-private sets of ARC-AGI-2 are difficulty-calibrated. In principle, scores on both should be similar. Although a score greater than 0% shows some capabilities of the model, we do not consider the 2% score to be a meaningful advancement on ARC-AGI-2.

Note: We chose to include 10 optional checkpoints (each about 5 minutes), adding approximately 50 minutes in total. Although HRM’s submission exceeded the 12-hour runtime limit, we still consider it valid.

Analyzing HRM’s Contribution to ARC Scores

In a deeper analysis, the question we most want to answer is: “What are the key components in the HRM architecture that contribute most to success on ARC-AGI?”

We carefully examined the 4 main components of the HRM paper: HRM model architecture, H-L hierarchical computation, outer optimization loop, and the use of data augmentation. Researcher Konstantin Schürholt led this analysis.

We tested:

• The performance contribution of the “hierarchical” H and L loops

How much performance improvement does HRM provide compared to a baseline Transformer?

What is the impact of changing the parameters of the hierarchical computation?

• Changing the maximum number of “stop or continue” loops

How does the adaptive computation time (ACT) scorer perform compared to a fixed number of loops (without stop decisions)?

• The impact of cross-task transfer learning

What is the impact of including training set tasks and ConceptARC tasks during training compared to training only on the evaluation tasks?

• The number of augmentations

Changing the number of augmentations created from each task.

• Model/training variants (size and duration)

Finding 1:

Compared to a similarly sized Transformer,

the impact of the “hierarchical” architecture on performance is minimal

The paper proposes that the HRM architecture is key to achieving hierarchical reasoning—combining slow-paced guidance (H-level) and fast-paced thinking (L-level).

To understand the impact of this architecture, we conducted 2 experiments:

• Changing the number of iterations in the hierarchical components.

• Replacing the HRM model with a similarly sized Transformer.

For comparison, the Transformer had the same number of parameters as the HRM model (approximately 27 million). In all experiments, we kept all other components of the HRM process unchanged.

Comparing HRM with a standard Transformer yielded two interesting results, as shown in Figure 3. First, a standard Transformer, without any hyperparameter optimization, performed about 5 percentage points worse than the HRM model. When there was only one outer loop, the gap was minimal, with both models performing similarly.

Figure 3: HRM model and similarly sized Transformer performance at pass@2 under different outer loop steps. The performance difference between the Transformer and HRM is a few percentage points without any hyperparameter optimization.

When the number of outer loops exceeds 1, HRM performs better, but the gap narrows as the number of outer loops increases. Note that despite matching parameter counts, HRM used more computational resources, which may partially explain the performance differences. The benefits of increased computational resources may exhibit diminishing returns as the number of outer loops increases, which aligns with our results.

We further varied the number of steps for H-level and L-level to analyze their impact. We found that increasing or decreasing the number of iterations (baseline being L = 2, H = 2) led to performance degradation.

These results suggest that the performance on ARC-AGI is not fundamentally due to the HRM architecture itself. While it does provide some benefits, substituting a baseline Transformer in the HRM training process can achieve comparable performance.

Finding 2:

The relatively under-discussed “outer loop” optimization process

led to significant performance improvements

In addition to the hierarchical architecture, the HRM paper also proposed using an outer loop (“loop connection”) externally to the model. This feeds the model’s output back to itself, allowing the model to iteratively optimize its predictions.

Additionally, it uses “adaptive computation time” (ACT) to control the number of iterations spent on specific tasks. ACT determines whether to stop predicting or continue optimizing.

This part of the HRM method is similar to the Universal Transformer, which has both a loop optimization cycle around the Transformer model and a version of ACT.

In our second set of experiments, we wanted to understand the impact of the outer optimization loop and ACT on overall performance. We varied the maximum number of outer loops during training and used the maximum loop count during inference (following HRM’s implementation).

Figure 4: pass@2 performance under different numbers of training and inference optimization loops. Iterative optimization of data has a powerful impact, as shown by the leap in performance from 1 loop (no optimization) to 2 loops (1 optimization).

As shown in Figure 4, the number of outer loops has a significant impact on model performance—from no optimization (1 loop) to just 1 optimization, performance jumped by 13 percentage points. Increasing from 1 optimization loop to 8 doubled the performance on the public evaluation set.

A minor finding is that using ACT during training does indeed reduce the actual number of optimization steps per task. However, while using ACT can improve performance, the difference is only a few percentage points compared to a fixed 16 loop run.

The results indicate that optimizing the outer loop is a key driver of HRM performance.

To understand the impact of optimization during training versus inference, we further independently varied the number of inference optimization loops.

Figure 5: pass@2 performance under different numbers of inference optimization loops. The blue bars represent models using the same number of loops for training and inference. The orange bars represent models trained with 16 optimization loops but evaluated with different numbers of optimization steps. For example, the blue bar at 1-Loop indicates a model trained with one optimization step and evaluated with one optimization step. The orange bar indicates a model trained with 16 optimization steps but evaluated with only one optimization step.

To this end, we compared two setups. In the first, we used the same number of optimization loops for training and inference. In the second, we trained a model with up to 16 optimization loops and evaluated it with 1, 4, 8, and 16 optimization steps respectively.

Comparing these two types of models showed substantial differences, especially at low inference optimization steps (1 and 4), as shown in Figure 5. Training with more optimization steps can improve the performance of single optimization loop predictions by over 15 percentage points, even though a single loop means no optimization during a single forward pass. Increasing the number of optimization loops during inference has a lesser impact. This indicates that training with optimization is more important than using optimization during inference.

Finding 3:

The impact of cross-task transfer learning is limited

In the original paper, HRM was trained on an augmented version of 960 task pairs:

• 400 tasks from the ARC-AGI-1 training set.

• 160 tasks from the closely related ConceptARC benchmark.

• 400 tasks from the ARC-AGI-1 evaluation set, which are also the targets to be solved during evaluation.

It is important to note that this does not imply data leakage, as the model never saw the test pairs of evaluation tasks during training—which is precisely what the model is evaluated on.

We wanted to understand how cross-task transfer learning compares to remembering specific task solutions seen during evaluation. We conducted the following experiment: we trained only on the 400 evaluation tasks—removing the 400 tasks from the training set and the 160 tasks from the ConceptARC dataset.

In this setup, the model achieved a pass@2 accuracy of 31%, with only a slight decrease compared to the initial 41%. This indicates that the vast majority of performance is driven by training on tasks seen during evaluation. This means that HRM’s approach is essentially a zero pre-training test-time training method, similar to Liao and Gu’s “no pre-training ARC-AGI”. This is akin to using the model as a program synthesis substrate—using gradient descent on the task pairs to encode the program that executes the task into the model’s weights.

Notably, there still exists some degree of cross-task transfer learning in our experiments—between different tasks in the evaluation set. A stronger version of our experiment would be to run the HRM process independently on each evaluation task, so the model would only see the augmented version of the demonstration pair for the specific task it will be evaluated on. This would align HRM’s setup with Liao and Gu’s setup completely. We did not have time to run this experiment, but we speculate that the results would be very close to Liao and Gu’s (21% pass@2).

Finding 4:

Pre-training task augmentation is crucial

Another important component of the HRM process we studied is task augmentation.

Data augmentation is a common method in deep learning used to increase the number of samples in a dataset and improve the model’s generalization ability. This means applying transformations to tasks such as rotation, flipping, or color swapping to generate new data without altering the fundamental concepts of the task.

HRM predicts on all augmented versions of the tasks, then restores (or “de-augments”) the augmented predictions to return to the original task format. The model then conducts a majority vote on these predictions to select the final candidates.

We tested the following modifications to the baseline HRM:

• The maximum number of augmentations during dataset compilation.

• The maximum number of predictions used for majority voting.

Since HRM can only handle the types of augmentations encountered during training, we limited the second point during inference to reducing the number of augmentations, not increasing it.

The results in Figure 7 show two trends. First, using data augmentation does significantly improve performance. However, using only 300 augmentations, rather than the 1,000 used in the paper, is close to achieving maximum performance. Using only 30 augmentations (3% of the total used in the paper) results in performance that is less than 4% away from maximum performance.

Secondly, using data augmentation during training seems to be more important than having a larger pool for majority voting. Models trained with more augmentations show much less performance drop when using a smaller pool during inference.

Other Technical Learnings

Diving deeper into the internal mechanisms of HRM also revealed some other interesting findings.

First and foremost, HRM decomposes ARC-AGI tasks into individual input-output pairs, which they refer to as puzzles. Each puzzle receives a puzzle_id, which consists of a task hash value and the augmentation code applied to that specific puzzle.

During training and inference, the model only receives the input and puzzle_id—without any few-shot context containing other input-output examples from the task. The HRM model must learn to associate a puzzle_id with a specific transformation so that it can predict the output from the input.

To achieve this, it inputs the puzzle_id into a large embedding layer. This embedding layer is key—without it, the model would not know how to process the input. This brings a major limitation: the model can only apply to puzzle_ids it has seen during training.

In discussions with the authors on this topic, they explained that changing puzzle embeddings for few-shot contexts is a complex engineering challenge. In earlier versions, they compared and found that on larger class ARC datasets, few-shot contexts performed well, but on the sample-limited ARC, puzzle embeddings performed significantly better. We did not replicate these experiments, but this points to interesting directions for future work.

For the same reasons, in this version of HRM, inference data must be part of the training dataset. For example, independently changing the number of augmentations is not a straightforward task.

Finally, while the optimization loop has a clear impact on performance, HRM is purely transductive. While optimization steps can be unfolded, the underlying program remains implicit. Our hypothesis is that this approach lacks generalization capability.

Unresolved Issues and Future Work

Our experiments with HRM revealed the reasons for its outstanding performance on ARC-AGI, but other questions remain. In addition to those mentioned above, we invite the community to explore the following additional questions:

• How much does the puzzle_id embedding affect model performance? How does it compare to providing the remaining examples in the task as context?

• How does HRM’s generalization capability outside its training data fare? Are there any methods to fine-tune HRM on new data?

• What impact does the learned stopping mechanism have during inference? Are its benefits limited to saving computational resources, or can it also improve performance?

• Can the idea of optimization be generalized to other methods, such as inductive methods for synthesizing explicit programs?

• What performance can be achieved by training and evaluating independently on each evaluation task? (Eliminating all cross-task transfer learning.)

• What specific types of augmentations lead to higher performance? Rotation? Color swapping? Why?

Conclusion

The analysis from ARC Prize is very thorough, and many have remarked that this is true “peer review”.

Others have said that this analysis is as important as the paper itself. If more analyses like this can be conducted on highly regarded research in the future, the community can gain new knowledge more efficiently.

© THE END

For reprints, please contact this public account for authorization

Submissions or inquiries: [email protected]