ComposeCL

Continual Learning Mechanisms Compose for Long-Horizon Memorization

Zheyuan Zhang* Alvin Zhang* Daniel Khashabi† Tianmin Shu†

Johns Hopkins University

* Equal contribution † Equal advising

Paper Code Explore the results

Complementary mechanisms work better together.
We test their compositions over 100 sequential tasks.

34.9%Average final retentionUp from 1.2% with naive fine-tuning
Top 3On all three datasetsOne fixed composition of all mechanisms
100Tasks per datasetThree datasets, three seeds

Final retention is accuracy on all learned tasks after task 100. Rankings compare the 16 factorial combinations.

A systematic test of composition

Four mechanisms, sixteen combinations.

We turn each of the three anchors and merged LoRA on or off in a full 24 factorial experiment. Every combination is evaluated for 100 tasks on each dataset with three seeds.

Final retention for all sixteen combinations. The best means are 23.2% on Symbol-QA, 41.8% on LLM-QA, and 54.8% on Real-QA. Filled markers identify active mechanisms.
All 16 combinations, averaged over three seeds. Filled markers identify active mechanisms. The strongest compositions substantially outperform standalone mechanisms.
What the factorial reveals

Replay and merged LoRA provide the largest average gains. Their joint benefit exceeds the sum of their individual gains on all three datasets. Combining all three anchors with merged LoRA is the only composition among the top 3 on every dataset.

The central finding

Compositions of multiple continual learning mechanisms make memory last longer.

Memory accuracy declines much more slowly for compositions than for naive fine-tuning or the best single mechanism. The all-mechanism composition has memory half-lives of 19, 32, and 32 tasks on Symbol-QA, LLM-QA, and Real-QA.
Memory accuracy after learning additional tasks. Lines show means and bands span the minimum and maximum over three seeds. Half-life is the number of later tasks before accuracy falls below half its initial value. Compositions substantially extend memory lifetime.

Abstract

Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-level successive halving to search the combinatorial design space and use a factorial experiment to measure individual and interaction effects. Our best method combines all three anchors with merged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.

Two design dimensions

What to preserve. Where to retain updates.

Anchors constrain model changes while learning new tasks. Low-rank allocation determines which parameters are updated and what is retained for later tasks.

Data, function, and weight anchors jointly constrain learning. Shared LoRA reuses the same low-rank factors, while merged LoRA merges each learned update into the model and initializes fresh factors for the next task.
The three anchors act together, not sequentially. The weights before and after each task are successive versions of the same model.
01 / Data anchor

Revisit generated examples

Generate replay from the previous model and use its soft targets to retain earlier information without keeping past training data.

02 / Function anchor

Keep outputs consistent

Use self-distillation to match the previous model’s output distribution on current-task inputs.

03 / Weight anchor

Protect important parameters

Limit changes to parameters identified as important for earlier tasks, using Synaptic Intelligence or EWC.

04 / Low-rank allocation

Make room for the next update

Merged LoRA incorporates each task’s update into the model and initializes fresh low-rank factors for the next task.

Our primary compositions maintain a constant retained state size with respect to the number of tasks.

Evaluating long-horizon memory

Three datasets. One hundred tasks each.

A model learns tasks one at a time, without retaining earlier training examples or receiving task identifiers at inference. We measure recall of the queries it has learned.

Random associations

Symbol-QA

Randomly generated key-value associations test memorization without semantic clues.

10,000query-answer pairs

Fictional knowledge

LLM-QA

LLM-generated queries and answers span 100 fictional topics, reducing reliance on pretrained knowledge.

10,000query-answer pairs

Natural questions

Real-QA

Natural queries from 10 public QA datasets are filtered to exclude those answered correctly by the base model.

5,000query-answer pairs

Each query has exactly one target answer in the two synthetically generated datasets.

Citation

If you use this work, please cite:

@article{zhang2026continual,
  title  = {Continual Learning Mechanisms Compose for Long-Horizon Memorization},
  author = {Zhang, Zheyuan and Zhang, Alvin and Khashabi, Daniel and Shu, Tianmin},
  year   = {2026}
}