Part of my research on coordination without collapse: how do neural networks at different timescales share knowledge without fast learners overwhelming slow ones?
Collaborative Nested Learning
Continual Learning Through Multi-Timescale Optimization
Updated Nov 4, 2026Neural networks suffer from catastrophic forgetting—when trained on new tasks, they lose previously learned knowledge. Collaborative Nested Learning addresses this by maintaining separate optimization processes at different temporal scales, enabling rapid adaptation while preserving long-term knowledge.
+88.6% at reg=5 (single seed; accuracy up in 5/5 settings, forgetting down in 1/5)
Verified Nov 2, 2026 against commit 1362814 of the real simulation code, not a copy of this page's numbers. CI run · source · reproduce full scale in Colab
Each weekly run re-uses the same seed (42), so it shows this one run still reproduces — not that the effect holds across seeds.
Nested Learning as Meta-Architecture
Google's Nested Learning paper frames this as a model-internal paradigm. But the pattern is more general—it appears at every layer of the AI stack where components learn or adapt at different rates.
The Universal Challenge
Any system with components that learn or adapt at different rates faces the same fundamental challenges:
- How do fast and slow learners communicate without destroying each other's knowledge?
- How do you prevent fast learners from overwriting slow learners' consolidated patterns?
- How do you prevent slow learners from bottlenecking fast learners' adaptation?
Where the Pattern Appears
| Implementation Layer | Fast Component | Slow Component | Bridge Challenge |
|---|---|---|---|
| Native model | Inner optimization loops | Outer parameter consolidation | Continuum memory systems |
| Agentic orchestration | Task-specific specialists | Orchestrator / meta-learner | Specialists inform orchestration |
| RAG systems | Context window / attention | Vector store / corpus | Context consolidates to retrieval |
| Fine-tuning pipelines | Rapid adaptation layers | Frozen base model | Adapters inform base understanding |
| Human organizations | Frontline workers | Executive strategy | Operational signal reaches strategy |
Two Key Architectural Insights
Bidirectional Flow
Knowledge must flow both ways—not just top-down. Otherwise you get catastrophic forgetting (fast overwrites slow) or stagnation (slow ignores fast).
- Agentic: Specialists feed patterns back to orchestrators
- RAG: Generation informs retrieval ranking
- Orgs: Frontline insights reach strategy
Non-Adjacent Bridges
Critical signals shouldn't traverse every intermediate layer. Direct connections between distant timescales prevent information bottlenecks and fidelity loss.
- Agentic: Task execution → Orchestrator directly
- RAG: Working context → Corpus curation directly
- Orgs: Frontline → Executive (skip middle management)
At the model layer, we added 5 non-adjacent bridges (ultra-fast↔slow, fast↔ultra-slow, etc.) reducing max path length from 4 to 2 despite having 5 levels. The same architectural principle—skip connections for critical signals—applies at every layer of the stack.
Nested Learning in Agentic Systems
Key insight: Standard systems only have top-down flow. Adjacent bridges add bidirectional communication. Non-adjacent bridges let execution-level signals reach orchestration directly, bypassing intermediate layers for critical patterns.
Nested Learning in RAG Systems
Key insight: Standard RAG only has downward flow. Adjacent bridges add consolidation between layers. Non-adjacent bridges let working context directly inform corpus curation, bypassing the retrieval strategy layer for critical patterns.
What I Personally Built
Implementation timeline: Complete system designed and implemented in a single day, demonstrating rapid research-to-production capability when combining deep domain knowledge with production engineering discipline.
5-Level Multi-Timescale Optimizer
Extended Google's 3-level architecture to 5 levels with geometric 5× progression inspired by brain oscillation patterns. Implemented the complete optimizer hierarchy with proper gradient handling and update scheduling.
PyTorch · Optimizer Design · Multi-timescale
9 Bidirectional Knowledge Bridges
Designed and implemented the novel non-adjacent bridge architecture (5 non-adjacent + 4 adjacent). Built the gated adaptive transfer mechanism with gradient-surprise detection for selective knowledge consolidation.
Novel Architecture · Gated Transfer · LayerNorm
Experimental Framework & Evaluation
Built the complete experimental pipeline: regularization sweep experiments, Pareto frontier analysis, accuracy-retention trade-off visualization, and business use case mapping. Measured a +89% relative accuracy gain at regularization 5 (9.8% → 18.5%; single seed, best of five regularization settings).
Experiments · Visualization · Analysis
Research Codebase
Implemented with 109 passing tests (59% line coverage of src/), type hints, documentation, and a CI pipeline. Code structured for reproducibility and extension by other researchers.
59% Line Coverage · Type Hints · CI/CD
Why Continual Learning Matters
Neural networks suffer from a fundamental limitation: when trained on new tasks, they tend to forget previously learned information. This phenomenon, known as catastrophic forgetting, severely limits the practical deployment of deep learning systems in real-world scenarios where continuous adaptation is required.
Consider a clinical AI system that needs to learn new medical protocols while retaining knowledge of established ones. Or an autonomous vehicle that must adapt to new road conditions without forgetting how to handle familiar situations. Traditional neural networks fail at these tasks because gradient updates for new information overwrite the weights responsible for old knowledge.
Multi-timescale learning offers a promising solution by maintaining separate optimization processes that operate at different temporal scales—allowing the network to balance rapid adaptation with long-term knowledge retention.
Multi-Timescale Optimization with Knowledge Bridges
Google's NeurIPS 2025 paper introduced Nested Learning, a framework that maintains three optimizers operating at different timescales: fast (every step), medium (every 10 steps), and slow (every 100 steps). Each optimizer captures patterns at its characteristic temporal scale, with the slow optimizer preserving long-term knowledge while the fast optimizer handles immediate adaptation.
Extension 1: Five Optimization Timescales
We extend the architecture from 3 to 5 optimization levels with a geometric 5× progression that mirrors brain oscillation patterns:
| Level | Update Frequency | Brainwave Analog | What It Learns |
|---|---|---|---|
| Ultra-Fast | Every step | Gamma (~40 Hz) | Token-level patterns |
| Fast | Every 5 steps | Alpha (~8-13 Hz) | Local sequences |
| Medium | Every 25 steps | Theta (~4-7 Hz) | Contextual patterns |
| Slow | Every 125 steps | Delta (~0.5-4 Hz) | Task-level concepts |
| Ultra-Slow | Every 625 steps | Infraslow (<0.5 Hz) | Cross-task invariants |
Extension 2: Non-Adjacent Knowledge Bridges
The original nested learning transfers knowledge only between adjacent levels (fast→medium→slow). This creates information bottlenecks: knowledge must traverse every intermediate level, losing fidelity at each hop.
We add 5 non-adjacent bridges that enable direct cross-scale communication:
- Ultra-Fast ↔ Medium: Rapid pattern recognition can directly inform contextual learning
- Ultra-Fast ↔ Slow: Token patterns can consolidate directly to task-level memory
- Fast ↔ Slow: Sequence patterns bypass the medium timescale when appropriate
- Fast ↔ Ultra-Slow: Local patterns can inform cross-task invariants
- Medium ↔ Ultra-Slow: Contextual patterns connect to long-term memory
Total: 9 bidirectional bridges (4 adjacent + 5 non-adjacent) across 5 levels. The non-adjacent bridges are the key architectural contribution beyond Google's original approach.
Experimental Results
We evaluated bidirectional knowledge bridges at five regularization strengths, one run each (seed 42, Split-MNIST). Average accuracy was higher with bridges in 5 of 5 settings, with the largest gains where the baseline approach struggles most. Forgetting was lower with bridges in 1 of 5; accuracy and retention are reported separately below because they do not move together.
Bridges Never Hurt Accuracy, and Rescue It Where the Baseline Collapses
Largest relative accuracy gains at higher regularization strengths (single seed); hover for forgetting, which bridges lowered at only one setting
At low regularization (0.1-1.0), both approaches perform similarly—the baseline hasn't yet collapsed. But as regularization increases to prevent forgetting, the baseline accuracy drops to ~10% while bridges maintain 14-19% accuracy. The +89% relative gain at reg=5.0 (9.8% → 18.5%) and +62% at reg=20.0 are where bridges move accuracy most, but not where retention improves (see the next section). Both are single-seed results on a near-floor baseline, and +89% is the best point in the sweep rather than the typical one — see Limitations.
Accuracy vs. Retention (illustrative)
Illustrative: points were read off a figure, not serialised from a run. The shaded area marks where the with-bridges curve is above the baseline curve; it says nothing about whether bridges retain more.
The chart plots average accuracy against retention (1 − forgetting) as a trade-off curve. Its points were read off a figure rather than serialised from the run, so treat it as an illustration of the shape of the trade-off, not as a measurement. The run itself says something narrower than a frontier shift: bridges raised accuracy in 5 of 5 settings but lowered forgetting in only 1. At reg=5.0, the headline setting, forgetting was 94.0% with bridges against 85.6% without — so the +89% accuracy gain there came with less retention: a different operating point, not one that dominates the baseline.
Whether the non-adjacent bridges widen the achievable accuracy–retention frontier is not established by this run. A multi-seed sweep that measures retention directly would be the test.
Tunable for Different Business Requirements
Tune accuracy-retention trade-off for your use case
Different applications have different requirements. Trend forecasting prioritizes adaptability over retention—new patterns matter more than historical ones. Medical diagnosis requires maximum retention—you cannot forget established diagnostic criteria. Safety-critical systems need both high accuracy and high retention. The regularization parameter lets you tune the accuracy-retention trade-off for your specific use case.
Mathematical Formulation: Baseline vs. Novel Contributions
This section presents a side-by-side comparison of Google's baseline Nested Learning formulation and our novel extensions.
Architecture Comparison
Google: 3 Timescales
Three optimization levels with 10× progression between scales. Adjacent-only knowledge transfer.
Ours: 5 Timescales + Bridges
Five optimization levels with geometric 5× progression. 9 bidirectional bridges including non-adjacent.
Parameter Update Rule
Google Baseline
Transfer only from adjacent level to . Knowledge must traverse every intermediate level.
Our Extension
Transfer from all connected levels . Non-adjacent bridges enable direct cross-scale communication.
Bridge Connectivity
Google: Adjacent Only
Fast → Medium → Slow
Linear chain topology. Max path length = .
Ours: Adjacent + Non-Adjacent
UF↔F↔M↔S↔US + UF↔M, UF↔S, F↔S, F↔US, M↔US
Rich connectivity. Max path length = 2 (vs. 4).
Gated Adaptive Transfer (Novel)
Learned gating controls when transfer occurs. LayerNorm preserves timescale character.
Convergence Guarantee
This condition ensures knowledge transfer is a contraction mapping, preventing runaway amplification across timescales.
Summary: What's Novel
| Aspect | Google Baseline | Our Contribution |
|---|---|---|
| Timescales | 3 levels (10× progression) | 5 levels (5× progression) |
| Bridges | 2 (adjacent only) | 9 (4 adjacent + 5 non-adjacent) |
| Max Path Length | K-1 = 2 | 2 (despite K=5) |
| Transfer | Fixed linear | Gated adaptive |
| High-λ Accuracy | ~10% | ~19% (+89%) |
PyTorch Implementation
1class CollaborativeNestedOptimizer:
2 """5-level multi-timescale optimizer with bidirectional knowledge bridges.
3
4 Extends Google's 3-level nested learning with:
5 - 5 optimization timescales (geometric 5× progression)
6 - 9 bidirectional bridges including non-adjacent connections
7 - Brainwave-inspired frequency hierarchy
8 """
9
10 def __init__(self, params, bridge_config):
11 # 5 timescales with geometric 5× progression
12 self.ultra_fast = DeepMomentumOptimizer(params, update_freq=1) # Gamma (~40Hz)
13 self.fast = DeepMomentumOptimizer(params, update_freq=5) # Alpha (~8-13Hz)
14 self.medium = DeepMomentumOptimizer(params, update_freq=25) # Theta (~4-7Hz)
15 self.slow = DeepMomentumOptimizer(params, update_freq=125) # Delta (~0.5-4Hz)
16 self.ultra_slow = DeepMomentumOptimizer(params, update_freq=625) # Infraslow (<0.5Hz)
17
18 # 9 bridges: 4 adjacent + 5 non-adjacent (key contribution)
19 self.bridges = BridgeManager(bridge_config)
20 self.step_count = 0
21
22 def step(self, loss):
23 self.ultra_fast.step(loss)
24 self.step_count += 1
25
26 if self.step_count % 5 == 0:
27 self.fast.step(loss)
28 self.bridges.transfer_adjacent('ultra_fast', 'fast')
29 # Non-adjacent: ultra_fast can reach medium directly
30 self.bridges.transfer_non_adjacent('ultra_fast', 'medium')
31
32 if self.step_count % 25 == 0:
33 self.medium.step(loss)
34 self.bridges.transfer_adjacent('fast', 'medium')
35 # Non-adjacent bridges prevent information bottlenecks
36 self.bridges.transfer_non_adjacent('ultra_fast', 'slow')
37 self.bridges.transfer_non_adjacent('fast', 'slow')
38
39 if self.step_count % 125 == 0:
40 self.slow.step(loss)
41 self.bridges.transfer_adjacent('medium', 'slow')
42 self.bridges.transfer_non_adjacent('fast', 'ultra_slow')
43
44 if self.step_count % 625 == 0:
45 self.ultra_slow.step(loss)
46 self.bridges.transfer_adjacent('slow', 'ultra_slow')
47 self.bridges.transfer_non_adjacent('medium', 'ultra_slow')Limitations
The +89% is one of the few numbers on this site with a full-scale CI job standing behind it. That makes it worth being precise about what the job does and doesn't establish, because a green weekly badge can be read as more than it is.
One seed, no error bars
BridgeAblationConfig fixes seed = 42 and the benchmark runs each setting once. There are no repeats, no standard deviation and no confidence interval. The weekly job re-runs that same seed, so what it demonstrates is that the pipeline is deterministic and still reproduces — not that the effect is stable across initialisations. GCL's hold-up figure carries a bootstrapped 95% interval; this one carries nothing comparable and should not be read as though it does.
The gain is large because the baseline is near the floor
At reg=5.0 the no-bridges baseline averages 9.8% across the five Split-MNIST tasks — approximately chance for ten classes. Bridges lift that to 18.5%. The relative improvement is big because the denominator is small, and the best absolute accuracy anywhere in the sweep is 19.5%. These are research-scale continual-learning numbers demonstrating a mechanism, not deployable ones.
The headline is the best point in a five-point sweep
Across regularization strengths 0.1, 1, 5, 10 and 20 the gains are +0%, +3%, +89%, +39% and +62%. The chart above plots all five, which is why the sweep is on this page rather than the number alone — but +89% is the maximum, not the typical result, and at the weakest regularization bridges make no measurable difference at all.
Accuracy improves; forgetting mostly doesn't
The published run reports bridges helping accuracy in 5 of 5 settings but improving forgetting in only 1 — and that one is reg=0.1, where forgetting is 99% either way. At reg=5.0, where the accuracy gain is largest, bridges forget more (94.0% against 85.6%). Accuracy and retention are therefore separate results: the accuracy half is much better supported than the retention half, and nothing here supports a claim of improved retention or of dominating the baseline.
The Pareto chart's points were read off a figure
The regularization sweep (accuracy and forgetting) is pinned to benchmark output and held equal to the prose constants by headlineStats.test.ts, so the chart and the copy cannot drift apart. The Pareto chart has no such tie: its data is annotated in source as extracted from the plot rather than serialised from a run, and no test checks it. Its points are labelled illustrative on the page. Treat it as the shape of the trade-off, not as a second independent measurement.
Coverage is 59%, not 95%
Correction record: this page originally claimed 95% test coverage — including in the contribution card and badge near the top — and no measurement backed it. Running the suite the way CI does (pytest tests/ --cov=src, last measured 2026-11-04 at commit 98d6a9a) gives 109 passing tests (one skipped) and 59% line coverage of src/ — statement coverage, no branch measurement, and benchmarks/ is outside the measured scope — with src/optimizers/multi_scale.py at 0% — 157 statements never executed by a test. The figure has been corrected. This card previously argued that the headline result was nonetheless “better covered than the library around it,” on the grounds that the bridge ablation benchmark has its own smoke-test job in CI. That inference was wrong, and is retracted. The smoke-test job runs against a committed fixture and does not publish; the benchmark's own workflow states that a small-fixture run is not a reliable stand-in for the published claim, because the bridge effect is a scale-threshold phenomenon and is not there to detect below full scale. It shows the code path still executes, and nothing about whether the number holds. The headline figure is re-derived weekly, at full scale, by a separate job.
Split-MNIST only, and the named domains are illustrations
Validation is Split-MNIST-scale; larger-scale testing is future work rather than something already done. The tunability chart names trend forecasting, medical diagnosis and safety-critical systems — those illustrate what an accuracy-versus-retention knob would be for, not domains anything here was evaluated in.
Why the number still stands: unlike a figure quoted from a planning document, this one re-derives weekly from real MNIST on hardware nobody here controls, and the run that produced the currently displayed value is linked from the badge above. What it lacks is variance, not provenance. The honest summary is a reproducible single-seed result with a clearly identified mechanism — and the next thing it needs is a multi-seed run with intervals, not a bigger claim.