Research infrastructure for systematic strategy evaluation. This is a backtesting and analysis framework—not a trading system, not live performance, not financial advice.
Strategy Tournament
Research-Grade Backtesting & Evaluation Framework
Updated Jan 2026A tournament-style evaluation pipeline for comparing systematic strategies under realistic execution constraints and adversarial robustness tests. Designed to surface overfitting, regime sensitivity, and statistical artifacts before they become expensive lessons.
Why This Exists
The problem: Most backtesting frameworks make it easy to fool yourself. They optimize for convenience, not correctness. The result is strategies that look great in-sample and fail out-of-sample—or worse, strategies that "work" due to subtle lookahead bias, survivorship bias, or data snooping.
What This Is NOT
- ✗A trading system or execution platform
- ✗Live trading performance or results
- ✗Financial advice or recommendations
- ✗A "get rich quick" tool
What This IS
- ✓Research infrastructure for strategy analysis
- ✓Adversarial testing framework
- ✓Statistical validation pipeline
- ✓A tool for learning and research
Core Philosophy
The framework is built on three principles that prioritize correctness over convenience:
1. Adversarial by Default
Every strategy faces stress tests designed to break it. Regime shifts, parameter perturbations, execution degradation. If a strategy only works under ideal conditions, the framework will expose that. Survival is the first filter.
2. Statistical Rigor
No result is reported without proper statistical validation. Multiple hypothesis correction, bootstrap confidence intervals, effect size requirements. p-hacking is a bug, not a feature.
3. Realistic Execution
Backtests include market impact, latency, partial fills, and realistic commission structures. The gap between backtest and reality is where most strategies die. We simulate that gap explicitly.
Tournament Architecture
Strategies compete in a structured evaluation pipeline. Each phase is designed to eliminate a specific class of false positives.
Phase 1: Baseline
Performance under realistic execution constraints. Slippage, latency, partial fills. This is the "best case" that still accounts for market friction.
Phase 2: Robustness
Stress tests across regime shifts, parameter sensitivity, and bias detection. Strategies must demonstrate stability, not just performance.
Phase 3: Validation
Statistical significance with multiple testing correction. Bootstrap confidence intervals. Effect size thresholds. No p-hacking allowed.
1# Strategy Tournament Configuration
2# Research infrastructure for systematic strategy evaluation
3
4tournament:
5 name: "Q1-2026-Momentum-Variants"
6 description: "Comparing momentum strategies under adversarial conditions"
7
8 # Evaluation regime - NOT live trading
9 mode: research
10
11 execution_constraints:
12 slippage_model: "volume_impact" # Realistic market impact
13 latency_ms: [10, 50, 100, 500] # Test across latency regimes
14 fill_probability: 0.85 # Partial fills
15 commission_bps: 2.5
16
17 robustness_tests:
18 - regime_shift # Performance across market regimes
19 - parameter_sensitivity # Stability under parameter perturbation
20 - data_snooping # Multiple hypothesis correction
21 - lookahead_bias # Strict temporal validation
22 - survivorship_bias # Include delisted instruments
23
24 statistical_validation:
25 bootstrap_samples: 10000
26 confidence_level: 0.95
27 multiple_testing: "bonferroni"
28 effect_size_threshold: 0.3 # Minimum Cohen's dEvaluation Pipeline
The core evaluation loop processes each strategy through the full pipeline, generating comprehensive reports that highlight both strengths and vulnerabilities.
1class StrategyTournament:
2 """Research infrastructure for systematic strategy evaluation.
3
4 Key principle: Strategies compete under adversarial conditions.
5 No strategy advances without surviving robustness stress tests.
6 """
7
8 def __init__(self, config: TournamentConfig):
9 self.config = config
10 self.execution_simulator = ExecutionSimulator(
11 slippage_model=config.slippage_model,
12 latency_distribution=config.latency_ms,
13 fill_model=config.fill_probability
14 )
15 self.robustness_suite = RobustnessTestSuite(config.robustness_tests)
16 self.statistical_validator = StatisticalValidator(
17 bootstrap_n=config.bootstrap_samples,
18 alpha=1 - config.confidence_level,
19 correction=config.multiple_testing
20 )
21
22 def evaluate_strategy(self, strategy: Strategy, data: MarketData) -> EvaluationReport:
23 """Full evaluation pipeline with adversarial testing."""
24
25 # Phase 1: Baseline performance under realistic execution
26 baseline = self._run_backtest(strategy, data,
27 execution=self.execution_simulator)
28
29 # Phase 2: Robustness stress tests
30 robustness_results = {}
31 for test in self.robustness_suite:
32 robustness_results[test.name] = test.evaluate(strategy, data)
33
34 # Phase 3: Statistical validation
35 validation = self.statistical_validator.validate(
36 baseline_results=baseline,
37 robustness_results=robustness_results,
38 null_hypothesis="strategy_has_no_edge"
39 )
40
41 return EvaluationReport(
42 strategy=strategy.name,
43 baseline_metrics=baseline.metrics,
44 robustness_scores=robustness_results,
45 statistical_significance=validation,
46 recommendation=self._generate_recommendation(validation)
47 )
48
49 def run_tournament(self, strategies: List[Strategy], data: MarketData):
50 """Tournament-style comparison across strategy variants."""
51 results = []
52 for strategy in strategies:
53 report = self.evaluate_strategy(strategy, data)
54 results.append(report)
55
56 # Rank by robustness-adjusted performance
57 ranked = self._rank_strategies(results,
58 metric="sharpe_ratio",
59 robustness_weight=0.4)
60
61 return TournamentResults(
62 rankings=ranked,
63 pairwise_comparisons=self._pairwise_tests(results),
64 regime_analysis=self._regime_breakdown(results)
65 )Key Design Decision: Robustness Weight
Final rankings weight robustness at 40% alongside raw performance. A strategy with slightly lower returns but higher stability will rank above a fragile high-performer. This reflects the reality that consistency matters more than peak performancein systematic strategies.
Robustness Test Suite
The robustness suite is designed to be adversarial. Each test targets a specific failure mode that commonly afflicts backtested strategies.
Regime Shift Test
CriticalTests performance across identified market regimes (trending, mean-reverting, volatile, calm). A strategy must maintain positive expectancy in the worst regime, not just on average.
Pass criteria: Sharpe > 0 in worst regime AND consistency score > 0.6
Parameter Sensitivity Test
CriticalPerturbs strategy parameters within ±20% and measures performance stability. Overfitted strategies show high sensitivity; robust strategies are stable.
Pass criteria: Stability score > 0.7 (performance variance under perturbation)
Data Snooping Test
CriticalApplies Bonferroni correction based on estimated degrees of freedom (implicit parameter combinations tested). Addresses the multiple comparisons problem.
Pass criteria: 95% of bootstrap samples significant at adjusted α
Lookahead Bias Test
StructuralStrict temporal validation ensuring no future information leaks into past decisions. Includes point-in-time data reconstruction for corporate actions.
Pass criteria: Zero lookahead violations detected
Survivorship Bias Test
StructuralIncludes delisted instruments in the universe. Strategies that only work on survivors are exposed when forced to trade the full historical universe.
Pass criteria: Performance degradation < 30% with full universe
1class RobustnessTestSuite:
2 """Adversarial conditions for strategy stress testing.
3
4 Philosophy: A strategy that only works under ideal conditions
5 is not a strategy—it's an artifact of overfitting.
6 """
7
8 def regime_shift_test(self, strategy, data) -> RobustnessScore:
9 """Test performance stability across market regimes."""
10 regimes = self.regime_detector.identify_regimes(data)
11
12 regime_performance = {}
13 for regime_name, regime_data in regimes.items():
14 perf = self._evaluate_in_regime(strategy, regime_data)
15 regime_performance[regime_name] = perf
16
17 # Score based on consistency, not just average
18 consistency = self._calculate_consistency(regime_performance)
19 worst_regime = min(regime_performance.values(), key=lambda x: x.sharpe)
20
21 return RobustnessScore(
22 test_name="regime_shift",
23 passed=worst_regime.sharpe > 0 and consistency > 0.6,
24 details={
25 "regime_performance": regime_performance,
26 "consistency_score": consistency,
27 "worst_regime": worst_regime
28 }
29 )
30
31 def parameter_sensitivity_test(self, strategy, data) -> RobustnessScore:
32 """Test stability under parameter perturbation."""
33 base_params = strategy.get_parameters()
34 perturbations = self._generate_perturbations(base_params, n=100)
35
36 results = []
37 for perturbed_params in perturbations:
38 strategy.set_parameters(perturbed_params)
39 perf = self._quick_evaluate(strategy, data)
40 results.append(perf)
41
42 strategy.set_parameters(base_params) # Restore
43
44 # Strategy should be stable under small perturbations
45 stability = self._calculate_stability(results)
46
47 return RobustnessScore(
48 test_name="parameter_sensitivity",
49 passed=stability > 0.7,
50 details={
51 "stability_score": stability,
52 "performance_distribution": self._summarize_distribution(results)
53 }
54 )Execution Simulation
The gap between backtest and reality is where most strategies fail. We simulate realistic execution constraints to surface these issues during research.
Market Impact Model
Volume-weighted slippage that increases with order size relative to average volume. Large orders move the market against you.
Latency Distribution
Orders execute with variable latency drawn from a configurable distribution. Tests strategy sensitivity to execution speed.
Partial Fill Model
Not all orders fill completely. Fill probability decreases with order size and increases with time-in-market.
Commission Structure
Realistic commission and fee modeling including per-share costs, exchange fees, and regulatory fees.
Why This Matters
A strategy that shows 2% annual alpha in a frictionless backtest might show -1% after realistic execution costs. The execution simulator ensures you discover this during research, not after deployment.
Statistical Validation
Every reported result includes proper statistical validation. We treat statistical significance as a minimum bar, not a goal.
Bootstrap Confidence Intervals
10,000 bootstrap samples for all performance metrics. Report 95% confidence intervals, not point estimates. A strategy with Sharpe 1.5 ± 0.8 is very different from Sharpe 1.5 ± 0.1.
Multiple Testing Correction
Bonferroni correction by default, with options for Holm-Bonferroni or Benjamini-Hochberg FDR control. If you test 100 strategies, expect 5 false positives at α=0.05.
Effect Size Requirements
Statistical significance alone is insufficient. We require minimum effect sizes (Cohen's d > 0.3) to filter out strategies that are "significant" but practically meaningless.
Out-of-Sample Validation
Strict temporal splits with embargo periods. Walk-forward validation with expanding or rolling windows. No peeking at test data during development.
What I Built
Tournament Framework
Complete evaluation pipeline with configurable phases, robustness tests, and statistical validation. Designed for reproducibility and extensibility.
Execution Simulator
Realistic market friction modeling including volume impact, latency, partial fills, and commission structures. Configurable for different market microstructures.
Robustness Test Suite
Adversarial testing framework with regime detection, parameter sensitivity analysis, and bias detection. Extensible architecture for custom tests.
Limitations & Scope
Current Scope
- •Research Only: This is a backtesting framework, not a trading system
- •Historical Data: All analysis is on historical data with known biases
- •No Guarantees: Past performance does not predict future results
Known Limitations
- →Market Impact: Models are approximations; real impact is path-dependent
- →Regime Detection: Regimes are identified in hindsight
- →Data Quality: Garbage in, garbage out—data cleaning is critical
The Fundamental Limitation
No amount of backtesting can guarantee future performance. Markets are non-stationary, and strategies that worked historically may fail going forward. This framework helps you avoid obvious mistakes and quantify uncertainty—it cannot eliminate risk.