Fix O(n^2) blowup and look-ahead leak in SMC/divergence backtest scoring
market_structure() (SMC) and detect_divergence() were each precomputed once over the ENTIRE multi-year backtest range and reused unchanged for every candle, so every candle's score could see results derived from years of future price data — a look-ahead bug that inflated both single-run backtest and walk-forward results, undermining the very overfitting check walk-forward exists to provide. A prior fix bounded this to a per-candle trailing window, which closed most of the leak but still rescanned pivots from scratch on every candle (O(window) per candle), too slow to enable 15m/30m walk-forward runs. The real fix: pivot detection is itself a bounded rolling-window scan (each position only depends on a few bars on either side), so it can be precomputed once for the whole dataset just like BB/RSI/MACD. Per candle, _compute_scores_series now just advances a monotonic pointer over already-known pivots to whatever is causally confirmable as of that candle — O(1) amortized across the whole run instead of O(window) or O(n) per candle. Added an optional precomputed_pivots param to detect_divergence() (backward compatible) to reuse this for RSI/MACD divergence too. Net effect: 16,000 candles went from 16.1s to 1.7s (confirmed empirically, on top of an earlier ~10x from fixing the raw O(n^2)), and scaling stays linear at 32,000 candles (3.2s). Walk-forward's timeframe options are now 15m/30m/1h/4h/1d (up from 1h/4h/1d) since 15m at the 3-year default lookback now costs roughly 30s instead of 5+ minutes. Also wired walk_forward.py's grid search to actually reuse one computed score series across all 27 parameter combinations per fold (it was recomputing full classification for every combination despite the scoring/threshold split added earlier). 156 backend tests passing (3 new: causal-score regression, pivot-detection-runs-once, order-block-window-bounded). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -30,7 +30,7 @@ router = APIRouter(prefix="/walk-forward", tags=["walk_forward"])
|
||||
async def run(
|
||||
symbol: str = Query("BTC/USDT"),
|
||||
exchange: str = Query("mexc"),
|
||||
timeframe: str = Query("4h", description="1h or 4h recommended — lower timeframes multiply the candle count and runtime"),
|
||||
timeframe: str = Query("4h", description="15m/30m/1h/4h/1d — 1w/1M don't have enough candles over a multi-year lookback to form meaningful folds"),
|
||||
total_days: int = Query(1095, ge=180, le=1825, description="Total lookback in days (default ~3 years)"),
|
||||
train_days: int = Query(270, ge=30, description="Train window size per fold, in days"),
|
||||
test_days: int = Query(90, ge=14, description="Held-out test window size per fold, in days"),
|
||||
|
||||
Reference in New Issue
Block a user