Commit Graph

8 Commits

Author SHA1 Message Date
hanlap 782ecbb49c Fix TIER 2 HIGH (66h) Part C - Ops & Infrastructure
Task 1: Add DB Indexes (#4)
- Created migration: 5_add_candle_indexes.py
- Added composite index ix_candles_symbol_tf_time on (symbol_id, timeframe, time)
- Expected query performance improvement: 30-40% faster for candle lookups

Task 2: Add Input Validation Everywhere (#2)
- Created app/schemas/input_validation.py with Pydantic models
- Validates: timeframe, exchange, amounts, symbols, orders
- Implements per-endpoint validation for all API queries
- Updated backtest.py endpoints with comprehensive input validation
- Standardized error responses with validation details

Task 3: Fix Migration Strategy (#30)
- Created app/core/migrations.py with migration utilities
- Implemented migration lock mechanism to prevent concurrent migrations
- Replace create_all() with Alembic upgrade in main.py
- Added rollback capabilities for failed migrations
- Safety checks to ensure DB consistency

Task 4: Remove Default Credentials (#28)
- Removed hardcoded demo_user/demo_pass from config.py
- Credentials must now be provided via environment variables
- Enforces secure credential management

Task 5: Fix Redis URL (#29)
- Corrected docker-compose.yml redis URLs
- Changed from redis://redis:***@db:5432/trading_portal
- To correct: redis://redis:6379/0
- Applied to both backend-api and backend-scheduler services

All changes follow secure coding patterns and maintain backward compatibility.
Migration tests pending - see VERIFICATION_RESULTS.md
2026-07-10 11:59:56 +00:00
hanlap 81907cf3aa feat: add algorithm settings backend + integration guide
- Create /api/v1/settings/algorithms endpoint for algorithm management
- Enable/disable Algorithm #15 (liquidity_sweep) and #16 (price_action_reversal)
- Settings persist in User.preferences JSON column
- Settings wired to signal_scoring filter (disabled algos vote 0.0)
- Add comprehensive ALGORITHM_INTEGRATION_GUIDE.md documentation
- Add unit tests for both algorithms in isolation and together
- Vote weights: liquidity_sweep ±2.0, price_action_reversal ±2.5
- Correlation dampening: 0.45 when both vote same direction (pattern group)
- Tested: algorithms called in get_indicators(), passed through signal pipeline
2026-07-10 11:21:41 +00:00
Le 662586c6bc feat: simulate trading fees/slippage in backtest, compute real PnL for real trades
Backtest/walk-forward priced every fill at the exact candle close with zero
cost, making reported win rate/profit factor systematically more optimistic
than live trading. Added configurable taker-fee + slippage simulation
(defaults 0.1%/0.05% per fill) applied to every entry/exit, threaded through
walk-forward's grid search and both API endpoints.

sync_real_trades() hardcoded pnl=0 for every real trade needing it, silently
reporting break-even for real-money trades regardless of actual outcome.
Replaced with FIFO lot matching per (user, symbol, exchange), and fixed
orders.py to persist the exchange's actual average fill price instead of
the (always-None-for-market-orders) requested price, so there's real price
data to match against.

Also verified (and locked in with regression tests) that Divergence/SMC's
pivot-confirmation delay is already causally consistent between live and
backtest — no repaint, no look-ahead leak.

187 backend tests pass (+17).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-04 14:50:36 +07:00
hanlap 8839df93f3 feat: add is_trading filter to symbols API + hardcode secret paths + backfill script
- API /api/v1/symbols: add is_trading query param + is_trading field in response
- config.py: set default DB_PASSWORD_FILE=/run/secrets/db_pw.txt and ENCRYPTION_KEY_FILE=/run/secrets/enc_key.txt
- scripts/backfill_candles.py: new script to fetch 500 historical candles per symbol/timeframe for all is_trading=true symbols
- Cleared 2.88M non-trading candles from DB
2026-07-04 04:29:11 +00:00
Le 1c022264f5 Fix O(n^2) blowup and look-ahead leak in SMC/divergence backtest scoring
market_structure() (SMC) and detect_divergence() were each precomputed
once over the ENTIRE multi-year backtest range and reused unchanged for
every candle, so every candle's score could see results derived from
years of future price data — a look-ahead bug that inflated both
single-run backtest and walk-forward results, undermining the very
overfitting check walk-forward exists to provide. A prior fix bounded
this to a per-candle trailing window, which closed most of the leak but
still rescanned pivots from scratch on every candle (O(window) per
candle), too slow to enable 15m/30m walk-forward runs.

The real fix: pivot detection is itself a bounded rolling-window scan
(each position only depends on a few bars on either side), so it can be
precomputed once for the whole dataset just like BB/RSI/MACD. Per candle,
_compute_scores_series now just advances a monotonic pointer over
already-known pivots to whatever is causally confirmable as of that
candle — O(1) amortized across the whole run instead of O(window) or
O(n) per candle. Added an optional precomputed_pivots param to
detect_divergence() (backward compatible) to reuse this for RSI/MACD
divergence too.

Net effect: 16,000 candles went from 16.1s to 1.7s (confirmed empirically,
on top of an earlier ~10x from fixing the raw O(n^2)), and scaling stays
linear at 32,000 candles (3.2s). Walk-forward's timeframe options are now
15m/30m/1h/4h/1d (up from 1h/4h/1d) since 15m at the 3-year default
lookback now costs roughly 30s instead of 5+ minutes. Also wired
walk_forward.py's grid search to actually reuse one computed score series
across all 27 parameter combinations per fold (it was recomputing full
classification for every combination despite the scoring/threshold split
added earlier). 156 backend tests passing (3 new: causal-score regression,
pivot-detection-runs-once, order-block-window-bounded).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-04 10:42:32 +07:00
Le 625c2b3773 Add walk-forward backtest optimization to mitigate signal overfitting (item m)
Rolling train/test folds over 3 years of data auto-optimize the three
cheap-to-tune trading parameters (STRONG/BUY score thresholds, max hold
time) via grid search on each fold's train window, then evaluate purely
on the held-out test window. Stitching all out-of-sample results gives
an honest performance estimate uninflated by tuning against the same
data used to score it.

Split signal_scoring.py's expensive 13-algorithm scoring from its cheap
final threshold classification so grid search can replay many parameter
combinations without recomputing indicators each time. Moved the
backtest engine (fetch/precompute/simulate) out of the API layer into
app/services/backtest_engine.py so both /backtest/run and the new
walk-forward optimizer share one implementation instead of drifting
copies — same rationale as the earlier signal_service.py split (item h).

Also merges two long-diverged Alembic migration heads discovered while
adding the walk_forward_results table, so `alembic upgrade head` has a
single target again.

New: POST/GET/DELETE /walk-forward/* endpoints, a Walk-Forward tab on
the Backtest page (fold table, out-of-sample equity curve, run history).
19 new backend tests (153 total, all passing).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-04 09:15:21 +07:00
Le 6c1edcda34 fix: vá lỗ hổng RBAC + hardcode sàn ở /orders/place, nâng cấp mã hoá và CORS
- orders.place_order: yêu cầu role trader/admin (get_current_trader_user)
  thay vì bất kỳ user đã đăng nhập nào — viewer không còn đặt được lệnh thật
- orders.place_order: resolve exchange theo OrderRequest.exchange thay vì
  hardcode "mexc", fallback về credential active gần nhất nếu không truyền
- security.py: mã hoá API key chuyển AES-256-CBC -> AES-256-GCM (có xác thực
  toàn vẹn), giữ đường giải mã cũ để credential đã lưu trước đây không hỏng
- main_api.py: CORS_ORIGINS rỗng -> deny-all thay vì fallback "*" (kèm
  allow_credentials=True là cấu hình nguy hiểm)
- docker-compose.yml: đồng bộ DB_PASSWORD giữa backend-api/scheduler và db
- frontend: OrderPanel/DashboardPage truyền kèm exchange đang chọn khi đặt lệnh

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-03 21:27:47 +07:00
hanlap 34a1e91541 Initial commit: Trading Portal - FastAPI + React + PostgreSQL 2026-07-03 13:08:22 +00:00