A backtester you can put a golden test on

A backtest is a promise: “if you had run these rules on this data, this is what would have happened.” Most backtesters break that promise quietly. They fill on the bar that generated the signal, ignore costs, assume a stop and a target can’t both be hit in one bar, or produce slightly different numbers after a library upgrade. A trader can’t see any of that. They see a Sharpe ratio, and they trust it until it costs them money.

āagman’s backtesting engine was built around one requirement: the same strategy document on the same data produces the same trades, to the timestamp, every time. This part covers how:

• The pipeline from compiled plan to results, inside a sandbox
• Fill and bar-timing rules, including what happens when a stop and a target are both touched in one bar
• Costs: 5 bps fees and 3 bps slippage by default
• Golden tests: frozen fixtures, fixed documents and exact expectations
• What turns a backtest into closer-to-evidence: a shuffle test and paper tracking

Background: where backtests lie#

Before writing engine code, we listed the ways a backtest can mislead. The following table shows each one and how the engine answers it:

Most of these are one-line decisions. The work is making them impossible to bypass.

The pipeline#

The engine is a Python service. A job carries a compiled ExecutionPlan (Part 2); a worker picks it up from a Postgres-backed queue and runs it inside an E2B sandbox. The sandbox has no outbound network, a read-only mount of historical Parquet files and an output directory, so even a malicious document could only produce a wrong number.

graph LR
    P["ExecutionPlan\nintent: BACKTEST"] --> R["Path resolver\nvenue/asset/tf/date/symbol"]
    R --> L["OHLCV loader\nschema, UTC index,\nwarmup bars"]
    L --> I["Indicator registry\nTA-Lib and custom"]
    I --> S["Rule engine\nboolean series\nat bar close"]
    S --> A["Backtest adapter\nshift, exits, sizing"]
    A --> V["VectorBT\nfrom_signals"]
    V --> O["Summary, trades,\nequity curve"]

    class R,L blue
    class I,S amber
    class A,V violet
    class O green

The stages, in order:

1. Path resolution. The universe, timeframe and date range deterministically produce a list of Parquet paths, for example venue=NSE/asset=EQUITY/tf=15m/date=2026-01-03/symbol=TCS.parquet. The resolver trusts the date folders that exist on disk.
2. Loading. Files are read, checked for the required columns (ts_utc, open, high, low, close, volume) and indexed in UTC. Missing or malformed data fails the job instead of being repaired.
3. Indicators. Each indicator id maps to one implementation: custom code for SMA, EMA and RSI, and TA-Lib for the rest of the registry.
4. Rules. The rule evaluator walks the plan’s rule groups and produces boolean series at bar close. There’s no eval() and no dynamic code: each operator is a function from a fixed list.
5. Adapter. The backtest adapter shifts entries to the next bar, computes stop and take-profit exits from each bar’s high and low, and calls vectorbt.Portfolio.from_signals() with the plan’s direction, order settings, costs and a fixed seed.
6. Results. A summary, a trade table and an equity curve are serialized. The service stores job metadata and summaries in Postgres and the time series in ClickHouse, with provenance that includes the sir_hash.

VectorBT is pinned at 0.28.2, and the adapter emits a runtime warning if the installed version differs. Pinning wasn’t optional: unpinned versions can shift results across minor releases, and a backtesting product can’t explain to a user that their numbers changed because of a dependency bump.

Bar timing and fills#

The core timing rule: conditions are evaluated at the close of bar t, and the resulting entry fills at the open of bar t+1. The service’s own documentation states it, and so do the public backtesting page and playbook, because users ask.

entries = evaluate(plan.rules.entry, frame)    # True at bar t close
entries = entries.shift(1, fill_value=False)   # act at bar t+1
fill_price = frame["open"]                     # t+1 open

The one-bar shift costs something real on daily bars: a gap up after a breakout means a worse entry than the signal bar’s close. That’s the point. A user who trades this strategy live can’t buy at a close they’ve only now seen.

Intrabar exits#

Stop-loss and take-profit are percent-based (or points) from the entry price and are checked against each bar’s high and low. When one bar’s range contains both, we don’t know which was hit first without lower-timeframe data, so the engine assumes the worse outcome: the stop wins. The service documents this as an explicit path assumption for a long position: price moves open → low → high → close, so the stop executes before the target. For a long position, in simplified form:

hit_stop   = low[t]  <= stop_price
hit_target = high[t] >= target_price
if hit_stop:     exit(t, reason="STOP")      # checked first: pessimistic
elif hit_target: exit(t, reason="TARGET")

The engine also handles the case where the stop or target triggers on the entry bar itself, which VectorBT can’t do through from_signals, so the adapter builds those order records separately. The golden fixtures have scenarios for both.

What the engine supports#

• Long and short entries, with market orders that fill fully or fail. Limit and stop entry orders are modelled at the trigger price on the bar after the signal.
• One timeframe per backtest. Multi-timeframe rules are computed on resampled, closed bars only, so a daily filter never sees a day that has not finished.
• Parameter sweeps, run as many independent jobs that share one pinned engine.
• Entry session filtering (trade only between two clock times), including excluding the first or last minutes of the session.
• Percent-of-equity and fixed-quantity sizing.
• Options and futures strategies, with legs priced from the same bar-timing rules.

Costs#

Every real strategy pays fees and slippage, so the engine charges them on every fill. The defaults are 5 bps for fees and 3 bps for slippage, the same assumptions the backtest page shows next to each result. The SIR schema carries optional fees_bps and slippage_bps fields that override them, and the engine itself, called with no cost fields, charges zero, so a raw engine run is explicit about being gross. The app always fills in the default model before the plan is compiled.

Two consequences follow. First, the explanation layer states which cost model produced a result. Second, backtest, paper and live comparisons use the same cost model on every side.

Metrics#

The summary comes from VectorBT’s portfolio statistics, normalized into a typed result: total return, maximum drawdown with its start and end, win rate, profit factor and the other fields the app shows, plus the trade table and an equity curve. The result assembly is careful about statistics that some VectorBT versions compute unreliably, so it requests a safe subset and silences warnings from the rest.

Sample-size warnings live in the explanation step, written as code, not prose: a strategy with a handful of trades doesn’t get to show a confident Sharpe.

Golden tests#

Golden tests are how we kept the requirement from the first paragraph. A golden test is a fixed input, a hand-verified expected output, and an exact comparison.

In the repository they have three parts:

1. Frozen fixtures. Small, immutable Parquet fixtures laid out exactly like production data. Each scenario has its own: an RSI crossing on the final bar, a stop-loss that triggers on the entry bar, a take-profit that triggers on the entry bar. The README says not to modify them once created.
2. Fixed documents. Golden strategy payloads for the same behaviors, including EMA and RSI trend-pullback and regime-filtered strategies.
3. Determinism tests. The same plan on the same data must produce the same trades and results run after run.

A runbook describes how to run any golden payload end to end on a clean machine: ingest the data, export it to Parquet, and run the job in the sandbox, so that the whole path is exercised and not just the engine in isolation. An intraday SBIN scenario is part of that work.

The following sketch shows the shape of a golden test; the numbers in a real test are computed by hand from the fixture, not copied from the output:

def test_stop_loss_on_entry_bar():
    plan = load_plan("golden/stop_loss_entry_bar.json")
    result = run_backtest(plan, base_dir="fixtures/golden/stop_loss_entry_bar")

    assert result.trade_count == EXPECTED_TRADES        # counted by hand
    assert result.trades[0].exit_reason == "STOP"
    assert result.trades[0].exit_ts == EXPECTED_EXIT_TS  # from the fixture

Three rules governed the suite:

• Any engine change that moves a golden number is a breaking change. The test fails, and the change needs an explicit update to the expected values with a reviewer’s approval and a note explaining why the new number is more correct.
• Every bug becomes a golden test. A fix for a wrong fill starts with a fixture and a failing test, then the code change.
• Fixtures are never edited. If the data needs to change, it is a new fixture.

The suite was also a team practice. Writing a golden test forces you to compute the expected answer by hand, which is where most misunderstandings about bar timing surfaced. The three engineers I led all learned the engine by writing them, and that shared method is one of the things from āagman I’d reuse anywhere.

From a backtest to evidence#

A clean backtest on one stretch of history isn’t evidence of anything. The public picks page describes four gates that every published strategy must clear: rules fixed in advance, tested on ten years across separate stretches, beating 200 shuffled versions of its own trade timing, and tracked on paper against the backtest. The page lists closed ideas and the gate each one failed, which is the most credible thing on it.

Rules fixed in advance. The sir_hash is deterministic, so “the rules were fixed before the run” is checkable by comparing hashes.

The shuffle test. Keep the strategy’s number of trades and holding periods, randomize when they happen, and repeat 200 times. If the real result doesn’t beat the shuffles, the edge is indistinguishable from being in the market at random times. The shuffles run as ordinary jobs on the same engine and the same fixtures.

Paper tracking. The execution service runs the approved plan on closed bars as they finalize, with simulated fills, and the app compares each day’s paper trades with what the backtest predicted for the same sessions.

Known limits#

I’d rather list them than pretend they don’t exist: one timeframe per backtest, no partial exits or pyramiding, bar-level (not tick-level) fills, an assumed stop-first order when a bar contains both stop and target, and no point-in-time fundamentals. Each is a deliberate scope cut, not a claim the engine handles it.

What I’d do differently#

Write a reference loop first. A vectorized engine is fast and hard to reason about. A slow, plainly correct loop is a specification, and for random strategies on the fixtures the two should match trade for trade. We had one for the simpler strategies, and I’d extend it to every feature before adding the next.

Wire costs through from the first release. Fees and slippage were fields in the schema before they were charged in the engine. I’d ship the default cost model (and later liquidity-tiered slippage) in the first version, because a result that ignores costs flatters every strategy.

Conclusion#

The backtester is the part of āagman users trust most and can check least. That’s why it was the most tested: pinned dependencies, explicit bar timing, pessimistic intrabar rules and golden tests on frozen data. The known-limits list is the roadmap. The next part covers where its data comes from: the market-data platform and the screener.

I built āagman as Founding Software Engineer at KotiLabs, July 2025 to April 2026.