Does a Simple Trend Following Strategy Beat Buy and Hold?¶
A moving average crossover backtest on SPY, with trading costs, an out of sample test, and checks on whether the result is just luck¶
Matin Nikoomanzar
The question. The 50/200 day moving average crossover (the "golden cross") is one of the oldest trend following rules around. Hold the S&P 500 when its 50 day average is above its 200 day average, sit in cash otherwise. Does that actually beat just buying and holding the index, after trading costs, on years it wasn't tuned on?
What I did.
- Got about 26 years of dividend adjusted SPY prices.
- Split them. 2000 to 2017 is in sample, used to pick the windows. 2018 to now is out of sample, and I only look at it once at the end.
- Tried a small grid of window pairs on the in sample years only.
- Froze the winner and tested it out of sample with a 5 bps cost every time the position changes.
- Then asked how much to trust that one test, using a wider parameter sweep, a walk forward test, a bootstrap, the deflated Sharpe ratio and the probability of backtest overfitting.
Short answer. No. The rule lost to buy and hold out of sample, and the later checks show that picking the "best" windows from history was close to a coin flip.
import warnings
warnings.filterwarnings("ignore", message=".*OpenSSL.*")
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import backtest_core as bc # strategy, costs and metrics
import robustness as rb # the luck checks in section 6
plt.rcParams["figure.figsize"] = (11, 5)
plt.rcParams["axes.grid"] = True
COST_BPS = 5.0 # 0.05% per side, roughly an ETF spread plus slippage
SPLIT_DATE = "2018-01-01" # before is in sample, after is out of sample
1. Data¶
Daily dividend adjusted closes for SPY since 2000. Adjusted prices matter. Without dividends, buy and hold looks about 2% a year worse than it really was, which would make the strategy look better than it is. The prices are cached in spy_prices.csv so every run gives the same numbers.
prices = bc.load_prices("SPY", start="2000-01-01", cache_path="spy_prices.csv")
in_sample = prices[prices.index < SPLIT_DATE]
out_sample = prices[prices.index >= SPLIT_DATE]
print(f"Full sample: {prices.index[0].date()} to {prices.index[-1].date()} ({len(prices)} days)")
print(f"In sample: {in_sample.index[0].date()} to {in_sample.index[-1].date()} ({len(in_sample)} days)")
print(f"Out of sample: {out_sample.index[0].date()} to {out_sample.index[-1].date()} ({len(out_sample)} days)")
prices.plot(title="SPY adjusted close", logy=True)
plt.axvline(pd.Timestamp(SPLIT_DATE), color="red", ls="--", label="in / out of sample split")
plt.legend(); plt.show()
Full sample: 2000-01-03 to 2026-06-10 (6649 days) In sample: 2000-01-03 to 2017-12-29 (4528 days) Out of sample: 2018-01-02 to 2026-06-10 (2121 days)
2. The rule, and the usual ways backtests cheat¶
Rule. Long SPY when the fast average is above the slow one, cash otherwise.
Lookahead. The averages use today's close, so the earliest you could trade on them is tomorrow. ma_crossover_positions shifts the position by a day to make sure of that.
Free trading. Every position change pays 5 bps, so a round trip costs 10 bps. This rule barely trades, so the cost is small, but it should still be in the numbers.
Cold start. The averages need 200 days of history. If you start computing them on the first test day, the strategy sits in cash for most of 2018 for no real reason. My first version did exactly that. Now the signal is built on the full history and then sliced at the split, so it's already warmed up on day one of the test.
3. Picking the windows, in sample only¶
A small grid of (fast, slow) pairs on 2000 to 2017, ranked by Sharpe. The test years play no part in this.
FAST_GRID = [20, 50, 100]
SLOW_GRID = [100, 150, 200, 250]
grid = bc.grid_search(in_sample, FAST_GRID, SLOW_GRID, cost_bps=COST_BPS)
grid.round(3)
| fast | slow | sharpe | cagr | max_dd | trades | |
|---|---|---|---|---|---|---|
| 0 | 50 | 200 | 0.738 | 0.077 | -0.193 | 19 |
| 1 | 50 | 250 | 0.722 | 0.075 | -0.202 | 11 |
| 2 | 20 | 200 | 0.718 | 0.072 | -0.173 | 25 |
| 3 | 20 | 250 | 0.685 | 0.069 | -0.217 | 19 |
| 4 | 50 | 150 | 0.671 | 0.069 | -0.203 | 27 |
| 5 | 100 | 250 | 0.666 | 0.069 | -0.195 | 9 |
| 6 | 50 | 100 | 0.603 | 0.062 | -0.196 | 39 |
| 7 | 100 | 150 | 0.571 | 0.060 | -0.282 | 31 |
| 8 | 100 | 200 | 0.554 | 0.057 | -0.190 | 15 |
| 9 | 20 | 150 | 0.495 | 0.047 | -0.288 | 43 |
| 10 | 20 | 100 | 0.440 | 0.043 | -0.246 | 51 |
best = grid.iloc[0]
FAST, SLOW = int(best["fast"]), int(best["slow"])
print(f"Picked: fast={FAST}, slow={SLOW} (frozen from here on)")
print(f"In sample Sharpe: {best['sharpe']:.2f} | trades: {best['trades']}")
Picked: fast=50, slow=200 (frozen from here on) In sample Sharpe: 0.74 | trades: 19.0
Even this small search is a bit of data mining. With 11 pairs, the winner partly won by luck. Section 6 tries to measure how much.
4. Out of sample¶
The frozen rule on 2018 to now, data it never saw.
pos_full = bc.ma_crossover_positions(prices, FAST, SLOW) # warmed up on all history
results_full = bc.run_backtest(prices, pos_full, cost_bps=COST_BPS)
results_oos = results_full[results_full.index >= SPLIT_DATE]
results_ins = results_full[results_full.index < SPLIT_DATE]
pos_oos = pos_full[pos_full.index >= SPLIT_DATE]
summary_oos = bc.summarize(results_oos)
summary_oos.round(3)
| strategy | buy_hold | |
|---|---|---|
| CAGR | 0.092 | 0.143 |
| Annual Vol | 0.162 | 0.192 |
| Sharpe (rf=0) | 0.626 | 0.794 |
| Max Drawdown | -0.337 | -0.337 |
equity = (1 + results_oos).cumprod()
fig, axes = plt.subplots(2, 1, figsize=(11, 8), sharex=True,
gridspec_kw={"height_ratios": [2, 1]})
equity["strategy"].plot(ax=axes[0], label=f"MA {FAST}/{SLOW} (after costs)")
equity["buy_hold"].plot(ax=axes[0], label="Buy and hold SPY")
axes[0].set_title("Out of sample growth of $1 (2018 to now)")
axes[0].set_ylabel("Growth of $1"); axes[0].legend()
for col in equity:
(equity[col] / equity[col].cummax() - 1).plot(ax=axes[1], label=col)
axes[1].set_title("Drawdown"); axes[1].set_ylabel("Drop from peak"); axes[1].legend()
plt.tight_layout(); plt.show()
n_trades = int(pos_oos.diff().abs().sum())
print(f"Out of sample trades: {n_trades}")
print(f"Time invested: {pos_oos.mean():.0%} of days")
print(f"Total cost drag: about {n_trades * COST_BPS / 100:.2f}% of capital")
Out of sample trades: 8 Time invested: 81% of days Total cost drag: about 0.40% of capital
5. In sample vs out of sample¶
comparison = pd.concat(
{"In sample (2000 to 2017)": bc.summarize(results_ins),
"Out of sample (2018 on)": summary_oos},
axis=1,
)
comparison.round(3)
| In sample (2000 to 2017) | Out of sample (2018 on) | |||
|---|---|---|---|---|
| strategy | buy_hold | strategy | buy_hold | |
| CAGR | 0.077 | 0.054 | 0.092 | 0.143 |
| Annual Vol | 0.108 | 0.194 | 0.162 | 0.192 |
| Sharpe (rf=0) | 0.738 | 0.368 | 0.626 | 0.794 |
| Max Drawdown | -0.193 | -0.552 | -0.337 | -0.337 |
In sample the rule beat buy and hold on every metric. Out of sample it only won on volatility. The gap between the two columns is a first measure of overfitting, but it's one number from one split. The next section looks harder.
6. Is the result real, or luck?¶
One out of sample test can still mislead, in either direction. Maybe 2018 to 2026 just happened to be a bad stretch for trend following, or maybe 50/200 happened to be a lucky pick. These checks look at it from a few angles. The code is in robustness.py.
6.1 Every window pair, not just 50/200¶
89 pairs, fast from 10 to 100 and slow from 100 to 300. If 50/200 were special, nearby pairs should do about as well, and pairs that did well in sample should keep doing well out of sample.
F_LIST = list(range(10, 101, 10))
S_LIST = list(range(100, 301, 25))
sens = rb.sensitivity_grid(prices, SPLIT_DATE, F_LIST, S_LIST, COST_BPS)
heat = sens.pivot(index="fast", columns="slow", values="sharpe_out") - sens["bh_out"].iloc[0]
fig, ax = plt.subplots(figsize=(9, 5))
im = ax.imshow(heat.values, cmap="RdBu", vmin=-0.4, vmax=0.4, aspect="auto", origin="lower")
ax.set_xticks(range(len(heat.columns)), heat.columns); ax.set_yticks(range(len(heat.index)), heat.index)
ax.set_xlabel("slow window"); ax.set_ylabel("fast window"); ax.grid(False)
ax.set_title("Out of sample Sharpe minus buy and hold (blue = strategy wins)")
plt.colorbar(im, ax=ax); plt.tight_layout(); plt.savefig("figures/sensitivity.png", dpi=150); plt.show()
rho = np.corrcoef(sens["sharpe_in"], sens["sharpe_out"])[0, 1]
print(f"Pairs that beat buy and hold out of sample: {sens['beats_bh_out'].sum()} of {len(sens)}")
print(f"Correlation between in sample and out of sample Sharpe: {rho:.2f}")
Pairs that beat buy and hold out of sample: 3 of 89 Correlation between in sample and out of sample Sharpe: 0.13
Almost the whole map is red. And the in sample ranking barely predicts the out of sample one, so "the best pair from history" isn't telling you much about the future.
6.2 Walk forward¶
A single split is a bit artificial. Here, every January I pick the best pair using only the years before it, hold it for that year, and repeat. That starts in 2008, so it also includes the financial crisis.
picks, wf, bh_wf = rb.walk_forward(prices, F_LIST, S_LIST, first_year=2008, cost_bps=COST_BPS)
print(pd.DataFrame({"walk forward": [bc.cagr(wf), bc.sharpe(wf), bc.max_drawdown(wf)],
"buy and hold": [bc.cagr(bh_wf), bc.sharpe(bh_wf), bc.max_drawdown(bh_wf)]},
index=["CAGR", "Sharpe", "Max drawdown"]).round(3))
print("\nPairs it picked:", picks["pick"].value_counts().to_dict())
ax = (1 + pd.DataFrame({"walk forward": wf, "buy and hold": bh_wf})).cumprod().plot(logy=True)
ax.set_title("Walk forward, 2008 to now (log scale)"); plt.show()
walk forward buy and hold
CAGR 0.081 0.111
Sharpe 0.652 0.630
Max drawdown -0.360 -0.519
Pairs it picked: {'30/225': 11, '10/200': 6, '20/300': 1, '40/225': 1}
With 2008 in the window the rule does better on risk. It sidestepped a lot of that crash, so its Sharpe ends up about even with buy and hold, but it still grows a lot less money. So trend following isn't useless. It's insurance, and you pay for it in the good years.
6.3 Bootstrap: how sure can one test period be?¶
I resample the test period in random blocks (a stationary bootstrap, average block of 20 days, so calm and wild stretches stay together) and recompute the Sharpe gap 2,000 times.
diff = rb.bootstrap_sharpe_diff(results_oos["strategy"], results_oos["buy_hold"], n_boot=2000, seed=0)
lo, hi = np.percentile(diff, [2.5, 97.5])
print(f"Sharpe gap (strategy minus buy and hold): {diff.mean():.2f}")
print(f"95% interval: {lo:.2f} to {hi:.2f}")
print(f"Share of resamples where the strategy wins: {(diff > 0).mean():.0%}")
plt.hist(diff, bins=60, color="#5B6B8C"); plt.axvline(0, color="k", lw=1)
plt.title("Bootstrap distribution of the out of sample Sharpe gap"); plt.savefig("figures/bootstrap.png", dpi=150); plt.show()
Sharpe gap (strategy minus buy and hold): -0.15 95% interval: -0.50 to 0.22 Share of resamples where the strategy wins: 19%
The interval includes zero. So eight and a half years isn't enough to say for sure the strategy is worse on a risk adjusted basis. It just isn't better, and in about four out of five resamples it loses.
6.4 Deflated Sharpe ratio¶
When you try a bunch of rules and keep the best one, its Sharpe is inflated just from the trying. The deflated Sharpe ratio (Bailey and Lopez de Prado, 2014) asks how likely the winner's true Sharpe is above zero once you account for the number of rules tried and for fat tails.
for name, pairs in [("small grid (11 pairs)", rb.make_pairs(FAST_GRID, SLOW_GRID)),
("big grid (89 pairs)", rb.make_pairs(F_LIST, S_LIST))]:
R_in = rb.returns_matrix(prices, pairs, COST_BPS)
R_in = R_in[R_in.index < SPLIT_DATE]
daily_sr = R_in.mean() / R_in.std()
winner = daily_sr.idxmax()
d = rb.deflated_sharpe(R_in[winner], daily_sr.values)
print(f"{name}: winner {winner}, deflated Sharpe probability {d['dsr']:.3f}")
small grid (11 pairs): winner 50/200, deflated Sharpe probability 0.992 big grid (89 pairs): winner 30/225, deflated Sharpe probability 0.990
Even after deflating, the in sample winner very likely had a real positive Sharpe. That's not a surprise, since being in the stock market most of the time makes money. The real question is whether it beats just holding, and that's what the next check is about.
6.5 Probability of backtest overfitting¶
This one is from Bailey, Borwein, Lopez de Prado and Zhu (2014). Cut 2000 to 2017 into 16 chunks. For every way of choosing 8 of them as a pretend in sample (12,870 ways), pick the best of the 89 pairs there and check where it ranks on the other 8. PBO is the share of times the winner lands in the bottom half. 0 would mean picking the best works every time, 0.5 means it's no better than picking at random.
R_big = rb.returns_matrix(prices, rb.make_pairs(F_LIST, S_LIST), COST_BPS)
p = rb.pbo_cscv(R_big[R_big.index < SPLIT_DATE], n_blocks=16)
print(f"PBO: {p['pbo']:.2f} ({p['n_splits']:,} splits)")
plt.hist(p["logits"], bins=40, color="#5B6B8C"); plt.axvline(0, color="k", lw=1)
plt.title("Where the in sample winner ranks out of sample (left of 0 = bottom half)")
plt.savefig("figures/pbo.png", dpi=150); plt.show()
PBO: 0.57 (12,870 splits)
About 0.57. Picking the best looking windows from history did slightly worse than picking at random. That's the clearest sign the "edge" from tuning was noise.
6.6 Costs¶
rb.cost_sensitivity(prices, SPLIT_DATE, FAST, SLOW).round(3)
| cost_bps | sharpe_out | bh_out | |
|---|---|---|---|
| 0 | 0 | 0.629 | 0.794 |
| 1 | 5 | 0.626 | 0.794 |
| 2 | 10 | 0.623 | 0.794 |
| 3 | 20 | 0.617 | 0.794 |
| 4 | 50 | 0.599 | 0.794 |
Costs barely matter here because the rule only trades a handful of times. That rules out costs as the reason it lost.
7. What I take from this¶
It lost out of sample. 9.2% a year vs 14.3% for buy and hold, Sharpe 0.63 vs 0.79. Lower volatility, but the same worst drawdown (33.7%), because the COVID crash in 2020 was too fast for a 200 day average. It got out near the bottom and missed part of the rebound.
The tuning was mostly noise. Only 3 of 89 window pairs beat buy and hold out of sample, the in sample ranking barely predicted the out of sample one, and the probability of backtest overfitting was about 0.57.
But it isn't worthless. In a walk forward test that includes 2008, the risk adjusted return was about even with buy and hold, and the bootstrap interval for the Sharpe gap includes zero. It behaves like insurance. It helps in slow crashes, it costs you in bull markets, and it can't react to fast crashes.
Limitations¶
- Risk free rate is 0 in the Sharpe. With cash earning interest, the time spent out of the market should get credit, which would help the strategy a bit.
- One asset. A real test would run the same thing on other indexes and asset classes.
- Execution is simple. Fills at the close, flat 5 bps, no taxes. Taxes alone could wipe out any edge for a taxable account.
What I learned¶
Most of the work in a backtest is making sure it isn't lying to you. Lookahead, costs, cold starts and picking the best of many rules all make results look better than they are, and each one had to be found and fixed on purpose. I even found one of those bugs in my own first version.