Walk forward optimization is the cross validation method that is commonly used to calibrate time series.
# anchored walk-forward optimization
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=3, test_size=int(0.2*X.shape[0]))
dict_thresholds = {}
for threshold in np.linspace(0.05,.5,10):
bad_threshold = False
precision_list = []
f1_list = []
for i, (train_index, test_index) in enumerate(tscv.split(X)):
X_train = X.iloc[train_index]
y_train = y.iloc[train_index]
X_test = X.iloc[test_index]
y_test = y.iloc[test_index]
clf.fit(X_train, y_train)
prob_pos = clf.predict_proba(X_test)[:,1]
# we don't consider the threshold if it gives less than 5 trades
if np.sum(np.where(prob_pos>=threshold, 1, 0))<5:
bad_threshold = True
break
precision = metrics.precision_score(y_test, np.where(prob_pos>=threshold, 1, 0))
f1 = metrics.f1_score(y_test, np.where(prob_pos>=threshold, 1, 0))
precision_list.append(precision)
f1_list.append(f1)
if not bad_threshold:
# average to have an aggregate score across the folds
dict_thresholds[threshold] = (np.mean(precision_list), np.mean(f1_list))
Sharpe ratio
The Sharpe ratio is the most common way to assess the profitability of a strategy.
\[\text{Sharpe} = \frac{\mu_{portfolio}-\mu_{risk-free}}{\sigma_{portfolio}}\]Note (1): make sure that $\mu$ and $\sigma$ are on the same scale!
Note (2): as stated in Hull, it is common to use the Treasury bills as risk-free rate. However, this rate is artificially low because financial institutions must purchase them for a regulatory requirements (so prices increase and yields decrease). In practice, derivatives traders use the LIBOR rate.
Note (3): when comparing strategies, it is common to not adjust with the risk-free return (source RL).
When $\text{Sharpe} = 1.0$, the risk is perfectly rewarded by the performance.
A good annualized Sharpe ratio would start from 0.8.
ret_free = .05 # annualized
ret_free_daily = ret_free/252
ret_daily = pd.DataFrame(portfolio_values).pct_change(periods=1).mean().values[0]
vol_daily = pd.DataFrame(portfolio_values).pct_change(periods=1).std().values[0]
sharpe = np.sqrt(252)*(ret_daily-ret_free_daily)/vol_daily