Personal Online Notedesk

Site Blog

Evaluating the system: Part 1

Posted on January 6, 2011 at 5:00 AM

Among the kinds of inferential statistics that are most useful to traders are t-tests. T-tests are useful for determining the probability that the mean or sum of any series of independent values (derived from a sampling process) is greater or less than some other such mean, is a fixed number, or falls within a certain band. For example, t-tests can reveal the probability that the total profits from a series of trades, each with its individual profit/loss figure, could be greater than some threshold as a result of chance or sampling. These tests are also useful for evaluating snipes of returns, e.g., the daily or monthly returns of a portfolio over a period of years. Finally, t-tests can help to set the boundaries of likely future performance (assuming no structural change in the market), making possible such statements as “the probability that the average profit will be between x and y in the future is greater than 95%”.


Although, for statistical reasons, the system developer should seek the largest sample possible, there is a trade-off between sample size and representativeness when dealing with the financial markets. Larger samples mean samples that go farther back in time, which is a problem because the market of years ago may be fundamentally different from the market of today-remember the S&P 500 in 1983? This means that a larger sample may sometimes be a less representative sample, or one that confounds several distinct populations of data! Therefore, keep in mind that, although the goal is to have the largest sample possible, it is equally important to try to make sure the period from which the sample is drawn is still representative of the market being predicted.


Let us look at how statistics are used when developing and evaluating a trading system. The examples below employ a system that was optimized on one sample of data (the m-sample data) and then run (tested) on another sample of data (the out-of-sample data).



Categories: Articles