Watch Out for Those Splits

日期:2013/03/14

Written in honour of all splitters, rookies and lookbackers:

A common pursuit in decision making is to compare data of many kinds and find the best way of separating the observations into different classes.

For example you mite be considering what are the factors that contribute to long life. Weight, height, mid range of parents age , % of meat consumed , exercise mite be the variables. You mite find that when parents mid range was above 140, and meat eaten less than 2 times week, and exercise more than 5 hours a day, the longevity is 10 years higher than the average.

Trees are usually used to separate the groups. An example picture of such a tree is here: [pic ] Lots of good pictures of such trees and a discussion of a typical tree sort is in An Introduction to Classification and Regression Tree (cart) by Roger Lewis. And programs like CART and AID and many modern variants are widely used in medicine and markets. In fact all market people systematically or implicitly consider such problems. When does the market go up? When gold is down more than 10, when bonds are up more than 100, and the previous 5 day sp move is between -1% and 0 %? The problem is that there are many variables to consider, and many levels at which one would reasonably split the data.



A good starting point in considering such problems is to note that when there are two mutually exclusive groups X and Y the variance of the difference between means is the sum of the variances of the means. The variance of a mean is the variance of an individual observation divided by the number of observations. For example, consider the average move in sp during a day has a sd of 10 and variance of 100. If you take two independent samples of 10 from such a distribution , the variance of one mean will be 100/10= 10 and the variance of the difference between means is 20. The standard dev of the difference is 4.4 What does this mean in practice? One will round. A 50% confidence interval in which 50% of the observation range is 0.65 x 4.4 = 3 points on either side of the difference between means which we'll assume to be 0. That means that by chance half of all the observations will show a difference in means ranging from -3 to +3. To find a region where 95% of the observations lie, ie. A 95% confidence interval we'd multiply 2 x 4.4 =8.8. In other words to find a difference that has less than a 5% chance of occurring thru randomness we would have to find a difference between means of 8.8 on any binary split.

But of course that's not all. Normally we look at 2, or 3 or 4 different splits to find the one that gives the greatest difference. It turns out that if we look at the highest of two differences, the average difference is 1.5 times as high as for one difference. In other words by randomness , the standard deviation for the highest of 2 difference between means is 1.5 x 4.4 = 6 a 50% confidence interval assuming randomness for the highest of 2 differences is 0.65 x 6 or from -4 to + 4. A 95% confidence interval would be between 2×6 and -2 x 6 i.e. between -12 and +12.

Now there were several assumptions made in this analysis. We assumed normality. And we didn't take account that there is sampling without replacement for one. And there are two cases: you look at the algebraic difference (2 tailed test, as above) or you try to maximize the *magnitude* of the difference (the 1 tailed approach). To get a good handle on it, Doc and I simulated what would happen if we divided up 20 random observations from a distribution that had a mean of 0 and a standard deviation of 10. This corresponds to the most frequent and typical thing one does when dividing up stock market changes in points (S&P rises or falls by about 10 full points a day). We then took two random subsamples of 10 each and calculated the mean of each of the two groups of 10. We then looked at the absolute value of the average difference between means when we repeated the process 1000 times. It turns out that the average absolute difference between means is 3.5 for a single split (this 3.5 absolute difference is in line with the std dev of 4.4 previously estimated)), 5 for the highest of two splits, and about 6 for the highest of 3 splits. This leads to incredibly high differences that you must be aware of when you split data to have anything more than a 50% chance that the difference occurred thru luck alone. To be 95% sure that you have something departing from luck for the highest of 3 splits you'd need a difference between the groups of 10. As the numbers in the group get less than 10 ,i.e. for a second or third split, the numbers would get increasingly large. This is a warning to all who search for regularities in data using methods that are implicitly or explicitly like tree sorts. By vic and doc.

Here is a more extensive table of simulation results

Table 1
Col A: number of observations being split
Col B: number of splits considered, best one is chosen
——-
Col C: mean absolute difference generated by best split (in S&P points)
Col D: standard deviation of absolute difference
Col E: lower 5% confidence interval
Col F: upper 95% confidence interval

20 1 | 3.56 2.66 0.27 8.65
20 2 | 4.97 2.67 1.24 9.81
20 3 | 5.84 2.64 2.07 10.55
20 4 | 6.45 2.55 2.84 11.05
48 1 | 2.26 1.72 0.18 5.63
48 2 | 3.25 1.71 0.90 6.40
48 3 | 3.75 1.66 1.36 6.78
48 4 | 4.20 1.68 1.79 7.24

Table 2 same as above but
Col C: average (algebraic) difference between means (in S&P points)
Col D: standard deviation of difference between means

20 1 | -0.06 4.61
20 2 | -0.13 5.62
20 3 | 0.01 6.42
20 4 | -0.07 6.88

Let's try to apply this to a typical example.

The 30 year bonds each day have a move of about 80 points.i.e a move from 142 exactly to 142 and 26 /31. Call that the stand dev . Let's say you have 100 observations of bonds in your sample. And you split it into two equal halves based on whether gold is up the previous day or down. The variance of any mean difference would be 6400 x2 /50 =256 . That gives a stand dev of 16.

Turn to the prob that a standard normal deviate will be between -z and + z. the prob is 50% that a normal deviate will be etween -0.65 and + 0.65, the prob is 90% that a stand norm dev will be betweeen -1.6 andd +1.6. Thus if you find a difference that's less than 0.6 x 16 = 10 its 50% chance, and its 90% chance to be 1.6 x 16