"As a rough rule of thumb, you probably need at least twenty-five years of fund performance to distinguish at the 95% significance level whether a manager has above-average competence." (R. Brealey, "Portfolio theory versus portfolio practice", J. Portfolio Management, Summer 1990.)

A performance attribution system breaks down portfolio return into a set of underlying components. For example, the PART system in BondEdge breaks total return down into the following broad categories: income return, treasury curve effects, sector/quality spread changes and bond selection effects. (This is only an outline of the system; in fact, PART provides a much more detailed analysis within each of these categories, takes account of factors such as transaction costs and strategic effects - i.e. active vs. passive returns - and provides comparisons with a specified performance benchmark.)

Performance attribution figures are often used to evaluate managers' performance, by determining whether a manager's yield curve, sector/quality or selection bets have paid off. A system which reports both portfolio-level and security-level return attributions will give an objective answer to this question.

For example, suppose a manager believes that the finance sector will outperform, and is particularly bullish on Chase Manhattan spreads; on this basis, she switches out of some existing Treasury holdings into the non-callable 6.5% 8/2005 issue. If performance attribution figures subsequently show a positive sector/quality effect and a positive selection effect for the portfolio relative to the index - and a security-level breakdown reveals that this was due to the purchase of the Chase security - then this establishes that the bet paid off.

The next question is: was the result due to the manager's skill, or merely to luck? There are two fundamentally different ways to tackle this question: one is statistical and one is based on informed judgment. For the reasons described below, Interactive Data Fixed Income Analytics recommends the approach based on informed judgment.

The statistical method involves running rigorous statistical tests on performance attribution histories. This method has many advocates in the academic literature, particularly among statisticians. The advantage of the statistical method is that it is completely mechanical and objective: for example, standard methods can be used to produce Bayesian estimates showing the expected future return distribution (relative to the index) based on past performance figures.

The statistical method suffers from two major disadvantages. The first is that it ignores the fact that management styles change over time: they may change as a fund gets larger, as market conditions change, or simply as the manager becomes more experienced. A change in style will partly or totally invalidate the performance history. Furthermore, it is not apparent from the performance history alone whether there has indeed been a change in style.

The second disadvantage is that a very long history is required to obtain statistically significant results. For example, assuming that both index and manager returns were normally distributed with the same constant variance, and the manager appeared to be outperforming by about 0.5% per annum, around 6-7 years of data would be required before one could be 95% confident in making this assertion.

In practice, the situation is even worse than this analysis suggests. It is not safe to assume that return variances are constant; it is not safe to assume that index and manager returns have the same variance; and it is not even safe to assume that returns will be normally distributed. For example, a manager who is typically overweight mortgages or callable bonds will tend to have negatively skewed returns, while a manager who makes use of portfolio insurance or stop-loss strategies may have positively skewed returns. All these considerations mean that much more complicated statistical tests are required, together with much longer histories.

The difficulty is compounded when one is trying to evaluate, not simply relative returns, but risk-adjusted returns. Prudent decision-making is based on an assessment of risk/return; but to determine whether a particular management style has been a good risk/return trade-off, one needs to estimate, not just the return difference, but its whole distribution. This requires an unrealistically long performance history.

There is no way to evade this problem. In fact, academic studies have shown that apparently high levels of outperformance can be achieved with surprisingly low levels of forecasting ability: outperformance requires only a slight "edge", which can be economically significant even though it is statistically insignificant. Thus, a dogmatic statistical evaluation would tend to undervalue the performance of most managers - and market efficiency suggests that the few who could pass the statistical tests would already have left to set up their own hedge funds.

The other way to distinguish between skill and luck is to apply the informed judgment method. Thus, in the above example, one would ask why the Chase Manhattan bonds outperformed. If the manager justified her view in a strategy paper setting out an analysis of the finance sector and Chase in particular, and if the conclusions of that paper were borne out by subsequent news and events, one would conclude that the outperformance was due to skill. On the other hand, if Chase spreads had tightened due to an unanticipated takeover by a AAA rated bank, most of the outperformance was probably due to luck.

The disadvantage of the informed judgment method is that, unlike the statistical method, it has a large subjective component: it relies heavily on market knowledge and experience. It is also more time-consuming than merely running statistical tests.

The advantage of the informed judgment method is that it makes use of all relevant information - both quantitative and qualitative - rather than relying solely on total return time series. In particular, performance attribution figures are simply one input into the process. Thus, even a single quarter of performance attribution statistics can provide useful information, if it is correctly interpreted. A second advantage of the informed judgment method is that it can take account of changes in investment style.

Like successful trading, performance evaluation cannot be reduced to a mechanical process. Judgment is always critical. Indeed, an important part of the value added by an organized investment operation is the ability to monitor the performance of its investment managers in a meaningful way, and to take appropriate action. This process must combine a rigorous quantitative analysis of historical return data - as provided by BondEdge, Compare and PART - with a qualitative analysis of trading strategies and individual trading decisions.

"The implication of this is not that the plan sponsor should go on vacation for twenty-five years but that he should not focus exclusively on achieved returns while ignoring other information about a manager's competence and diligence." (R. Brealey, "Portfolio theory versus portfolio practice", J. Portfolio Management, Summer 1990.)