Clinically meaningful change: False positives in the estimation of individual change

In applied research as well as in clinical practice, it is common to evaluate the change that patients experience as a consequence of the treatment they receive. Various methods designed to evaluate this change are reviewed in this study. This review has focused on a specific aspect that has not been given proper attention: the false positive rate. For this reason, a nonchange situation was simulated (pre-post design with no differences between pre and post) and the behavior of 8 different methods was evaluated in this scenario. Different distributions as well as different sample sizes were simulated. A thousand samples were created for each simulated condition. The percentage of times each method detected a change was obtained in order to evaluate the behavior of the selected methods. Because the simulated situation is a nonchange one, any change alert was considered as a false positive. Out of the 8 evaluated methods, the standardized individual difference and the confidence intervals of linear regression are the ones with the best performance. The remaining methods fail to correctly identify and reject sampling random fluctuations. In most cases there is a tendency to consider changes that respond only to random variations as statistically reliable. However, a simple modification related in reliability estimation allows a considerable improvement in the behavior of some methods.