Confidence Intervals in CFTool

Uncertainty

Where CFTool’s coefficient errors come from — the covariance matrix returned by the fit, corrected with Student’s t for the residual degrees of freedom.

01General assumptions

CFTool uses one of three non-linear regression algorithms to compute the best fit of a given function to your data. Like most fitting methods, the uncertainty estimates only make sense if some general assumptions are reasonably true:

• The sample is reasonably representative of the population.
• The chosen fitting function is a sensible description of the data.
• The residuals are not strongly correlated with one another.
• The spread of the residuals is reasonably consistent across the fitted range.
• The independent variables are measured with negligible error compared with the dependent variables.*

* Except in the case of orthogonal-distance regression, where errors in the independent variables are also taken into account.

02Where the coefficient errors come from

If these assumptions are reasonable, the fitting algorithm returns a covariance matrix for the fitted coefficients. The square root of each diagonal element gives the standard error of the corresponding coefficient, and these standard errors are then used to construct the confidence intervals.

So the quoted errors on the fitted coefficients do not come from the spread of repeated fits, but from the covariance matrix returned by the fitting procedure itself.

03The Student’s t correction

CFTool does not stop at the raw standard errors — it applies a correction using the Student’s t distribution. This depends on the residual degrees of freedom: the number of fitted data points minus the number of fitted coefficients. For example, fitting y = ax + b to three data points leaves one degree of freedom, whereas fitting y = ax leaves two.

This matters because when there are only a few excess degrees of freedom, the uncertainty on the coefficients is greater than a simple large-sample Gaussian approximation would suggest. The Student’s t correction allows for this with a larger multiplier, and therefore wider confidence intervals. As the number of degrees of freedom increases, the Student’s t distribution approaches the ordinary normal distribution, so the difference becomes small.

In practical terms, CFTool estimates the coefficient errors from the covariance matrix returned by the fit, then applies a Student’s t correction based on the residual degrees of freedom. For small data sets this usually gives a more cautious, more realistic estimate of coefficient uncertainty than a pure Gaussian approximation alone.

04Important limitations

The ± value quoted on each coefficient is its margin of error — the standard error multiplied by the Student’s t factor from the previous section — and the confidence interval is simply the coefficient plus or minus that margin.

These intervals should still be regarded as approximate, especially for non-linear fitting, because they are based on the local covariance estimate of the fitted model. If the function is poorly constrained, or the model is not a good description of the data, the reported coefficient errors may still be misleading.

In weighted fitting, the interpretation also depends on whether the supplied σ values are treated as relative weights or as absolute measurement errors:

• Absolute σ — the supplied errors set the scale of the covariance estimate directly. In this case the Student’s t correction makes the reported intervals, if anything, slightly conservative.
• Relative σ — the covariance is scaled using the scatter of the residuals.

05Very small data sets