Data Fitting

Data Analysis

With Psykinematix, collected data may be fitted with psychometric functions immediately after each session in addition to fitting automatically performed by a method (Bayesian or of constant stimuli). To access the fitting capabilities, click on the "Plotter" icon in the toolbar to display the Plotter Panel (or press shift–⌘R) and select a graphable data set (last level in the results hierarchy).

Because different data types need to be fitted with different psychometric functions or psychophysical models, Psykinematix provides custom settings for the following data types:

1) Performance
2) Reaction Times
3) Measurements

The most important customizable option is the fitting function which is constraining the range (e.g. performance is limited to the 0-100% range) and shape (e.g. monotonically increasing or decreasing) of the data distribution. Each available tab provides access to the most used psychometric functions and psychophysical models associated with each data type. These tabs can be used to find out what the most appropriate fitting function is given a data set: simply select the fitting function and some of its properties to perform a single fit to the data. This provides the best-fit values for its free parameters (e.g. the alpha and beta parameters for a performance-fitting psychometric function). However you have to remember than these values are only estimates that do not take variability in the data into account.... If you want to know how reliable these free parameter values are in terms of confidence intervals, standard error, goodness of fit, and stability, you will have to use the advanced features provided buy the "Fit..." tab.

Fitting Psychometric Functions to Performance Data

Subject's performance (% correct) as a function of some stimuli parameters (dependent variable) can be freely fitted from the "Performance" tab: the psychometric function, its direction (the relationship between performance and the dependent variable is assumed to be monotonic, either increasing or decreasing), miss rate, chance level and threshold criterion should be chosen appropriately to reflect the experimental constraints (eg: a chance level of 50% in a 2AFC).

All the available psychometric functions are cumulative distribution function (CDF) types of the form:

	for a monotonic increase
	for a monotonic decrease

where is the performance as a function of some stimulus parameter ,

is the chance level (eg: 50% in a 2AFC),

is the miss rate,

is the cumulative distribution function, with being the stimulus parameter, and are the sensitivity parameters that control the shape of the function. The available psychometric functions are the ones typically used in visual psychophysics and can be also used as models for the Method of Constant Stimuli or the Bayesian Method:

Weibull CDF
Logistic CDF
Gaussian CDF
Cauchy CDF
Right Gumbel CDF
Left Gumbel CDF

Note that for the Weibull function, and are analogous (but not similar!) to the threshold and slope respectively, while for all other functions they are analogous to the offset and spread respectively. The threshold (t) and slope (s) for a specified probability level (p) (threshold criterion) of the psychometric functions are defined as:

and

A typical scenario is illustrated below where a method provides 2 graphable data sets: a "Performance" set that plots % correct as function of the dependent variable, which can be freely fitted, and a "Model/Fit" set that plots the same data but pre-fitted with the psychometric function that was indicated under the Method Panel (the resulting fitted parameters are those shown under the "Fitting" tab in the data table associated to the 1st level of the session results). The "Model/Fit" fitting cannot be modified, and the fitting of the "Performance" set should be used at the pilot stage to discover the most appropriate psychometric function or when the pre-specified function does not properly fit a specific data set.

Fitting Reaction Times

The distribution of reaction times collected in a procedure can be plotted and fitted from the "RT" tab. The graphical representation of the reaction times can be customized: reaction times below a given level (anticipatory responses) and above a given level (late responses) can be filtered out, and the bin width can be set to any value between 10 and 100 ms. Post-stimulus RTs are included by default, but can be excluded from the histogram representation by unchecking the related check box.

The reaction time distribution can be fitted with a Weibull distribution using its probability density function:

where k > 0 is the shape parameter, λ > 0 is the scale parameter of the distribution, and is a translation parameter.

The mean and standard deviation of the Weibull distribution are given by:

Note that the accuracy of the fitting procedure depends on the bin width.

Fitting Custom Models to Measurements

The "Results" tab gives access to custom fitting of experimental measurements such as thresholds and slopes as function of some independent variables or conditions. These measurements are generally associated with summary data presented at the 1st level of the session results (e.g. "Contrast" tab). A typical example is contrast thresholds measured as function of spatial frequency (obtained through a method of constant stimuli or a staircase) which can then be fitted with a contrast sensitivity function (this example is illlustrated in the Contrast Sensitivity Task Tutorial: Effect of spatial frequency).

To perform a custom fit you simply need to specify the x and y data to be fitted, the fitting function and optionally the standard deviation of the y data. You can add your own fitting functions but some popular ones are already included: linear regression, contrast sensitivity function (CSF), variance summation model (VSM) and dipper function (TvC).

Each fitting function consists in the specification of Y(y), an optional y-transform in case the y data needs to be plotted differently than they have been collected (e.g. plotted as sensitivities instead of thresholds), and F(x) the function to be fitted to Y(y). If no y-transform is required, then simply specify 'y' for Y(y). The fitting function can be any arbitrary expression where:

the # symbol is used to provide comments,
the free parameters and their valid ranges are defined by the format: 'var = minimum value : maximum value',
the last value without assignment is the value returned by the function ('a*x+b' in the above example),
output variables are defined by the format '? var = expression'. Output variables should be descriptive properties of the model derived from the best-fit values of the free parameters (e.g. the peak frequency of the CSF).

To add a new function or modify a pre-existing one, select the 'Add New' option in the Fit pop-up menu to duplicate the previous selection and then change its name and definition. Select 'Remove' to delete the currently selected function (note that only user-defined functions can be modified or deleted). User-defined fitting functions are automatically saved in Psykinematix preferences.

Fitting Settings

The curving fitting procedure provided by Psykinematix is based on the Levenberg-Marquardt weighted least squares minimization technique (Press et al. 1986). This fitting algorithm can be also associated with a bootstrap analysis, a particular kind of Monte Carlo technique, to estimate the variability of free parameters in the fitting functions or models (Maloney 1990, Foster 1997, Wichmann 2001a, Wichmann 2001b). The bootstrap method is a resampling technique relying on a large number of simulated repetitions of the original experiment. It is well suited to the analysis of psychophysical data because its accuracy does not rely on large numbers of trials as do methods derived from asymptotic theory (like the probit analysis).

In short, the weighted least-squares fitting procedure minimizes the sum of the square of the errors between the n measured data points (data_i) and the corresponding model predictions (model_i) weighted by some factor (weight_i), expressed as:

The weights can take different forms (see Weighting scheme option below). (Chi-Square) is also used to specified the goodness of the fit.

Fitting methods:

Standard: The standard procedure fits the original data with random initial values for the free parameters, reports the best-fit parameter values without providing any precision, and optionally checks the stability of the solution. To obtain an estimate of the variability of the free parameters, a bootstrap analysis is required:
Nonparametric bootstrap: many data sets are generated by adding noise to the original data set (the amount of noise depends on the know variability of the original data) and each of them is fitted with the same function. Statistics of the free parameters are then derived from the distribution of the best-fit parameter values.
Parametric bootstrap: many data sets are generated by adding noise to the fit performed on the original data set (the amount of noise depends on the Root Mean Squared Error, RMSE, of the original fit) and each of them is fitted with the same function. Statistics of the free parameters are then derived from the distribution of the best-fit parameter values.

Note that the nonparametric bootstrap uses the measurements to generate the simulaled datasets, while the parametric bootstrap uses the model responses for the simulated data sets. The choice between nonparametric and parametric bootstrap analysis depends then on how much confidence you have in the measured data as against the confidence you have about the underlying mechanism that gives rise to the measured data.

Weighting scheme:

None: the least squares fitting is often done by simply minimizing the sum-of-squares of the vertical distances of the data from the function values, i.e. without weighting (weight_i = 1 for all data points). However in many experimental conditions, the average absolute value of these distances of the points from the curve is higher when the data point has a higher value, i.e. the standard deviation of the error is not consistent along the graph, and some weighting is required to reflect this variability and better balance the influence of each data point. Several weighting options are available:
Standard deviation: this weighting is appropriate when the variance of the measurement error is known. Note that this weighting scheme is not always the best choice if a small number of measurements have been collected because the standard deviation may then considerably vary just by chance. The relative and Poisson weighting schemes are sometimes more appropriate in these cases.
Relative weighting: this weighting is appropriate if you want all the points to have about equal influence on the goodness-of-fit.
Poisson weighting: this weighting is appropriate if you want a compromise between minimizing the absolute distance squared and minimizing the relative distance square, or if you know that the values follow a Poisson distribution (i.e. standard deviation being approximately equal to the square root of the mean).

Goodness of fit:

R² (coefficient of determination): This statistic measures how successful the fit is in explaining the variation of the data. Put another way, R² is the square of the correlation between the response values and the predicted response values. It is also called the square of the multiple correlation coefficient and the coefficient of multiple determination. R² can take on any value between 0 and 1, with a value closer to 1 indicating that a greater proportion of variance is accounted for by the model. For example, an R² value of 0.8 means that the fit explains 80% of the total variation in the data about the average.
RMSE (Root Mean Squared Error): This statistic is also known as the fit standard error and the standard error of the regression. It is also an estimate of the standard deviation of the random component in the data, and is defined as the square-root of the mean square error or the residual mean square (the residual sum of squares divided by the degrees of freedom df defined as the number of response values n minus the number of fitted coefficients m estimated from the response values). Just as with, an RMSE value closer to 0 indicates a fit that is more useful for prediction. Note that the RMSE returned by Psykinematix is an unweighted version (see the reduced Chi-Squared below for a weighted version).
(reduced Chi-Squared): The reduced chi-squared statistic is simply the chi-squared divided by the number of degrees of freedom df. A rule of thumb is that a “typical” value of for a “moderately” good fit is χ² ≈ df, that is ≈ 1 which indicates that the extent of the match between observations and estimates is in accord with the error variance. A large indicates a poor model fit. However < 1 indicates that the model is 'over-fitting' the data (either the model is improperly fitting noise, or the error variance has been over-estimated). A > 1 indicates that the fit has not fully captured the data (or that the error variance has been under-estimated).
Q: The Q measure is a distribution function that gives the probability that the minimum is as large as it is purely by chance. For small Q values, the deviation from the model is unlikely to be due to chance and the model may be incorrect. For larger Q values, the deviation from the model is more likely to arise by chance suggesting the model is an adequate description of the data. A Q of 0.1 suggests an acceptable model fit. If Q is larger than, say, 0.1, then the goodness-of-fit is believable. If it is larger than, say, 0.001, then the fit may be acceptable if the errors are nonnormal or have been moderately underestimated. If Q is less than 0.001 then the model and/or estimation procedure can rightly be called into question (Press et al. 1986). At the opposite extreme, it sometimes happens that the probability Q is too large, too near to 1, literally too good to be true!

Free Parameter Precision:

Standard Errors: the standard deviation of the best-fit parameter values obtained through the bootstrap analysis.
95% CIs: the 95% confidence intervals obtained through the bootstrap analysis, i.e. mean ± std * t(95%, df).
99% CIs: the 99% confidence intervals obtained through the bootstrap analysis, i.e. mean ± std * t(99%, df).

Check stability: because the fitting procedure with the specified psychometric function or model may be sensitive to starting values of the free parameters, it may be necessary to verify the stability of the fits. This is performed by adding noise to the free parameters and refitting to the data. The fitting procedure is considered as stable when the refitting provide values that match the free parameters statistics. Unstable parameters are followed by the * symbol in the plotted graph.

Nb samples: the number of iterations for finding the initial best-fit parameters, running the bootstrapping procedure and checking the stability.

Start/Stop: this button launches or stops the computer-intensive fitting procedure running in a separate thread so it does not interfer with the normal functions of Psykinematix. Note that the progress of the fitting procedure is reflected with a progress indicator in the Plotter panel and in the toolbar.

Remember that nonlinear curve fitting is no magic: a successful fit is highly dependent on the quality of the measured data, the selection of an appropriate function and a reasonable degrees of freedom. We recommend you to take a look at this free book available online (Motulsky 2003) as this section cannot cover the complex methodology of nonlinear fitting.

References:

Foster, D. H., & Bischof, W. F. (1997) Bootstrap estimates of the statistical accuracy of thresholds obtained from psychometric functions. Spatial Vision, 11(1), 135-139 (PDF)

Klein, S. A. (2001) Measuring, estimating, and understanding the psychometric function: A commentary. Perception & Psychophysics, 63 (8), 1421-1455 (PDF)

Maloney, L. T. (1990) Confidence intervals for the parameters of psychometric functions. Attention, Perception, & Psychophysics, 47(2), 127-134 (PDF)

Motulsky, H., & Christopoulos, A. (2003) Fitting Models to Biological Data Using Linear and Nonlinear Regression: A Practical Guide to Curve Fitting. 2003, GraphPad Software Inc., San Diego CA, graphpad.com. (PDF)

Press, W .H., Flannery, B. P., Teukolsky, S. A., & Vetterling, V. T. (1986) Numerical Recipes. Cambridge University Press (HTML Link)

Wichmann, F. A. & Hill, N. J. (2001a) The psychometric function: I. Fitting, sampling and goodness-of-fit. Perception and Psychophysics 63(8), 1293-1313 (PDF)

Wichmann, F. A. & Hill, N. J. (2001b) The psychometric function: II. Bootstrap-based confidence intervals and sampling. Perception and Psychophysics 63(8), 1314-1329 (PDF)