Most clinicians have an intuitive idea of what the concept of reliability means, and that being able to demonstrate that one’s measuring instruments have high reliability is a good thing. Reliability concerns the consistency of repeated measurements, where the repetitions might be repeated interviews by the same interviewer, alternative ratings of the same interview (as a video recording) by different raters, alternative forms or repeated administration of a questionnaire, or even different subscales of a single questionnaire, and so on. One learns from elementary texts that reliability is estimated by a correlation coefficient (in the case of a quantitative rating) or a kappa (κ) or weighted κ statistic (in the case of a qualitative judgement such as a diagnosis). Rarely are clinicians aware of either the formal definition of reliability or of its estimation through the use of various forms of intraclass correlation coefficient rho (ρ).

First consider a quantitative measurement X. We start with the assumption that it is fallible and that it is the sum of two components: the ‘truth’ T and ‘error’ E. If T and E are statistically independent (uncorrelated), then it can be shown that

where Var(X) is the variance of X (i.e. the square of its standard deviation), and so on. The reliability &Rgr;x of X is defined as the proportion of the total variability of X (i.e. Var(X)) that is explained by the variability of the true scores (i.e. Var(T)):

This ratio will approach zero as the variability of the measurement errors increases compared with that of the truth. Alternatively, it will approach one as the variability of the errors decreases. The standard deviation of the measurement errors (i.e. the square root of Var(E)) is usually known as the instrument’s standard error of measurement. Note that reliability is not a fixed characteristic of
an instrument, even when its standard error of measurement (i.e. its precision) is fixed. When the instrument is used on a population that is relatively homogeneous (low values of Var(T)), it will have a relatively low reliability. However, as Var(T) increases then so does the instrument’s reliability. In many ways the standard error of measurement is a much more useful summary of an instrument’s performance, but one should always bear in mind that it too might vary from one population to another—a possibility that must be carefully checked by both the developers and users of the instrument.

Now let us complicate matters slightly. Suppose that a rating depends not only on the subject’s so-called true score T and random measurement error E, but also on the identity R, say, of the interviewer or rater R. That is, each rater has his or her own characteristic bias (constant from assessment to another) and the biases can be thought of as varying randomly from one rater to another. Again, assuming statistical independence, we can show that, if X = T + R + E, then

But what is the instrument’s reliability? It depends. If subjects in a survey or experiment, for example, are each going to be assessed by a rater randomly selected from a large pool of possible raters, then

However, if only a single rater is to be used for all subjects in the proposed study, there will be no variation due to the rater and the reliability now becomes

Of course, Pxb > Pxa. Again, the value of the instrument’s reliability depends on the context of its use. This is the essence of generalizability theory.⁽¹⁾ The three versions of p given above are all intraclass correlation coefficients and are also examples of what generalizability theorists refer to as generalizability coefficients.

Now consider two simple designs for reliability (generalizability) studies. The first involves each subject of the study being independently assessed by two (or more) raters but that the raters for any given subject have been randomly selected from a very large pool of potential raters. The second design again involves each subject of the study being independently assessed by two (or more) raters, but in this case the raters are the same for all subjects. Equations (2.2.1) and (2.2.2) are relevant to the analysis of data arising from the first design, whilst eqns (2.2.3, 2.2.4 and 2.2.5) are relevant to the analysis of data from the second design.

Here we are concerned with the estimation of a simple proportion (or percentage). We calculate this proportion using data from the sample and use it to infer the corresponding proportion in the underlying population. One vital component of this process is to ensure that the sampled population from which we have drawn our subjects is as close as possible to that of the target population about which we want to draw conclusions. We also require the sample to be drawn from the sample population in an objective and unbiased way. The best way of achieving this is through some sort of random sampling mechanism. Random sampling implies that whether or not a subject finishes up in the sample is determined by chance. Shuffling and dealing a hand of playing cards is an example of a random selection process called simple random sampling. Here every possible hand of, say, five cards has the same probability of occurring as any other. If we can list all possible samples of a fixed size, then simple random sampling implies that they all have the same probability of finishing up in our survey. It also implies that each possible subject has the same probability of being selected. But note that the latter condition is not sufficient to define a simple random sample. In a systematic random sample, for example, we have a list of possible people to select (the sampling frame) and we simply select one of the first 10 (say) subjects at random and then systematically select every tenth subject from then on. All subjects have the same probability of selection, but there are many samples which are impossible to draw using this mechanism. For example, we can select either subject 2 or subject 3 with the same probability (1/10), but it is impossible to draw a sample which contains both.

What other forms of random sampling mechanisms might be used? Perhaps the most common is a stratified random sample. Here we divide our sampled population into mutually exclusive groups or strata (e.g. men and women, or five separate age groups). Having chosen the strata, we proceed, for example, to take a separate simple random sample from each. The proportion of subjects sampled from each of the strata (i.e. the sampling fraction) might be constant across all strata (ensuring that the overall sample has the same composition as the original population), or we might decide that one or more strata (e.g. the elderly) might have a higher representation. Another commonly used sampling mechanism is multistage cluster sampling. For example, in a national prevalence survey we might chose first to sample health regions or districts, then to sample post codes within the districts, and finally to select patients randomly from each selected post code. (See Kessler⁽⁶⁾ and Jenkins et al.⁽⁷⁾ for discussions of complex multistage surveys of psychiatric morbidity.)

One particular design that has been used quite often in surveys designed to estimate the prevalence of psychiatric disorders is called two-phase or double sampling. Psychiatrists frequently refer to this as two-stage sampling. This is unfortunate, since it confuses the two-phase design with simple forms of cluster sampling in which the first-stage involves drawing a random sample of clusters and the second-stage a random sample of subjects from within each of the clusters. In two-phase sampling, however, we first draw a preliminary sample (which may be simple, stratified, and/or clustered) and then administer a first-phase screening questionnaire such as the General Health Questionnaire (see Chapter 1.8.1). On the basis of the screen results we then stratify the first-phase sample. Note that we are not restricted to two strata (likely cases versus the rest), although this is perhaps the most common form of the design. We then draw a second-phase sample from each of the first-phase strata and proceed to give these subjects a definitive psychiatric assessment. The point of this design is that we do not waste expensive resources interviewing large numbers of subjects who not appear (on the basis of the first-phase screen) to have any problems. Accordingly, the sampling fractions usually differ across the first-phase strata. However, it is vital that each of the first-phase strata have a reasonable representation in the second-phase, and it is particularly important that all of the first-phase strata provide some second-phase subjects. The reader is referred to Pickles and Dunn⁽⁸⁾ for further discussion of design issues in two-phase sampling (including discussion of whether it is worth the bother).