The chosen cohort of study participants will inevitably affect the study findings. This is partly related to study setting (community, primary or secondary care; Sect. 5.2) but also to participant inclusion/exclusion criteria established at study outset. Factors which may be controlled for, such as patient age, ethnicity, presence of comorbidities, and stage of disease, may all differentiate the study population from patients typically seen in clinical practice. This will relate in part to the nature of the study, be it proof-of-concept/phase I/II or pragmatic/phase III, with less stringent inclusion/exclusion criteria in the latter, which ideally aims to recruit consecutive patients.

The inclusion of a control group, as occurs in proof-of-concept/phase I/II studies, will also need to be discussed, since this is not something which occurs in clinical practice. However, significant numbers of individuals with neither dementia nor mild cognitive impairment are referred to memory clinics, whose exact problem remains unclear but is probably heterogeneous, as reflected in the various diagnostic labels which have been applied, including “memory complainers”, “worried well”, subjective memory impairment, mild cognitive dysfunction, and functional memory disorder (Schmidtke et al. 2008). Some may in fact be in a prodromal phase of a dementia disorder (Mitchell et al. 2014).

It is generally recognised that patients with dementia included in clinical research studies are systematically younger than patients from the general population (Schoenmaker and Van Gool 2004). Patient gender ratio may also be an issue (Sect. 4.​1.​2.​2).

To ensure homogeneity of patient cohorts, multiple testing centres may be required, with need for standardization of index test and reference standard application across centres.

The chosen study design will also impact on test results. Cross-sectional studies, the idiom of clinical practice, will inevitably incur some diagnostic error, even in the most skilled clinical hands, with misclassification of patients and hence their study results, which may potentially inflate or dilute measures of discrimination. Reassessment of participants after a period of time, a longitudinal study, makes excellent clinical sense in order to review and, if necessary, to revise diagnoses, but obviously such a policy of delayed verification increases study duration and complexity, and will also lead to the problem of dropouts and how to handle missing data.

Consistent application of index test and reference standard, with operationalization of their administration if necessary, is also critical to the applicability of test results. Inevitably different clinicians will apply tests in slightly different ways (Larner 2014a), although this is less of a problem with laboratory and imaging technologies which undergo quality control and for which standardized protocols may be developed.

Because of case selection and use of a control group (i.e. the ideal circumstances of a case-control type study), proof-of-concept/phase I/II studies inevitably have better outcomes, in terms of sensitivity and specificity and other measures of discrimination, when compared to pragmatic/phase III where discrimination is more difficult, as in clinical practice.

A results related issue, which may need some consideration, concerns the apparently better outcomes for performance-based cognitive screening instruments used in cross-sectional studies as compared to informant-based scales, which might be anticipated to have “added value” because of their longitudinal/ambispective aspect, and biomarker studies, despite their apparently addressing disease biology (clinicobiological measures) and also having a longitudinal aspect (see for example Sect. 4.​3.​2; compare accuracy data in Tables 4.​3, 4.​4, and 4.​5; Landau et al. 2010; Richard et al. 2013). In particular, delayed memory recall paradigms appear to have particular diagnostic accuracy for Alzheimer’s disease (e.g. Duff et al. 2008; Gavett et al. 2012). Pragmatically this may be a blessing, in that “bedside” tests of memory may be as good as currently available sophisticated technological investigations, but with greater ease of use and availability and lesser cost. This may relate in part to the nature of the estimates of diagnostic accuracy used.

Comparing measures of discrimination for performance based and informant based cognitive screening instruments and for AD biomarkers, accepting that this may be comparing different things and in many cases not head-to-head, performance based tests seem to do as well as if not better than the others, despite their failure to address disease biology or to reflect fully the longitudinal process versus a cross-sectional assessment.

Since most diagnostic test accuracy studies cite test sensitivity and specificity as one, if not the key, amongst their outcome measures, as per the recommendations of both STARD (Bossuyt et al. 2003) and STARDdem (Noel-Storr et al. 2014) guidelines, something needs to be said about the selection of test cutoff(s), since this decision will critically influence sensitivity and specificity and a number of other measures of discrimination (Sect. 2.​2.​3). Indeed, the choice of cutoff or threshold determines most of the measures of test discrimination (Sect. 3.​3), namely sensitivity, specificity, predictive values, likelihood ratios, diagnostic odds ratio, clinical utility indexes and, by extension, weighted comparison. Other measures, such as area under the receiver operating characteristic curve (AUC ROC), Q* index, and Cohen’s d, are independent of the chosen cutpoint. Cutoff determined by maximal Youden index is of equal or greater sensitivity than cutoff determined by maximal test accuracy for all cognitive screening instruments examined in the author’s clinic (Larner 2015; Table 4.​7) but further studies on this point are required to confirm or refute the generality of this observation.