HMRS 5131 – Final Review
Reliability, Validity, Utility (Chapter 6)
• Three key terms related to effectiveness – reliability, validity, and utility (Chapter 6 – Pg. 187-192 “Selection Methods: Are They Effective?”) o The most important for a selection method is validity o The essence of the term validity is the extent to which scores on a selection method predict one or more important criteria o A necessary condition for validity is the reliability of measurement
 A necessary condition for a selection method to be valid is that it first be reliable
• Reliability (Chapter 6 – Slide 6) – concerns the degree of consistency or the agreement between two sets of scores on some measurement device o Refers to freedom from unsystematic errors of measurement o The consistency in measurement applies to the scores that derive from the selection method – the scores come from several types of methods that are used to measure characteristics and make decisions about people o The level of reliability can be represented by a correlation coefficient
 Correlations from 0 to 1 show the extent of the reliability – reliable methods have reliability coefficients that are .8 or higher (indicating a high degree of consistency in scores) o If raters are part of the selection method (i.e. interviewers), the extent to which different rates agree also can represent the reliability (or unreliability) of the method o Reliable methods tend to be long (SAT, GRE, GMAT, and LSAT – these tests are long because they have very high levels of reliability) o While high reliability is a necessary condition for high validity, high reliability does not ensure that a method is valid
• Validity (Chapter 6 – Slide 7-9, 33) – close in meaning by not synonymous with the critical legal term job relatedness o The extent to which a test is able to predict an important criterion such as performance is an indication of the test’s validity o Empirical or criterion-related validity involves the statistical relationship between scores on some predictor or selection method and performance of some criterion measure such as on-the-job effectiveness o A statistical relationship is usually reported as a correlation coefficient.
 The relationship between scores on the predictor and measures of effectiveness
 Correlations from -1 to +1 show the direction and strength of the relationship – higher correlations indicate stronger validity o In general, an increase in the validity of a selection method will translate into a proportional increase in the average dollar value of the annual output from employees who are selected with this method o While higher correlations are generally better, the size of the sample (and other factors) are very important for achieving statistical significance o A critical concept related to validity is generalizability
 The extent to which the validity of a selection method can generalize to other employment settings or situations o Validity generalization (VG) invokes evidence from past studies on a selection method that is the applied to a new and similar setting
 Meta-analytic techniques are used to establish VG for a method (Figure 6-2, pg. 190)
• Meta-analysis is a methodology for quantitatively accumulating results across studies
• Results are usually more reliable than results obtained from an individual study and help researchers draw conclusions
 VG is an excellent alternative to empirical validation for selection methods when a criterion-related validation study cannot be done because of inadequate sample sizes or other reasons
 A VG argument can be invoked if an organization can first locate previously conducted empirical studies showing that the same or similar methods (tests) are valid for a particular job or purpose o Content validity assesses the degree to which the contents of a selection method (i.e. the actual test or instrument items or components) represent (or assess) the requirements of the job
