Scott Bristol's 'Values@Hand'.

Validity and Reliability

Summary: Life Journey Map® values measurement methodology challenges many of the popular values measurement techniques.  During the ranking process the user rank orders a randomly generated list of 130 values, presented in sets of four. Value ranking scores are the sum of two complete rankings. Validity: 71% of the users report 'High Degree of Fit' upon seeing their top 6 priority values at the end of the values ranking process. Reliability: The between test 'Mental Model' -rank score correlation is .91. (n=2500, 46% men, 54% women).


Life Journey Map® Values Measurement Methodology

Defines these conditions as necessary for Values Measurement:

  1. Values Measurement is based on a values ranking process.
  2. The individual is asked to rank order three or more distinct values at a time (forced ranking).
  3. The individual is presented with a full domain of distinct values to rank order.
  4. The full domain of values is presented randomly to avoid any biasing.
  5. The affective (feeling) nature of values allows them to be either means or ends.
  6. The individual can discard values from the ranking process.
  7. The value definitions shown while ranking are the value definitions in their totality.
  8. Value scores relate directly to the individual’s ranking process.
  9. An immediate feedback loop at the end of data collection asks the individual to confirm the validity of his or her values ranking results.

Considers these techniques inappropriate for Values Measurement:

  1. Focusing on small set of values. (Content Validity Error)
  2. Absence of a feedback loop at the end of data collection that presents initial results and asks the user to evaluate relevancy of results. (Criterion-related Validity Error)
  3. Using a Likert scale when measuring values. (Construct Validity Error)
  4. Using Polarity scale when measuring values. (Construct Validity Error)
  5. Using Forced Choice when measuring values. (Construct Validity Error)
  6. Presenting the full domain of values in a fixed sequence (non-random) for ranking. (Construct Validity Error)
  7. Presenting partial value definitions during individual scoring that then map to different value definitions for reporting results. (Construct Validity Error)
  8. Classifying and measuring separately ‘means values’ and ‘goal values’. (Construct Validity Error)


Are we measuring what we think we are measuring?

There are three types of validity: 1) content, 2) criterion-related, and 3) construct, each attends to a different question (Kerlinger,1973):

1.    Content Validity: Do the number of elements (values) being measured adequately represent the full domain of all potential elements (all human values)?

Life Journey Map® methodology:

Common Errors in Values Measurement:

2.    Criterion-related Validity: Does the measurement predict the external variables or criteria? There are two criterion we are interested in:

a)    Does the measurement process result in people identifying a set of priority values?

Life Journey Map® methodology:

In response to the question: "According to your own experience and self awareness do these 6 statements reflect the values that you believe and experience as being most meaningful, significant, and important to you at this time?" From a sample of n=2500; 46% men, 54% women:

Common Errors in Values Measurement:

b)    Do the priority ranked values predict 'value behavior': spontaneous speech and problem-solving orientation?

Life Journey Map® methodology:

Consultant observations indicate 'yes'. No systematic study has been completed at this time.

3. Construct Validity: What factors or constructs account for variance in measurement?

Life Journey Map® methodology:

Common Errors in Values Measurement:


If we measure again and again will we get the same results.

One of the challenges of values measurement is to use a ranking schema to measure a full domain of values in a manner that reduces structural biasing.

Both Rokeach and Hartman use the ranking process for values measurement and both use philosophical arguments to define and classify values in a manner that results in the size of the domain of measured values being limited to groupings no larger than 18 (Rokeach,1968,1979; Hartman,1967; Edwards & Davis,1991). In different research settings Rokeach uses different sets of 18 values (Rokeach,1979).

Hall and Tonna (1978) expanded Rokeach’s domain to 125 values but in turn gave up using the ranking schema. They use a forced choice methodology.  The goal of this approach, also used by the University Michigan, World Values Survey, is to make the value set ‘full’ or robust enough to use cross culturally. 

Life Journey Map® methodology resolves this conflict by using interactive technology to randomly present a large domain of values (130) that are rank ordered in sets of 4. Two complete ranking rounds (steps) are completed. This methodology generates both a process measure and a content measure. 

Process Measure:  'Mental Model' refers to a analogical (Gentner & eds.,1983,2001, Johnson-Laird, 1983) manner of decision making. In the ranking process this results in different patterns of scoring; i.e. ranking one value over and over again as top priority vs. ranking many different values as priority. Consequently we can correlate the first step and second step scores according to descending magnitude of ranking.  Mental Model-rank score correlation = .91.  This is a measure of our cognitive unconscious and demonstrates the reliability of Life Journey Map® values measurement methodology.

Content Measure: Item measurement refers to specific value ranking score assigned in each step. The between step Item Correlation = .45. This is not a particularly high correlation but understandable given the random presentation of the values. This random presentation skews the scoring when an important value ends up at the end of the presentation list disallowing the accumulation of ranking points. The assumption is that the more ranking steps completed the less the influence of randomization and the stabilization of the rank order of specific values. 

Preliminary measures indicate that what is reliable over a period of time is the person's set of  Field of Meaning Values. This is supported by the 71% High Degree of Fit score reported on top 6 priority values, which are a subset of the individual's Field of Meaning.


Copyright by Scott Bristol, 2002-2005