|
By Lyz Sayer, Ph.D., Sayer and Associates
It is important when considering whether or not to use or believe the results of a “test” to question its validity. In plain language validity is usually understood to be “Does the test measure what it purports to measure?” This is a simple question but one that has many and complex and varied responses. One notable statistics textbook discussed twenty different types of validity. The classical definition of validity has been stated as:
“The validity of a test is the correlation of the test with some criterion. In this sense a test has a great many different ‘validities’. [An ability test may be valid for predicting grades in English but not in Latin, or for different grades of students]”. (Gullicksen, H. Theory of Mental Tests. 1950)
Currently there are about four primary forms of validity that are used to assess the worth of tests that are used for more abstract or personality types of topics – as opposed to testing for specific skills or abilities. These are discussed in general and in their application to the development and norming of the 4D-i®.
CONSTRUCT/ CONTENT VALIDITY
Is the content of the test clear and consistently in-line with the constructs or factors being measured?
In an ability test content validity is easy to assess because one looks for items which are sample of, for example, a certain level of arithmetic or language ability. For more abstract factors, validity will usually be evaluated by Subject Matter Experts (SME’s) who are considered highly qualified and knowledgeable about the constructs being measured.
The factors or constructs were formulated as a portfolio based upon four dimensions of essential skills for success in communications, collaboration, knowledge based work, problem solving, decision-making and innovation. These constructs form the basis of the 4D-i®, the four dimensions inventory. These dimensions were drawn, by Bob Wiele, founder of OneSmartWorld, Kevin Kelloway, Ph.D., of Dalhousie University and organizational psychologists from Jackson Leadership Systems, from a wide variety of sources of learning, thinking, communications, health, positive psychology and decision-making theories as well as from well established schools of psychology. This team wrote items which they felt described the 21 key factors or learnable strategies of the 4D-i®. Kelloway and Jackson Leadership Systems statistically examined the relationship and supportability of these strategies. Out of that work, the relationship of each strategy to its “colour” group was established.
Often Construct Validity is established by having a group of people complete two similar tests and then comparing the results of certain Factors within each test. The factors being assessed in the 4D-i® are quite new and ground-breaking, and to date, there doesn’t appear to be other scientifically based tests of a similar nature against which to compare the results.
The revamping and rewriting of the 4D-i® in the spring, summer and fall of 2003 was done by three SME’s who thoroughly understood the constructs but who also came from three different perspectives, based on their 4D-i® profiles. This ensured the continuance of both construct and content validity of the items, and thus of the constructs underlying 18 strategies, and the constructs for the Thinking (Cool) and Feeling (Warm) aspects of each three major test categories – Creativity, Analysis and Decision-Making. The fourth test category, Personal Spirit, was formulated based upon three key success factors positively co-related to health and high performance – viz. outlook, sense of control and initiative,
Further construct validity has been demonstrated many times in training sessions, as participants are able to sort the items by strategy, colour and “warmth” once they have learned the constructs.
FACE VALIDITY
Do the people taking the test feel that the results accurately reflect their thinking strategies and operating styles?
The answer from experience in many training workshops is “yes”. Once the session participants understand the strategies and colour constructs, there is almost unanimous agreement with the test results profiles. Some people have thought their scores might be a little stronger or weaker but the relative strengths (what is highest or lowest) are generally supported.
For the new revision of the 4D-i®, it is expected that there will be even stronger Face Validity for two reasons. First the directions for taking the instrument, and the items on the inventory are easier to understand and respond to. Second, some changes were made internally to the 4D-i® to ensure that a full and mathematically exact method was used to determine what items were paired for comparison. In addition the Thinking/ Feeling bar graph results should reflect a new and tighter approach to scoring in this area.
CONCURRENT VALIDITY
Do the test results correlate with other measures of these factors for this person?
For example, if a person completes a self-report measure like the MBTI and the results indicate he/she is ‘extroverted’, does a survey of people who know that person agree with the assessment?
There are some strategies assessed in the 4D-i® that might lend themselves to this type of analysis but no formal studies have been planned to date. Many of the strategies refer to thinking or ‘operating’ styles which are part of one’s approach to a situation or problem but may not be evident to those around one. To have others judge whether or not someone is high or low on strategies like Get into the Flow, Flash of Insight, Scan the Situation, etc. is difficult and may not be appropriate as they would have to speculate about the rationale behind one’s behaviour.
PREDICTIVE VALIDITY
Do the test results predict behavior or performance in the future?
When people think of validity they often think of this because tests are frequently used to predict how well someone will do in a job, in a new situation or some other aspect of performance. The biggest problem with trying to establish Predictive Validity is to account for what may or may not have happened between when the test was administered and the future behaviour was assessed. Did a new hire do well because their test ‘profile’ matched with top performers, or because they had a particularly good manager who trained and supported them well?
And, in the area of assessing thinking, personality and other intrapersonal factors which one can choose to develop, will one’s efforts to improve or change have any effect on taking this test at a later date? Someone who finds that they are low in Empathize with Others and Express Feelings might go out of their way to build these skills with the result that other would rate them higher in these areas than they actually scored on the 4D-i® and a statistical validity co-efficient would be low indicting poor predictive validity.
SUMMARY
The validity of the 4D-i® is most strongly supported by Construct Validity at this time. Also the majority of the thousands of people who have taken the 4D-i® have found it very “Face Valid” for them. More importantly, the statistical reliability of the test has been established ensuring that if one takes the test one time, and then again a few months later, the results will not be markedly different.
Therefore, the 4D-i® is amply valid for the use for which it is being offered – to help individuals understand their own operating styles and the operating styles of others vis a vis thinking, intuition, communications, problem-solving, decision-making as well as personal spirit, and through this to increase the efficiency and effectiveness of organizations, teams, and meetings anywhere where people are working together collaboratively.
|
Validity and the 4D-i®
