Week 6 Test Development Assignment
Your objective for this assignment is to put together everything we have learned to develop your own hypothetical psychological test. To achieve full credit, I will be looking to see how effectively you explain each of the following points. THERE ARE 20 QUESTIONS. Please type your responses in the box after each question. Use as much space as you need. Good luck!
Q1 Define the testing universe. Start by selecting a hypothetical construct your test will be measuring. Carefully operationally define your construct. Explain how you would decide on the definition of the construct. List 4-5 observable, measurable behaviors you would want your test to measure.
A

Q2 Define your target audience by listing the characteristics audience members would have. Would any test takers need special accommodations? Would test takers be willing to answer honestly? Why or why not? If not, what would you do to the test to get around this potential problem.

A

Q3 Define the testing purpose and explain how test users will be using test scores (i.e. what decisions might be made on the basis of test scores?)

A

Q4 Select a testing format. Explain what the typical stimulus and response for test takers will be. Will the test be objective or subjective? Justify your rationale by explaining the pros and cons of each in terms of sampling, construction, scoring, and response sets (see chapter 11, slides 29-30).
A

Q5 Specify your administration formats. Will the test be administered in groups or individually? Will it be timed? Who will score the test? How will the test be scored (i.e. will it use a categorical, cumulative, or ipsative method of scoring, and why)?
A

Q6 Select one of the following response set biases:
• Social desirability
• Acquiescence
• Random responding
• Faking
Explain what the bias is and address how your test will attempt to deal with the bias in scores resulting from the response set.
A

Q7 Provide 2 hypothetical test items (i.e., questions or requests you may make of the test taker).
A

Q8 Write one hypothetical test instruction for test administrator, test taker, and test scorer.
A

Q9 Explain how you will set up and conduct your pilot test.
A

Q10 Imagine you are looking at the data you have collected from your item analysis. Draw an item-characteristic curve for one item you will keep in your final test, and one for item you will not keep. Explain how you can tell whether the item is good or not by looking at the curves.
(HINT: To draw an ICC, you may use PowerPoint, search for representative images on the internet, hand-draw and scan into the document, etc.)
A

Examine the following item statistics matrix for your test:

Item Item-total Item-criterion Difficulty Discrimination Bias
Correlation Correlation
1 .4 .32 .8 2 No

2 -.23 .01 .4 1 No

3 .28 .12 .52 4 No

4. .11 .04 .23 -4 Yes
5 .3 .21 .6 2 Yes

Q11 Which items would you keep in your final version of the test? Which ones would you eliminate? Explain your decisions for each item.

A

Q12 Explain one way you would look for qualitative evidence of an item’s worthiness.

A

Q13 Give an example of a problem with your test that the pilot study could point out. Explain what change you would make in either test administration or instructions to fix the potential problem.

A

Q14 Imagine you are now at the validation study stage of test development. What procedure would you focus on most to ensure that your test results are meaningful? (I.e. see the techniques described in slides 25-29 of chapter 12 and choose one).
A

Q15 Make up hypothetical norms and cut scores for your test. Explain how your test manual will help test users interpret these raw scores. (I.e. what would different scores mean in terms of the person’s traits or abilities).

A

Imagine your test has been a great success, so much so that people are lining up around the block to purchase and start using your psychological test. Studies are now being conducted to address the reliability and validity of your measure.
Q16 Of the three types of reliability discussed in chapter 6, which would be most important in demonstrating your measure’s consistency? Explain your rationale.

A
Q17 Name a real world behavior or performance measure that would be a good criterion for establishing criterion-related validity, and explain why you choose the criterion.

A
Q18 How would future test users be able to demonstrate your measure has construct validity? (Hint: two word phrase, both words start with N).
A

Q19 Name one related construct that could be used to demonstrate convergent validity, and name one unrelated construct that could be used to demonstrate disciminant validity.

A
Q20 Just for fun – after considering your test so thoroughly, what would you name it? Keep in mind issues of face validity, creativity, marketing, and other factors.

A

Week 6 Test Development Assignment
Your objective for this assignment is to put together everything we have learned to develop your own hypothetical psychological test. To achieve full credit, I will be looking to see how effectively you explain each of the following points. THERE ARE 20 QUESTIONS. Please type your responses in the box after each question. Use as much space as you need. Good luck!
Q1 Define the testing universe. Start by selecting a hypothetical construct your test will be measuring. Carefully operationally define your construct. Explain how you would decide on the definition of the construct. List 4-5 observable, measurable behaviors you would want your test to measure.
A

Q2 Define your target audience by listing the characteristics audience members would have. Would any test takers need special accommodations? Would test takers be willing to answer honestly? Why or why not? If not, what would you do to the test to get around this potential problem.

A

Q3 Define the testing purpose and explain how test users will be using test scores (i.e. what decisions might be made on the basis of test scores?)

A

Q4 Select a testing format. Explain what the typical stimulus and response for test takers will be. Will the test be objective or subjective? Justify your rationale by explaining the pros and cons of each in terms of sampling, construction, scoring, and response sets (see chapter 11, slides 29-30).
A

Q5 Specify your administration formats. Will the test be administered in groups or individually? Will it be timed? Who will score the test? How will the test be scored (i.e. will it use a categorical, cumulative, or ipsative method of scoring, and why)?
A

Q6 Select one of the following response set biases:
• Social desirability
• Acquiescence
• Random responding
• Faking
Explain what the bias is and address how your test will attempt to deal with the bias in scores resulting from the response set.
A

Q7 Provide 2 hypothetical test items (i.e., questions or requests you may make of the test taker).
A

Q8 Write one hypothetical test instruction for test administrator, test taker, and test scorer.
A

Q9 Explain how you will set up and conduct your pilot test.
A

Q10 Imagine you are looking at the data you have collected from your item analysis. Draw an item-characteristic curve for one item you will keep in your final test, and one for item you will not keep. Explain how you can tell whether the item is good or not by looking at the curves.
(HINT: To draw an ICC, you may use PowerPoint, search for representative images on the internet, hand-draw and scan into the document, etc.)
A

Examine the following item statistics matrix for your test:

Item Item-total Item-criterion Difficulty Discrimination Bias
Correlation Correlation
1 .4 .32 .8 2 No

2 -.23 .01 .4 1 No

3 .28 .12 .52 4 No

4. .11 .04 .23 -4 Yes
5 .3 .21 .6 2 Yes

Q11 Which items would you keep in your final version of the test? Which ones would you eliminate? Explain your decisions for each item.

A

Q12 Explain one way you would look for qualitative evidence of an item’s worthiness.

A

Q13 Give an example of a problem with your test that the pilot study could point out. Explain what change you would make in either test administration or instructions to fix the potential problem.

A

Q14 Imagine you are now at the validation study stage of test development. What procedure would you focus on most to ensure that your test results are meaningful? (I.e. see the techniques described in slides 25-29 of chapter 12 and choose one).
A

Q15 Make up hypothetical norms and cut scores for your test. Explain how your test manual will help test users interpret these raw scores. (I.e. what would different scores mean in terms of the person’s traits or abilities).

A

Imagine your test has been a great success, so much so that people are lining up around the block to purchase and start using your psychological test. Studies are now being conducted to address the reliability and validity of your measure.
Q16 Of the three types of reliability discussed in chapter 6, which would be most important in demonstrating your measure’s consistency? Explain your rationale.

A
Q17 Name a real world behavior or performance measure that would be a good criterion for establishing criterion-related validity, and explain why you choose the criterion.

A
Q18 How would future test users be able to demonstrate your measure has construct validity? (Hint: two word phrase, both words start with N).
A

Q19 Name one related construct that could be used to demonstrate convergent validity, and name one unrelated construct that could be used to demonstrate disciminant validity.

A
Q20 Just for fun – after considering your test so thoroughly, what would you name it? Keep in mind issues of face validity, creativity, marketing, and other factors.

A