The Rapid Elearning Blog

The Rapid E-Learning Blog: quiz question 

Assessing the learner’s progress is important.  How else can we provide the best feedback or certify that the learner’s met a certain level of understanding?  That’s why we need to ask the right questions.  Avoid the following mistakes and you’ll create a more effective learning experience.

Here’s a simple quiz that demonstrates some of the mistakes I discuss below.

The Rapid E-Learning Blog: common mistakes demo

Click here to view quiz demo.

The questions are either too easy or downright stupid.

We’ve all seen them.  You take a quiz and there’s one good answer choice and all of the rest are obviously not right (or plain silly).  It’s like the course designer just went through the motions.  This type of quiz question does nothing to measure the learner.  What’s the point of the quiz?  Why’s it even there?  Put some thought into the quiz and make it meaningful.

  • Get better wrong choices from your subject matter expert.  Sometimes the reason these types of questions exist is because you don’t get enough viable wrong answers from your subject matter expert.  So you make up stuff as filler.
  • Use fewer choices.  Instead of a four or five multiple choices, just use three.  Or better yet, make it a true or false.  That means a little less work and you’re less likely to have to come up with a wasted choice.
  • You don’t always need a quiz to end your course.  Not all courses require a quiz.  If you have no interest or reason to assess the learner, create an easier way to end the course.

    The questions are set up as "gotcha" questions. 

    I worked on a project once where the customer gave me a list of questions and half of them were trick questions.  His rationale for the trick questions was that if the learner really understood the content, they’d pick up on the nuances of the questions.  That’s nonsense!

    • Don’t be an elearning fascist.  The goal’s not to make your learners look stupid or to trick them.
    • Align your questions to your objectives.  Start with clear learning objectives.  Then determine how you will know that the learner has met them.  Now you can create quiz questions based on that criterion.
    • Sometimes a quiz question is not the right way to measure understanding.  Don’t assume that the learner can work through all of the nuances of the new content.  If you really want them to learn the nuances, use simple scenarios or case studies instead of quiz questions. 

    Questions ask about content that’s not covered in the course. 

    Sometimes there’s a tendency to pack more teaching into the question.  We figure that we only have access to the learner for a short period of time, why not just add more content as we ask a question.

    • Keep it simple and direct.  Use the question to assess where the learner is at that moment.  Don’t confuse the learner by adding more course content in the question.
    • If the content is important, put it in the course.  Going back to the first point, if the content is critical to the learner’s understanding, then put it into the course prior to the quiz.  If you can’t find a place for it, perhaps that’s a good indication of whether or not it should be in the course in the first place.
    • Use the answer feedback to build a little more understanding.  Based on the learner’s response, you’re in a position to provide more information.  If you want to give more to the learner, this is a better place than in the question.

    The questions are way too wordy and make it difficult to understand. 

    This is a common issue with policy training or courses that deal with regulations.  Questions that could be simple look like they were written for someone taking a bar exam.

    • Limit the question to one sentence.  Try to keep the questions simple and concise.  Avoid paragraphs and adding a bunch of fluff.
    • Use standard question prompts to start the question.  Who?  What?  Where?  When?  Why?  How?
    • Don’t try to show off your fancy vocabulary.  Your learners come to the course at various levels of expertise and understanding.  Unless you teach certain words or concepts in the course, understand that not all of the learners will know the "fancy words."  Your best bet is to use simple or common words when possible.

    The learner doesn’t know what to do to answer the question.

    This issue is less to do with the question and more about providing enough direction to the learner.  Think about what you put on the screen and the directions you give the users. 

    • Be consistent in how the learner interacts with the screen.  I’ve made the mistake in the past where some of my text boxes look exactly like my buttons.  This confused the learners.  Make sure that those parts of the screen used for navigation and buttons are distinct and easily recognized.
    • Get rid of the junk.  The rapid elearning software gives you a lot of options and features.  If you don’t need them, get rid of the noise.  For example, if there’s no need to have a drop down list of questions, then consider turning it off.  There’s no reason to have your learners clicking around the interface when you want them focused on the question.
    • Tell the learner what to do.  The user interface might seem obvious to you, but I’ve seen plenty of people get frustrated because they’re not sure what to do.  For example, if you have a drag and drop question, let the user know that they need to drag the answer to a specific location.

    The goal is to assess your learner’s understanding of the course content.  By avoiding some of these common mistakes, you’ll create better quiz questions.  What are some mistakes that you’ve seen in the quizzes you’ve had to take?  Share them with the rest of us by clicking on the comments sections.


    Tidbits:

    • I’ll be at two conferences this week:  Learning2008 and Educause2008.  Feel free to swing by the booth and say, "Hi!"
    • Flash Player 10 was released last week.  This can impact some of your older courses.  If you’re an Articulate user, check out Gabe’s post on the Word of Mouth blog.
    • It’s not too late to cast your vote for your favorite elearning software.  Elearning! Magazine has opened up voting for best elearning software.  You can read more about it here and cast your vote.
    Related Posts with Thumbnails


    37 responses to “5 Common Quiz Question Mistakes (And How to Avoid Them)”

    Kia ora Tom!

    It’s like I said to
    Tony Karrer: When it comes to learning, the diversity can be as wide. Present day examination (standards) techniques tend to accept in a narrow band of interpretation/regurgitation and so this diversity is often looked on as ineffective learning. In fact, I believe that a lot of it can be put down to diverse interpretation.

    Ka kite
    from Middle-earth

    [...] 5 Common Quiz Question Mistakes (And How to Avoid Them) | The Rapid E-Learning Blog | Tom Kuhlmann | 28 October 2008 [...]

    October 28th, 2008

    Thanks for a great post, as always, Tom! I work in a regulatory field and we are often forced to add questions to our training, regardless of whether it makes sense to test on the material. Grrr! I’ve learned many of the things you mention along the way, based on mistakes my colleagues and I have made. Nothing makes a trainee shout louder than a double negative in a test question!

    As instructional experts, we have academic “recipes” for test question & answer formatting but unfortunately it often leads to overly complicated questions and tricky answers. The end result is that the trainee has learned nothing and is even more resentful of training. My motto is to be thoughtful and keep it simple!

    Ooh, pick me, pick me!

    My all time favourite was a list of questions submitted by the subject matter for inclusion in an assessment quiz, which included the question “Do you know where to find the first aid box?” The answer options were “yes” and “no”.

    I kid you not!

    October 28th, 2008

    Love your posts. I have one comment about using True/False questions. My high school valedictorian, one of the smartest men I’ve known, would ace every part of tests . . . except the T/F questions. I didn’t think much of it until in grad school when one of my professors discouraged us from ever using True/False questions. Why? He believed they penalized the brightest students, who read more into the questions than intended. Also, he believed few things in life are 100% true or 100% false, which makes it difficult to write an effective T/F question. I haven’t seen much hard data in this area, but how he explained it has convinced me never to use T/F questions. I’ll present users with two reposes, but use more specific options, allowing them to choose the “best” option instead of the “true” or “false” option.

    Keep up the great work!

    October 28th, 2008

    How about the “instructor” who has only 3 questions at the end of a course with six lessons! It goes back to setting your learning objectives, making sure your lessons “teach” the content effectively, and then developing a quiz that really measures something.

    And, something I learned recently (after 20 years in the computer-based training field – you can teach an old dog new tricks:) is that end of course quizzes don’t measure anything other than what they’ve retained in the last hour. According to David Sousa in “How The Brain Learns”, research on retention indicates that 70 to 90% of new learning is forgotten within 18 to 24 hours after the lesson. If we’re not testing for retention why are we testing at all? However, the real value of course quizzes may be that they are just another way for the learner to practice or rehearse new information and skills, that might help them retain information after they’ve turned off their computer.

    Good overview article. The one point on which I would disagree is your recommendation that True/False items be included. True/False items tend to be either too confusing, or too easy, and in most cases they don’t relate to the desired performance and behavioral outcomes of the course. In most workplace scenarios, the learner will not need to make a “true-false” decision when presented with information; instead, the learner is likely to need to either compose a solution independently (a parallel to a short answer response) or to choose from several available options (a parallel to a multiple choice response).

    I totally agree with the comments in the article. One thing I would add is that it is always a good idea to test run your quiz by a few folks who have reviewed the content but are NOT subject matter experts. It’s hard for us to unlearn what we know and questions that may make sense to us, don’t always make sense to our audience. Things like company speak, acronyms and other baddies invariably creep in and a beta run will always flush out the duds.

    October 28th, 2008

    I disagree with you logic of the cutting of the bread. You want to know how many slices you can get from a loaf that is 25 cm long, if each slice is .5 cm. If you sliced the bread into the number of correct slices and put it back together again, you would still have approx 25cm (OK allowing for crumbs). Of course you are changing the size of the bread everytime you make a cut, but that was not your question. JMHO.

    Nifty, shared with my Nursing faculty.

    >>the real value of course quizzes may be that they are just another way for the learner to practice or rehearse new information and skills. . . .

    Good point, Lisa. This, and more realism, is why branched scenerios are often better choices than multiple choice, T/F, etc. quizzes. They reinforce the learning in a fashion that is more similar to how the learner will use it.

    As to T/F questions, I was taught to avoid them when possible because all too often the only thing they measure is the learner’s ability to play the odds, especially if they can use POE (process of elimination) to tip the odds in their favor.

    –Allen

    [...] article on typical mistakes creating e-learning questions, which appeared on Tom’s Rapid Development E-Learning Blog, included a link to this nifty [...]

    MSome of my test pet peeves: One SME submitted material followed by a set of T/F questions in which every answer was true! She simply didn’t seem to be able to phrase any false statement about her material. Another SME submitted a test with T/F and multiple choice questions that were almost verbatim out of a policy manual. As “feedback,” she requested that we restate the policy verbatim, even if it was 3-4 paragraphs long. One more: An external vendor we sometimes work with includes tests in which any time the choice “All of the above” appears as a multiple choice option, that is always the correct response. It never appears otherwise.

    In my undergrad classes I remembered studying about designing educational assessments and measures, and one of the big ideas that has remained in my head from those many years ago is: “Begin With The End In Mind” (right about the time Covey’s 7 Habits came out).

    The point was this: after you’ve identified the expected outcome(s) from the course or lesson, you carefully design your measurement for success, whether it is a quiz, an essay, or some other type of authentic assessment. Only after completing these steps would you begin to design the learning lesson, using the expected outcomes and the evaluation as your guide. Then, simply design lessons which lead the learner toward success.

    It may seem overly simplistic: “teach to the test”, but instructional design masters (and proponents of Mastery Learning) will understand that there is a lot of power in this simplicity. I go could on and on about this, but I’ll stop here for now. Thanks Tom!

    My comment relates to True/False questions encountered on a recent self-paced lesson on Violence in the Workplace that all employees are required to take. The lesson included a narrator reading all the text that was on the screen, except the T/F questions. You had to read the T/F question, select an answer, and click the “next” or “forward” button. The next screen immediately had the return of the narrator reviewing the answer of the previous T/F question: “This is false.” the narrator would say and proceed to explain the reasons why the correct answer to the T/F question was “false”. But all the while, I was thinking the narrator was telling me my response was false or wrong. I could no longer see the question or how I answered it, and so I wasn’t sure how I had answered while I’m listening to a long-winded explanation from the narrator. The True/False questions in this lesson, combined with the narrator and the explanation of the answer, no matter how you responded, was not effective in helping the learner recognize whether they had understood the material. It was a very poor self-check design.

    I have to agree with Allen. There are methods one can use to measure the validity of an assessment question. A highly valid question would be one where, statistically, the Learner would have the best chance to score it correctly only if he or she actually knows the correct answer. Even better are questions structured in such a way that they simulate the actual work the Learner will be doing!
    I am certainly not able to assess my Learners’ understanding of the material if they have a 50/50 chance of getting the question right!
    As for reducing the number of possible answers, that only raises the statistical odds of guessing correctly. More possible answers are better – though only if those choices make sense. (It goes back to your point about getting more raw data from your SME). Don’t you just love questions like:

    How many eggs are in a package of one dozen?
    A. 4
    B. 12
    C. Red
    D. None of the above
    E. All the above

    Thanx, as always, for facilitating these great discussions
    =Chuck

    Hello! Great post, I do like the Elvis and Snail questions, as a wordy man, those are real pitfalls for me.

    I am just wondering what you made the quiz in? Maybe I’ve not plumbed the depths of Articulate Quizmaker, but it doesn’t look like what I get out of my application….

    Kind regards,
    Kev

    Good stuff. I tend to agree on the true/false. However, many times you have limited choices when creating the quizzes because you have to go with what the customer wants. So if I had to create a multiple choice with 1 good answer and three stupid ones, or a true/false that was somewhat viable, I’d go with the true/false.

    @Dennis: that’s the point of the question. It’s one of those gotchas intended to be tricky rather than test your knowlegde…unless of course you were a baker or worked for Wonder Bread where bread slicing efficiency was critical.

    @Kev: Now that our new studio upgrade has been released, I can start using the products. I created that in our new Quizmaker ’09. It’s pretty cool. You can quickly create quizzes in the old form view, or switch to a slide view where you can move your content around. Here’s part of a dinosaur demo I made for my kids a while back. I like the way I have more control now.

    Interesting points coming up in a few of these responses that put me in mind of a 9 year old kid I knew who was furious when she only got 19/20 on a school test.

    The test consisted of statements that she had to assess as being certain, likely, unlikely or impossible.

    The question she got wrong was “I will die someday.” She answered “likely”. The teacher marked this wrong, saying that the correct answer was “certain.” The child was adamant that “likely” was the correct answer because, as a practising Christian, she believed in the Second Coming and had to allow for the possibility, however small, that this might happen during her lifetime.

    Tough one.

    As someone has already indicated, some of the most facile questions can be interpreted more deeply than was the assessor’s intention.

    IDespite the fact that most people are educated, I still see the occasional double negative, which particulaly confuses the learner who is taking a true/false quiz.

    Know your audience is also still important when designing a test. I was once delivering training and an assessment (I was not the designer) to a group of adult learners who would not have been getting this particular job if they were brilliant test takers in the first place. Every time the correct multiple choice answer was “All of the above” they got it wrong. Since this training was regulatory, wrong answers had to be remediated. During discussions, I realized that they read the question, saw that the first answer was right and selected it. They’d never learned to read all the choices before making their selection. This is not a skill the course was designed to teach, but it caused lots of unecessary remediation. I now try to avoid “all” or “none” types of choices.

    Thanks for the input on “trick” questions. they annoy the heck out of me! I just took a training (for certification even) on a LMS and the quizzes were full of trick questions.

    Tom,

    I’m new to eLearning and this type of development. What tools did you use to create the Snail graphic?

    Flash, Captivate . . . ?

    Thanks

    @ Josh: the snail animation was built in Articulate Quizmaker ’09. It’s really easy to build. You don’t need any programming skills. If you can work in PowerPoint, you can work with the Quizmaker. Here’s a screen shot of the snail. If you want to look at the software, you can always download a free trial at the Articulate web site. Hope that helps.

    Snail

    October 31st, 2008

    I was frustrated by the example “who was the first U.S. President.” After playing the videos, there was no indication on how to choose the answer. However, selecting Submit told me I needed to select. So that was a good indication of the importance of providing clear instructions.

    Thanks for the post. I’ll share with my team. The new quizmaker looks good.

    Every now and then I see what you send and think that is great, but I have no idea how to do that, the common mistakes material was eye opening to say the least. Can the next post be a how you did it or at least a few clues (Please)

    I like the Einstein drag & drop question. But that’s survey question right? So no way to score or give feedback on right/wrong answer?

    @sjones: that’s a quiz question so all of the scoring and feedback features work. In fact, you can see the feedback in action when you look at the first Einstein question. It’s branches to the explanation screen, but that could be any feedback option.

    [...] Since the blank slide is empty, you can pretty much do what you want. Here are three ideas that work for issues common to a lot of courses. You can see all of the ideas in action in this demo that I did for the Rapid E-Learning Blog on 5 Common Question Mistakes (And How to Avoid Them). [...]

    thank you . iam waiting for your new letter

    [...] 5 Common Quiz Question Mistakes (And How to Avoid Them) [...]

    [...] There are probably some favorite blog posts you like to save too, like this one on getting past click and read elearning, this one on creating perfect screenshots to use in your course, or this one on common quiz mistakes. [...]

    August 17th, 2010

    I am loving this Tom.

    [...] This is a great example of how you can use Articulate Quizmaker's slide view feature to break out of that typical looking form-based quiz.  You can learn more about this quiz from the blog post, 5 Common Quiz Question Mistakes (And How to Avoid Them).  [...]