Searching and Asking Better Questions

An early mistake made in the search process is looking for the wrong thing. From our research this happens around 10% of the time. No matter how you phrase a question, about one in ten people will misinterpret it. Maybe more.

Not only is looking for the wrong thing often the first mistake one can make, it is THE unforgivable one. Using poor keywords, looking in the wrong database, browsing ineffectively are all costly in terms of time, but they an lead to a good answer with persistence and a little luck. Looking for the wrong thing, well that's a different story.

I recall the time we asked students to research the World War II perspectives of people living in the Middle West. Sure enough, there were some students who turned their attention to the Middle East. A fairly easy mistake, but fatal in terms of search success. (By the way these were highly gifted high school students--no one is immune).

Yesterday I led a workshop for grad students at a local university. I used the Declaration of Independence search challenge that I posted on July 4 (see two blogs ago) as a warm-up activity to get an idea about their search skills. It confirmed a reservation I've had about search questions for several years: there is no perfect question.

Unless the question reads like a legal document--and who really wants that--there is almost always a way to misinterpret it. This may be particularly problematic in the case of search challenge questions. It takes a bit of work to come up with a question that doesn't give away the answer. By design, one or more powerful keywords for a search are left out of the question. That's to replicate the reality of what we call the "1 in 5 rule:" on average, there are four other words than the one you used in a query that may be more effective. Finding those better keywords is the most common search challenge.

It is enlightening to see what a group will do with a question. Every time I try this, I think of a better way to pose a question. And I've had a lot of experience designing questions. That's an unintended consequence of using a search challenge, but has a lot of importance for teachers who use questioning in the classroom. It's a mistake to think you've asked a good question until you see what students do with it. If you don't catch the misunderstandings in the beginning, a lot of effort can be wasted, both students' and teachers'.

Here's an illustration using the question: "How many original copies of the Declaration of Independence are known to exist?"

If you take 'original' without considering 'copies' you might think this is a trick question. Of course, there's only one original: the one in the National Archives with all the original signatures on it. 'Copies' is a bit misleading. What the search is about is 'printed copies', of course there is a better keyword than this--and it can be discovered by searching with the word 'copy' or 'copies'. The better keyword is 'Dunlap Broadsides'. That's the printer and the name of the 200 copies of the Declaration of Independence he printed soon after the original was signed.

There were also questions (confusion) among the students about the meaning of 'known to exist'. Many of the grad students found the right answer despite having to overcome these hurdles. Some found a wrong answer. Some were still searching when I called time. If I would have included the term 'printed' in the question, or 'Dunlap Broadsides', more would have found the answer. Of course, that would have made this a pretty weak challenge.

Here's my point: if you're a teacher and want to improve the questions you give your students, try giving it to them first as a search challenge question. See what they do. What keywords do they use in a query? What words do they think are unimportant? What do they find as they search? Do they get stuck? In most cases, I think you'll find a better way to ask your question.