The Value and Challenge of Unsolicited References

I've been busy developing a new Information Fluency assessment and training program for Northwestern University's Center for Talent Development. As a result, I'm off my pace for posting new challenges.

The approach with the assessment and training is to focus on investigative searching skills. These are the competencies least engaged by Internet users. For one, they take time, although not as much time as you'd suspect. But the biggest reason is that the approaches are not easily "discoverable." They are best learned by training rather than trial and error.

One of the techniques, which I've written about before, is to investigate pages that link to the page or site being evaluated. This is accomplished by using the link: command. When I conduct workshops, very few people (teachers and students) already know about this operator.

Pages returned using the link: command are valuable because they constitute an unsolicited list of references. For the most part, authors of those pages voluntarily added a link to the site/page in question.

Just looking at a list of unsolicited references is not where the value lies. Herein lies the challenge. It requires reading and seeing patterns in the unsolicited references. This becomes difficult for students.

The easiest way to introduce link: to students is to see who links to bogus sites such as Help Save the Endangered Pacific Northwest Tree Octopus, etc.  The pages returned contain keywords such as 'hoax' which are easy to see in the snippets.

If a site/page is credible and a lot of highly respected organizations link to it that is also easy to detect. For example, links to the Institute for Library and Information Literacy Education include lots of libraries, schools and Websites for teachers. By association, if so many reputable sites link to it, it helps to establish credibility. Of course, just looking to see who links is not enough. What they have to say is also important. Plenty of schools link to the Tree Octopus site. What they say about it is what is important.

Querying link: for an organization you do not know may be helpful. By comparison, querying one you already know (like National Geographic) is a waste of time for the purpose of establishing credibility.

Seeing patterns in link: results is the challenge. And it becomes more difficult when the site/page being investigated doesn't lie at the extremes of universal respect or hoax.

The example I am wrestling with is an assessment item to determine if students know how to use the link: command. I'd also like to determine if they know how to make sense of the results. This is harder to do.  How would you intrepret the results for this query:  link:http://www.rense.com/general72/oinvent.htm   (Copy and paste this string into Yahoo).

The content is about suppression of alternative energy innovations. Over 500 pages link to it in Yahoo and many tend to support it. Does this mean it is credible? Who thinks it is credible is critically important. I'd like to hear how you interpret or wrestle with the results.

Searching and Asking Better Questions

An early mistake made in the search process is looking for the wrong thing. From our research this happens around 10% of the time. No matter how you phrase a question, about one in ten people will misinterpret it. Maybe more.

Not only is looking for the wrong thing often the first mistake one can make, it is THE unforgivable one. Using poor keywords, looking in the wrong database, browsing ineffectively are all costly in terms of time, but they an lead to a good answer with persistence and a little luck. Looking for the wrong thing, well that's a different story.

I recall the time we asked students to research the World War II perspectives of people living in the Middle West. Sure enough, there were some students who turned their attention to the Middle East. A fairly easy mistake, but fatal in terms of search success. (By the way these were highly gifted high school students--no one is immune).

Yesterday I led a workshop for grad students at a local university. I used the Declaration of Independence search challenge that I posted on July 4 (see two blogs ago) as a warm-up activity to get an idea about their search skills. It confirmed a reservation I've had about search questions for several years: there is no perfect question.

Unless the question reads like a legal document--and who really wants that--there is almost always a way to misinterpret it. This may be particularly problematic in the case of search challenge questions. It takes a bit of work to come up with a question that doesn't give away the answer. By design, one or more powerful keywords for a search are left out of the question. That's to replicate the reality of what we call the "1 in 5 rule:" on average, there are four other words than the one you used in a query that may be more effective. Finding those better keywords is the most common search challenge.

It is enlightening to see what a group will do with a question. Every time I try this, I think of a better way to pose a question. And I've had a lot of experience designing questions. That's an unintended consequence of using a search challenge, but has a lot of importance for teachers who use questioning in the classroom. It's a mistake to think you've asked a good question until you see what students do with it. If you don't catch the misunderstandings in the beginning, a lot of effort can be wasted, both students' and teachers'.

Here's an illustration using the question: "How many original copies of the Declaration of Independence are known to exist?"

If you take 'original' without considering 'copies' you might think this is a trick question. Of course, there's only one original: the one in the National Archives with all the original signatures on it. 'Copies' is a bit misleading. What the search is about is 'printed copies', of course there is a better keyword than this--and it can be discovered by searching with the word 'copy' or 'copies'. The better keyword is 'Dunlap Broadsides'. That's the printer and the name of the 200 copies of the Declaration of Independence he printed soon after the original was signed.

There were also questions (confusion) among the students about the meaning of 'known to exist'. Many of the grad students found the right answer despite having to overcome these hurdles. Some found a wrong answer. Some were still searching when I called time. If I would have included the term 'printed' in the question, or 'Dunlap Broadsides', more would have found the answer. Of course, that would have made this a pretty weak challenge.

Here's my point: if you're a teacher and want to improve the questions you give your students, try giving it to them first as a search challenge question. See what they do. What keywords do they use in a query? What words do they think are unimportant? What do they find as they search? Do they get stuck? In most cases, I think you'll find a better way to ask your question.