Searching and Asking Better Questions

An early mistake made in the search process is looking for the wrong thing. From our research this happens around 10% of the time. No matter how you phrase a question, about one in ten people will misinterpret it. Maybe more.

Not only is looking for the wrong thing often the first mistake one can make, it is THE unforgivable one. Using poor keywords, looking in the wrong database, browsing ineffectively are all costly in terms of time, but they an lead to a good answer with persistence and a little luck. Looking for the wrong thing, well that's a different story.

I recall the time we asked students to research the World War II perspectives of people living in the Middle West. Sure enough, there were some students who turned their attention to the Middle East. A fairly easy mistake, but fatal in terms of search success. (By the way these were highly gifted high school students--no one is immune).

Yesterday I led a workshop for grad students at a local university. I used the Declaration of Independence search challenge that I posted on July 4 (see two blogs ago) as a warm-up activity to get an idea about their search skills. It confirmed a reservation I've had about search questions for several years: there is no perfect question.

Unless the question reads like a legal document--and who really wants that--there is almost always a way to misinterpret it. This may be particularly problematic in the case of search challenge questions. It takes a bit of work to come up with a question that doesn't give away the answer. By design, one or more powerful keywords for a search are left out of the question. That's to replicate the reality of what we call the "1 in 5 rule:" on average, there are four other words than the one you used in a query that may be more effective. Finding those better keywords is the most common search challenge.

It is enlightening to see what a group will do with a question. Every time I try this, I think of a better way to pose a question. And I've had a lot of experience designing questions. That's an unintended consequence of using a search challenge, but has a lot of importance for teachers who use questioning in the classroom. It's a mistake to think you've asked a good question until you see what students do with it. If you don't catch the misunderstandings in the beginning, a lot of effort can be wasted, both students' and teachers'.

Here's an illustration using the question: "How many original copies of the Declaration of Independence are known to exist?"

If you take 'original' without considering 'copies' you might think this is a trick question. Of course, there's only one original: the one in the National Archives with all the original signatures on it. 'Copies' is a bit misleading. What the search is about is 'printed copies', of course there is a better keyword than this--and it can be discovered by searching with the word 'copy' or 'copies'. The better keyword is 'Dunlap Broadsides'. That's the printer and the name of the 200 copies of the Declaration of Independence he printed soon after the original was signed.

There were also questions (confusion) among the students about the meaning of 'known to exist'. Many of the grad students found the right answer despite having to overcome these hurdles. Some found a wrong answer. Some were still searching when I called time. If I would have included the term 'printed' in the question, or 'Dunlap Broadsides', more would have found the answer. Of course, that would have made this a pretty weak challenge.

Here's my point: if you're a teacher and want to improve the questions you give your students, try giving it to them first as a search challenge question. See what they do. What keywords do they use in a query? What words do they think are unimportant? What do they find as they search? Do they get stuck? In most cases, I think you'll find a better way to ask your question.

Detecting Bias

Objectivity is a characteristic of information that adds to its credibility. An objective article or posting either avoids taking sides or represents them fairly. An objective author typically has to overcome personal biases to write objectively.

Biases are hard to avoid. We all have them. (Did you catch my bias?)

Biases creep into our writing and into our reading. This latter condition can cause confusion for students who think they see bias in an author who really intended none. The bias occurs in the eyes of the beholder. I've seen this recently in students' response to an article by Carl Sagan, believing his intent was to frighten them. I was actually surprised by their conclusion and believe the words he used could unsettle someone--or make them feel existentially small--but that interpretation is in the mind of the reader.

Using bias or objectivity as the basis for accepting or rejecting information is problematic. Foremost, there's the problem of a skewed personal perspective that changes the way information was intended to be received. How can I tell if the problem is the author or me? Without other points of reference that may be impossible to determine.

That leaves a couple of options. Don't base the worth of information solely on whether it evidences bias or not. Include this in the mix of the author's qualifications, the publisher's reputation, the accuracy of claims and what others say about it.

The last point may be the critical one. If you believe in the "wisdom of the crowd" approach, then what a lot of other people think about an author's writing may come close to the truth. There's always the specter of 'group think' where the majority is wrong, but that tends not to happen in an open exchange of information or when information is volunteered freely. This is where checking to see what pages link to an author's work is extremely valuable.

The link: operator can be used to retrieve pages found in a search engine database (Yahoo's is the most powerful in this regard). Most of the external pages that link to a site don't have to. So why did they go to the trouble to put a link to a particular page on their site? Usually, it's to point to something of value (insight, humor, news, etc.) or to warn (as in the case of hoax sites). The best thing to ask before looking at the list of pages that link to information you want to evaluate is: "if this information is legitimate, who do I expect to find linking to it?" This involves some speculation on your part, but if you expect to find experts in a topic linking to, say, the Northwest Tree Octopus site (marine biologists, etc.) and you find none (which is the case), then you have the benefit of a crowd of witnesses expressing their personal views about the information. In the case of the Tree Octopus, people who link to it tend to find the information funny while others claim it's a fabrication. Based on the consensus of views you can determine there is insufficient support that the octopus has taken to the trees or is endangered.

Evidence of bias can also be found in the words an author uses. Strong language (often adjectives such as 'stupid' or worse) are seeded among biased views. The infamous Martin Luther King Jr. site is an example of this. Teaching students to ask, "why did an author use THAT word?" helps to uncover bias. As a student, I'm not sure I recall any of my teachers addressing how to detect bias, but it would have been valuable.

So I leave you with this challenge. Below is a link to a blog by an author who cannot be identified. Since he's self-published, there's no known publisher to fall back on. Using only the author's words, you can learn a lot about the individual. What would you say are this individual's values or biases? Under what circumstances would you use the information he authors? This could be a good activity for a group of students.

http://shroudedindoubt.typepad.com/

For more on the link: operator and how to use it, visit this tutorial.

Happy July 4!

Here's a new Search Challenge in time for Independence Day. It was inspired by reports of the discovery of a copy of the Declaration of Independence in the National Archives in London. Originally, around 200 copies were printed. The whereabouts of the majority of these is unknown.

http://21cif.com/tutorials/challenge/SC001/SC_034.swf


See if you can rise to the challenge.

Information Forensics goes to School

Yesterday, Dennis O'Connor and I presented "Information Forensics goes to School" at the NECC conference in Washington, D.C.*

We talked about how the investigative aspects of information fluency, which constitutes part of the missing curriculum at most schools, cuts across all subject matters, 21st Century Skills as well as ethics. The ability to track down and evaluate information about authors, publishers, publication dates, content and references is vital to research in any subject. The techniques used to locate and evaluate such information is integral to ICT literacy and the ability to cite information is definitely an aspect of ethics in scholarship. It's deplorable that the task of helping students develop these skills and attitudes is most often left to the school librarian. The entire faculty should share this responsibility.

We didn't actually rant that much during the session, but it needs to be said.

We also introduced a new resource being used by nearly 1,000 students at the Center for Talent Development at Northwestern University this summer: Investigative Searching 20/10. This is a performance-based assessment and training package designed using Moodle and an array of Websites we created specifically for the purpose of diagnosing students' skills. The focus is on investigative searching and evaluation.

The results are as we predicted: even the most gifted students in middle school and high school, on average, cannot locate critical online information and evaluate it. Without someone to show them effective techniques, they browse hoping to stumble upon information that may be helpful to identify an author, a publisher, the date of publication, etc. Couple this with students' general "need for speed"--which contributes to important information being overlooked--and you have a recipe for failure and poor research.

Following a pretest which requires students to demonstrate techniques like querying and truncation, a series of self-paced tutorials steps through 5 main techniques, each followed by a mastery check activity. The 6-8 hour experience is capped off with a posttest of items comparable to the pretest. Students, on average, improve 15 to 20 percentage points. Not bad for a few hours work. The improvement would be even greater with continued use.

Sound like something you could use?

We are offering educators a chance to experience this package starting July 7. You may enroll up to July 28. We'll leave this section just for educators open for 4 weeks, although it takes less than a week to complete it. Our hope is that you will mine it and discover ways to use it with students as part of a systematic approach to online research or as part of an instructional unit. The full preview is available for a modest fee ($25); 8 CPDUs are awarded for completion.

To learn more about Investigative Searching 20/10, visit this link. (log in as guest)

From the tutorial on finding the author, here's a taste of what you'll get.

Find the first and last name of the author whose initials appear at the bottom of this article: http://www.spacestaking.com/about/mission/greenvision.html

As you will discover from the associated tutorials, there are several ways you might try to solve this: browsing, querying and truncation. One of these is the most effective given the circumstances. See if you can do it.



*Our NECC presentation was recorded for anyone interested who wasn't there--I'll post the link when Apple publishes it.

Finding the Black Box

Homing in on information online is similar to locating a downed aircraft's black box in one respect: both send out signals. In the case of the black box, the signal gets stronger as you approach. In the case of information online, the contextual evidence (think clues) gets stronger.

Detecting those clues and interpreting them can be very difficult.

Homing in on information online depends on browsing for keyword evidence. Some people have a distinct advantage in this regard. Masterful browsers tend to have fairly developed language skills. This, coupled with a disposition to read carefully, puts them far ahead of otherwise skilled computer users. I know adults often feel inadequate when working with kids with lightning fast computer abilities. But adults, despite their technological shortcomings, tend to outperform prodigious children and teens when it comes to homing in on information.

I witnessed an example of this last evening when my wife pilot tested an assessment that Information Fluency is preparing to use with Center for Talent Development students at Northwestern University. Pat, not a computer user from birth, took about an hour to complete the 10 item pretest, most of which depends on careful reading and browsing--after all, investigative searching relies on being thorough and examining lots of clues. She only missed 2 items, which is typical of an adult who has mastered searching and evaluation. Middle school and high school students will take about half that time and miss the majority of the items.

The major difference boils down to reading and browsing carefully. Sure, there are other techniques in a skilled online investigator's toolkit, but careful reading and browsing can be used to solve most search challenges.

1. See every word as a clue It's really easy to overlook important keywords when skimming. The words and terms that matter most are typically nouns and numbers. Adjectives become important in detecting bias. Spotting these is easier by slowing down. If you have the attitude that words are clues--not just the specific keyword(s) you are looking for--you will probably have to adjust your speed.

2. Recognize and follow possible connections Here is where terms you weren't looking for become important and why it helps to have a good vocabulary. Synonyms and words used in the context of what you are looking for are all possible connections. It may help to think about what some of these other words are before reading. In terms of the Digital Information Fluency Model, this is known as finding better keywords as you search. You can't predict with 100% accuracy what keywords are necessary before you search. You have to pay attention and find them as you search.

An example may help Let's say you're looking for the publisher of this site: http://www.spacetoday.net. First steps are usually to read the header and footer for the name of an organization or copyright information. Not seeing those, what stands out is the About Us link. Let the browsing begin. The name of the publisher is in the text of this page. In this case, it's an individual: "Spacetoday.net was founded by Jeff Foust." This wasn't a particularly difficult challenge, but many students will miss it because the name is embedded in the text. Moreover, Mr. Foust's role is represented with the word "founded" rather than "published." You have to know what these words mean, how they are used and related. Is the founder always the publisher? Not always. To make sure we've got the publisher, another source of information is required.

Here's your Challenge: Can you find another source of information that confirms that Jeff Foust is actually the publisher? It might take another technique in addition to careful reading and browsing.

When do you learn evaluation?

Had a brief conversation yesterday with Don Tapscott, author of Growing up Digital, Wikinomics and Grown Up Digital. He uses a number of terms to characterize the digital generation. The one that puzzled me was skeptical. So I asked him if his research really shows that students are thinking about the information they find on the Web. After all, this generation has been described by others as very much the opposite: to them all information is equal.

He admitted that younger students (e.g., 13 year olds) weren't particularly skeptical but by college they have to be otherwise they'd be ridiculous. (I think his point was they have to be skeptical.) We were interrupted at that point by autograph seekers and the conversation shifted to other topics, but what I wanted to say is that many (the majority?) of students don't practice these skills when they enter college, which is why I've had plenty of discussions with college librarians wondering what they can do to quickly orient freshmen to the world of information fluency.

So when do you learn to be skeptical, and more importantly, decisive about information found online? It probably happens in fits and starts and is different for everyone--mainly because it is not systematically addressed. Few schools actually teach what I call information forensics, in which investigative search skills are applied to found information for the purpose of determining credibility. So it's left to the digital generation to discover on their own and in their own time the lesson that not all information is equal.

One of the teachable moments during which skepticism is developed happens when obviously false information is encountered. This is the real benefit of hoax sites. They stand out as red flags against a monochrome backdrop of information that has credibility by virtue of publication. Not all hoax sites are so obvious, but sites like Snopes and the Museum of Hoaxes do a pretty good job of cataloging pages that are intentional spoofs or malicious deceits. Hoax and hoax-related sites perform a valuable service by illustrating that information has more than one purpose (to be believed). There are parallels: students don't believe jokes or tall tales to be true once they know their purpose. They learn to be skeptical based on the intent of the information and/or its believability. They just don't recognize this online much of the time.

I think it's obvious that students need to be exposed to jokes, tall tales and Web hoaxes if they are going to learn what makes information not equal. It will happen by accident given enough time online. It will happen much more efficiently if educators make it happen. And it should happen before the freshman year of college. Let's aim early, say, before 9th grade.

This is why I really like the concept of the Internet Search Challenge. They are puzzles to be solved. Most students respond well to the idea of a challenge. They are easy to use and are useful for demonstrating (and diagnosing) skills needed for information fluency. Dennis O'Connor and I have spent the last few months creating a new set of challenges, which we call an assessment playground. We are publishing what amounts to an elaborate hoax. The playground is a suite of sites and blogs that is partly fact and partly fiction. Some authors are legit, some are using aliases, the publishers are questionable, evidence found in articles and elsewhere must be fact-checked. Northwestern University's Center for Talent Development (CTD) will use this site, together with challenges (pretest), self-paced tutorials and another set of challenges (posttest) for all middle school and high school students enrolled in their summer programs. The CTD leadership understands the need to be intentional about engaging young people in becoming skeptical and helping them develop requisite skills.

We'll have more to say about the assessment playground in future blogs and how you can put it to use in your school.

In the meantime, I leave you without a search challenge this time, but hope you will join in and fill out the discussion about "when did you or when do the students you know learn to be skeptical about information online?"

Five Keys to Investigative Searching

When it comes right down to it, there aren't a lot of techniques involved in investigative searching. As opposed to speculative searching--what we tend to do most of the time--investigative searching involves clear clues and places to look for information about credibility.

In speculative searching, we're not sure where to look (Google is the default) and what keywords we need to retrieve the information we are looking for.

The goals of investigation are clear: find and evaluate the author, find and evaluate the publisher, find and interpret the meaning of the date, find evidence and evaluate it for accuracy, objectivity and external verification. You don't need to collect all this information unless the stakes for using information are high (like your job depends on it or you'll be punished if you plagiarize or violate fair use). Most of the time, checking a few facts is enough.

It may be helpful to think about investigative search techniques as depending on careful reading and four other keys. Nothing takes the place of reading thoughtfully. Still, the tendency of students is to go as fast as possible, thereby overlooking important information and clues about credibility. I'd say that's the biggest problem and why students trust false information.

The rest of the problem, as I see it, is that students aren't taught some basic techniques. These are the four keys I'm referring to:


Queries [Enter Key]

Using a search engine to track down missing information and check its credibility is the most powerful tool in the investigator’s kit. Depending on the information you need, the database you search will vary. For most queries, Google is preferred. But if you need to look up the registered owner of a website—to find the publisher or someone to contact—that requires a different database: whois.net. If you need to find a list of pages that link to the page you are investigating, then Yahoo is preferred, as it returns more information than Google with the link: command.

There are keyword queries, when you need to check the facts about a person, a publisher or something an author wrote in a web page. If you can’t find information to back up an author’s claim, that lends no support to its credibility. Sometimes, you’ll find that other people have information that contradicts the author you are investigating. This is all part of triangulating: finding information from multiple sources that agrees. When that happens, there’s a better chance the information can be trusted.

There are also string queries. This is when you copy a portion of text and search for the exact phrase in a database. You may place quotes around a passage, but it’s not necessary. Quotes work best around a first and last name or a short phrase. By searching for a string, you will often find other instances of the article or person you are investigating. The other instance may contain additional information you need. You may also discover that the passage you are investigating was plagiarized: copied word for word without being cited.


Truncation [Backspace]

The URL is an important source of information. It can reveal the publisher’s name or whether the site is self-published by the author. Truncation is a good way to navigate toward the root of the site you are investigating. Not all the information you need may be found on a web page, but a directory page may contain important clues such as an author’s name, a publisher or the date an article was written. You truncate by removing parts of the URL with the backspace or delete key, starting from the right and stopping when you reach a folder marker (/).


Browsing [Left Click]

Browsing is a special kind of reading: paying attention to hyperlinks. Normally when you browse you are looking for words or images that stand out. On a web page, browsing is defined by the links you click. Think of successful browsing as a game of HOT and COLD. You are trying to use links to get you closer to information you need, though you’re not exactly sure where that information is hiding. Whenever you click a link you have to scan or read the new page to discover whether you are getting hotter or colder. If you seem to be getting colder, go back and try a different link or technique. If you are getting hotter you will discover keywords and clues that you can use in your investigation.



Page Information [Right Click]

If you are using a browser that provides page information (Firefox does this) you can right-click on a web page to bring up a menu that includes Page Information. Depending on how the web page was coded, information about the last time the page was updated may be provided. If the last updated information is ‘now’ then the coding on the web page doesn’t allow this information to be shown. Knowing the last update of a page can be helpful in determining the age of the material.



This is obviously only an introduction to the techniques, but with them you should be able to solve this challenge:

Who is the author of the Sellafield Zoo? http://www.brookview.karoo.net/Sellafield_Zoo/

Challenge Level: Intermediate (Don't forget careful reading!)