Is Reading Believing?

Unfortunately for many students, reading is believing.

There could have been a time when information that appeared in print may have been trustworthy--thanks to editors and (hopefully) experts who reviewed it before it went to press--but that's no longer a safe assumption.

Because of the Internet, the reader has become the editor.

Before an evaluation can be made, one needs to know something about the author, publisher, references, or content; probably a combination of these things.

So here's an author challenge that's appropriate for students. Look at this blog: The Future is Green. The content is about sustainability, energy, the green movement, etc. But can the views expressed be believed? Knowing some facts about the author (and possibly checking out those facts) provides some perspective on the author's words.
Try this or have your students try it:

1. Locate the author's name (not hard)
2. Find information about the author (requires browsing)
3. Determine if the author knows what he is talking about, based on his experience, education, associations, accomplishments, etc.
4. Fact-check information about the author. Having another source (other than the author) confirm what the author says about himself is always a good idea.

So, should the author be believed? Why or why not?

Yes, There's a Need

Take a good look at this chart. What do you see? A normal distribution with a negative skew? A test that's too hard? A need for improvement?

This chart depicts the performance of high school students attempting to apply investigative techniques to Web pages. Not too good, is it? The average score is 52% That means the vast majority of students in this sample (n = over 350) really struggle with the ability to find and evaluate online information to determine its credibility. The average for middle school students is 45%. Very few are at 80%--what we would consider mastery. (By the way, these are gifted students. The top 1-2%.)

If information fluency was a typical school subject, the majority of students would fail. Fortunately for them, this is not a typical school subject; it is neither widely addressed nor seldom practiced.

The result above is sampled from a new assessment we developed at 21cif. The results aren't surprising. These are the same results we've been getting for years. The computer-savvy generation consistently under-performs when asked to locate and evaluate information found on the Internet. They have real difficulty finding authors, publishers, dates of publication, salient facts and claims and references of information on web pages and blogs. Of course if they can't find those things, they have an even harder time evaluating their credibility.

After a few hours of targeted training in investigative techniques, students improve by 10% on average. With more practice, their gains would be even greater. By the way, adults tend to score 10-15% points higher than high school students. Not because they are better with computers; they take the time to read critically. They have better vocabulary skills. Despite all the new age bells and whistles, search mastery still comes down to careful reading and thinking.

The need for information fluency is why we're in this business. Research merely informs what training is needed. Having an opportunity to train students is the challenge of this business. If we continue to say there are more important things to study, we will continue to see the results above. A generation that cannot research proficiently online will lack the ability to use information resources wisely and profit from them.

Preachy, I admit, but what else can one say about the state of information fluency in schools today? It needs work.

If you would like to learn more about the assessment, I'll share more about it over the next few blogs.

Five Keys to Investigative Searching

When it comes right down to it, there aren't a lot of techniques involved in investigative searching. As opposed to speculative searching--what we tend to do most of the time--investigative searching involves clear clues and places to look for information about credibility.

In speculative searching, we're not sure where to look (Google is the default) and what keywords we need to retrieve the information we are looking for.

The goals of investigation are clear: find and evaluate the author, find and evaluate the publisher, find and interpret the meaning of the date, find evidence and evaluate it for accuracy, objectivity and external verification. You don't need to collect all this information unless the stakes for using information are high (like your job depends on it or you'll be punished if you plagiarize or violate fair use). Most of the time, checking a few facts is enough.

It may be helpful to think about investigative search techniques as depending on careful reading and four other keys. Nothing takes the place of reading thoughtfully. Still, the tendency of students is to go as fast as possible, thereby overlooking important information and clues about credibility. I'd say that's the biggest problem and why students trust false information.

The rest of the problem, as I see it, is that students aren't taught some basic techniques. These are the four keys I'm referring to:


Queries [Enter Key]

Using a search engine to track down missing information and check its credibility is the most powerful tool in the investigator’s kit. Depending on the information you need, the database you search will vary. For most queries, Google is preferred. But if you need to look up the registered owner of a website—to find the publisher or someone to contact—that requires a different database: whois.net. If you need to find a list of pages that link to the page you are investigating, then Yahoo is preferred, as it returns more information than Google with the link: command.

There are keyword queries, when you need to check the facts about a person, a publisher or something an author wrote in a web page. If you can’t find information to back up an author’s claim, that lends no support to its credibility. Sometimes, you’ll find that other people have information that contradicts the author you are investigating. This is all part of triangulating: finding information from multiple sources that agrees. When that happens, there’s a better chance the information can be trusted.

There are also string queries. This is when you copy a portion of text and search for the exact phrase in a database. You may place quotes around a passage, but it’s not necessary. Quotes work best around a first and last name or a short phrase. By searching for a string, you will often find other instances of the article or person you are investigating. The other instance may contain additional information you need. You may also discover that the passage you are investigating was plagiarized: copied word for word without being cited.


Truncation [Backspace]

The URL is an important source of information. It can reveal the publisher’s name or whether the site is self-published by the author. Truncation is a good way to navigate toward the root of the site you are investigating. Not all the information you need may be found on a web page, but a directory page may contain important clues such as an author’s name, a publisher or the date an article was written. You truncate by removing parts of the URL with the backspace or delete key, starting from the right and stopping when you reach a folder marker (/).


Browsing [Left Click]

Browsing is a special kind of reading: paying attention to hyperlinks. Normally when you browse you are looking for words or images that stand out. On a web page, browsing is defined by the links you click. Think of successful browsing as a game of HOT and COLD. You are trying to use links to get you closer to information you need, though you’re not exactly sure where that information is hiding. Whenever you click a link you have to scan or read the new page to discover whether you are getting hotter or colder. If you seem to be getting colder, go back and try a different link or technique. If you are getting hotter you will discover keywords and clues that you can use in your investigation.



Page Information [Right Click]

If you are using a browser that provides page information (Firefox does this) you can right-click on a web page to bring up a menu that includes Page Information. Depending on how the web page was coded, information about the last time the page was updated may be provided. If the last updated information is ‘now’ then the coding on the web page doesn’t allow this information to be shown. Knowing the last update of a page can be helpful in determining the age of the material.



This is obviously only an introduction to the techniques, but with them you should be able to solve this challenge:

Who is the author of the Sellafield Zoo? http://www.brookview.karoo.net/Sellafield_Zoo/

Challenge Level: Intermediate (Don't forget careful reading!)

The Trivial Nature of Search Challenges

If you google internet search challenge, the top results are published by 21cif, including this blog. But there are other challenges out there:

Internet Search Challenge

Kim Bauman put together 10 questions that can be answered using a search engine, such as:

1. Define garrulous.
2. Who stated that The real problem is not whether machines think but whether men do?
3. Where is Mosi-oa-Tunya located?

The answers are on the site and may be used as a quick set of challenges using Google.

Jr. High Internet Search Challenge

St. Marys Schools (OH) published this 73-item pencil and paper trivia information challenge with items such as:

1. Find a copycat recipe for Outback Steakhouse’s Bloomin’ Onion
2. Who was the voice of Darth Vadar in the Star Wars movies?
3. Who is the current head coach of the Arena Football League’s Columbus team?

They link to it as a Junior High Search Challenge, although the url or the pdf doesn't describe it that way. Searching for 73 items would take a pretty good block of time, but it's possible to select fewer items. No search key provided.

Internet Search Challenge

This one or two player game published by Boise State University has automated features, like dice to select the type of search tool to be used, and timed search challenges. The objective is to find the correct answer to the question before the hourglass runs out (there's also a stop watch function). Sample questions include:

1. What was the name for one of the first computers invented in 1946 by J. Presper Eckert and John Mauchly at the University of Pennsylvania?
   Bonus: How much space did it take up?
2. What is the name for a number followed by 100 zeros.
   Bonus: How do you write this number with exponents?

Internet Search Challenge

Tom Sloan at University of West Virginia has posted a search exercise (9 items) for his English 102 course, including questions like:

1. What is the real name for Method Man of Wu-Tang Clan?
2. What is the meaning of the Welsh word "cymru?"

The directions for this exercise include explaining how the answer was obtained.

As this sampling makes clear, the task in most search challenges is to locate obscure information, usually intended to be of interest to the searcher. In addition, the task may be to use a specific type of tool in the process and to keep track of one's search to explain how an answer was found (a difficult thing to remember, actually). The task in all these is what I consider 'speculative searching': you don't know for certain what words to use or where to look.

There are few 'investigative searches,' yet this is where students need the most help. They tend to accept information uncritically; they don't often have investigative questions in mind when they search. So let me leave you with one: what is the author's real reason for producing the site: Save the Pacific Northwest Tree Octopus? It's a hoax site, but why? Why does the author invest the energy in keeping the site fresh (and misleading)?

This is a deeper type of search challenge and requires more thought than answering trivia. In fact, the answer is not known. It's a real challenge. If you or your students want to tackle the question--what motivates the author of the Tree Octopus site?-- feel free to add your comments here.

Sometimes it is Too Good to be True

The list of clients whose money disappeared in the Bernie Madoff scheme was recently published. Thousands of investors, including Madoff's lawyer, are listed in the document. It's an overwhelming case of misplaced trust, the implications of which are bound to ripple extensively for some time to come.

If I'd had money to invest, I might have been convinced to do so merely on the basis of who else was already committed: Madison National Life Insurance Company, Marder Sosnick & Co, Marion Madoff, Mark Madoff, to name just a few from the M's in the list. I would have been in very good company with a lot of savvy speculators.

This illustrates the risk in trusting even someone you know. However, there were probably signs--I heard on NPR that the account statements Madoff issued made little sense. Still, he was such a solid commodity, what did that matter?

In hindsight, this not only has something to do with myopic greed, it's a problem with evaluation: putting too much stock in one source of information.

I've argued in a previous blog or two that becoming connected with a trusted individual's online network is one way to vet unknown authors with whom your contact is connected. Clearly this may not be enough to avoid the mistakes seen in the Madoff story. Evaluating other evidence remains critical, such as the clarity of statements.

In addition to investigating the author, publisher, links (all of which would have been a thumbs up in Madoff's case) and evidence (those monthly reports were problematic), I'd like to add one more: personal judgment. Don't discount intuitions when it comes to evaluation. If you've read Malcolm Gladwell's book "Blink," you'll realize how accurate first impressions often are. Snap judgments are informed by momentary clues and many times they turn out to be right. Cloudy monthly statements probably set off a lot of momentary judgments. Apparently these weren't enough.

When overlooking a 'bad feeling' about a situation, the stakes need to be considered. Most of the time we're involved in taking low stakes risks: the consequences of including information from a discredited source in one's school work are not life-threatening nor will it send creditors to your door. But there are times when you may be involved in a very high stakes gamble. In that case, it really does matter if you consider all the evidence. The higher the stakes, the more evidence you need--and not all the same type of evidence either (e.g., don't rely solely on link to results in determining the value of a source when your health relies on it).

So let me leave you with an evaluation challenge. A letter to Richard Branson complaining about the food on a Virgin Airways flight from India has been circulating for the last several weeks. Read it. Read the blog replies. What intuition do you have about this? Is the letter sincere or is it a marketing device?

Link To Evaluation

Knowing who links to a site can be very revealing.

If trustworthy people link to a site and say positive things about it, does that site gain in credibility? I think most people would say yes.

What if they just link to the site and say nothing about it? That's harder to evaluate.

Using the link: command by itself is not an evaluation shortcut. For example, pages that link to http://golfcross.com/ present ambiguous results. Google returns 4 pages: two of them are about hoax sites, another is an account of someone playing golfcross and the fourth is the 21cif website. Since three fourths of the pages seem to indicate there is some suspicious about golfcross, one might be tempted to conclude the sport is really a hoax. But that's not an accurate conclusion.

One of the problems is that Google no longer returns all the pages that link to a page. Within the last year, only a sample of the pages is returned. Try the link: search in Yahoo and you get
lots more (over 400), if you select the option for picking pages from the entire site, not just the home page. That's a good reason to try more than one database when searching.

Hoax-related pages still show up in Yahoo results, but now there are others: travel sites, blogs, wikipedia, and so on. Now it isn't so easy to conclude that the sport is a hoax.

It still requires reading and interpreting the pages that have a link to golfcross to figure out why there's a link there. Always ask: why did this author include the link?

Among the Yahoo results is a blog by Bernie DeKoven. The context of the page is all about fun and games, including wallyball, slamball and this game played with egg-shaped balls. A link to Bernie DeKoven leaves the impression that he is educated, was a teacher, is an author and has pursued game-playing as a serious pastime for years. He seems to be an expert in games. So does his testimony convince you that golfcross is real?

Maybe you know Bernie and maybe you don't. If you do, does his testimony persuade you that golfcross is legitimate? Does he say strong enough things about the sport?

If you could get to know Bernie and ask him why he thinks golfcross is legitimate, that might help. That's where Web 2.0 becomes very valuable. You can ask questions, join personal networks and get a lot more information than if you were just observing from a distance. Of course this takes a little more time than lurking, but it yields better information.

So, what do you think about golfcross?

An opportunity to learn more about Web 2.0 tools and evaluate will start on Feb 9. Consider joining our 4 week course on Power Searching in a Web 2.0 World. Here's a link to look at the course.

The Necessity of Evaluation: The Case of United Airlines Stock

When readers took action on the news last week that United Airlines had filed for bankruptcy, the company's stock literally took a nose dive. Within a period of minutes, the stock fell to 25% of its value. Trading was stopped to prevent further loss. All because of inaccurate information that investors failed to evaluate.

In the fast-paced world of trading, not acting right away may have dire consequences, so taking time to fact check the accuracy of the news might be costly. As it happens, not checking the facts proved costly to those who sold low.

Here's how selling low may have been avoided without taking a lot of time to evaluate. A basic investigatory strategy involves checking the credibility of the source and something significant about the content.

In this case, when the story was picked up by Bloomberg, a source many investors trust, the sell-off began. But Bloomberg was not the original source. That distinction goes to the Florida Sentinel Sun which ran the 6-year old story on the business page of its web site. Going to the source would be the first place to investigate.

Reportedly, the story did not have a time stamp, which would be a red flag. When Google's bot indexed the page early on the morning of Sept. 7, it was given a time stamp which resulted in some of the confusion. It's important to know that the date stamp that appears at the beginning of some Google snippets is triggered by when the page was crawled, not authored. In most cases, the dates agree, but since Google doesn't actually read the stories it crawls, the automatic vs. the actual date of the article were off by six years. So it pays to treat the crawl date with a grain of suspicion, although this may be an example of a rare disconnect.

Fact-checking can be as simple as googling a name, a fact or a claim. This is particularly effective in the case of evaluating the credibility of a site such as the Pacific Northwest Tree Octopus, which contains many fictitious facts. The Google results often come back empty or only link to the same site. In the United Bankruptcy case, copying some text and using it as a query can be revealing. The results of just googling the title of the article, "United Airlines Files for Bankruptcy" produces only 2,400 results. A quick look at several of them (BBC, PBS, CBC, etc.) reveals the date: Monday, Dec. 9, 2002.

If you need more, it's always wise to consider: "Who would really know the answer to this question?" A couple of authoritative sources come to mind: United Airlines or a bankruptcy filing database (e.g., bankruptcy.com). If neither site can confirm the news, something is amiss: exercise caution.

Even if it takes a couple of minutes, does it pay to check online stories? Ask any of the United investors.

Learn more about Web site evaluation by enrolling in our newest course: WSI, Web Site Investigator. It's only $68 and a good investment!

Tracking down the Loose Ends

Does this ever happen to you? You've located a relevant resource that you want to cite, but when you look for the name of the author--or maybe the date of publication--it's not there. Now what do you do?

Without information about the author, you have to rely on the reputation of the publisher. Without the publication date, you risk using information that is no longer current. The standard style guidelines (APA, MLA, etc.) allow for citing works without an author and/or a date, but this is risky. You really should try to find the missing information to be sure your source and content is credible.

Investigative searching is your best bet to track down elusive information. Start with the page you want to cite. Scour it for clues. An author's name or date is not always at the top or bottom of the page. Like most crime scene investigations, you don't want to look elsewhere until you have to. Careful reading is required.

If the page you are on turns up nothing, then you have to expand your search. Now the url and links on the page become important. Try truncating the url to navigate to pages closer to the root of the site. In the process you may find a directory that lists articles, including the one you want, including author and/or date information. Links on the page may do the same thing. Don't leave the site unless you have to. Again, look carefully for clues.

Following links (browsing) is a particularly challenging form of searching. If you get more than one link beyond a place where you detected any relevance to your search, back up and try another promising path.

If the site has a search engine, try entering the name of the article or any significant keywords from it. You may be surprised to find other references to the article on the site.

If you reach the point where the site is no help at all, conduct a broader search using a major search engine (Yahoo, Google, etc.). Query the name of the article or significant keywords from it. Articles often appear in more than one location on the Internet. You could find reviews or references to your loose ends.

If you've tried these options and still don't have a name or a date, decide if you should cite the work without them. Is the information good enough to stand on its own? What would be the consequences if this information were false?

To test your skills at tracking down an elusive author, try this challenge:

5 Essential SEO Techniques (Article is halfway down the page)

Whom would you cite? Look for the answer in my next post!

If you'd like an opportunity to sharpen your investigative skills, a new section of Web Site Investigator (WSI) starts August 11. > More information

Embedded Evidence, External Evidence

Over the weekend I created two new tutorial resources for the Website Investigator series (WSI): Accuracy and Evidence. In addition to knowing who is the author and/or publisher and when it was written or published, finding embedded evidence and external evidence can be very important in verifying the credibility of the source and the content.

In the simplest terms, credibility depends on source and content. Information about the author and publisher helps to define the source--where did these ideas originate? Is the author recognized as an expert? Does this publisher submit works to careful review before posting them? Embedded and external evidence helps to define the content--how are words used (signs of objectivity or bias)? When was this written? Who links to it? What do experts say about the content? A questionable source may produce brilliant content and a trusted author may produce flawed content--so it's important to check both before accepting information at face value.

As educators know, the majority of students today tend to accept information at face value. Somehow, merely finding information feels sufficient. Investigating it is unimportant.

To encourage investigation, students need to be shown and practice a few basic techniques. These are not hard to learn and don't take much time. Compared to searching (which I now call speculative searching--when you don't know exactly what words to search with and where to look), investigative searching is much more precise: the keywords are clues embedded in the information and the places (databases) to search are well-defined.

Think of embedded evidence as clues in the text, the url and metadata. These clues can be used to investigate the accuracy of information and often lead to external sources that have already done an evaluation.

The new Accuracy tutorial focuses on three areas: Finding powerful clues embedded in a Web page, checking the evidence by doing a secondary search and triangulating, checking what three different sources have to say about the information.

The companion Evidence tutorial emphasizes using queries to find external evidence, checking whether pages that link to the information support it or contradict it and triangulating information sources (examples that are different from the Accuracy module).

These tutorials are geared for middle schoolers through adults. There's increasing demand for similar activities aimed at elementary grade students, and that's my next project.