Importance of Deliberate Info Lit in AI-Enhanced Research

If we think that Research is just about finding “the right answer," we are wrong.

I was asked by several colleagues this week, "you do archives, AI, OER, database research assistance, and all sorts of stuff. What would you say is the main thing you do?" I thought a bit and said, “information literacy.” I would say that ninety percent of my work is helping people navigate information literacy in different contexts and environments.

Two months ago, I wrote about the shortcomings of the new “god,” AGI. I discussed that contrary to popular belief, “AI Is Not Smart…” It “pretends” to know the answers to our questions because it has been trained to always respond. They do not “understand” the context (let along the language, as such) of our prompts. Instead, they “dream up” responses, as Andrej Karpathy says.

Even the most educated and trained of professionals can slip up when thinking about AI responses. One of my most respected colleagues in Idaho, when he determined that he had received and incorrect and unhelpful response from ChatGPT, actually asked the AI why it had done so. It took him a few minutes to think through that impulse and understand that it did not “know” at all why it had responded the way it did.

Last week, I wrote about the importance of avoiding “insta-research.” I discussed that all information resources have bias, that search engines have their own problems, that AI tools have multiple problems, and that combining all of these together only creates more problems. This means that relying or acting on answers from these resources without fact-checking is dangerous.

As I write this article about the importance of deliberate research, the less-than-one-day TikTok ban has come and gone. You would not believe how many people (students but also the general public) are lamenting the (very short) ban of TikTok, because "oh, there goes my best source of reliable information."

I have seen the informational channels in TikTok, similar to those on FB and YT. They are great, and provide excellent short-form introductions to various fields and ideas.

That having been said, if you are relying on a social media app for definitive answers to questions, you need to brush up on information literacy skills.

I said it once and I'll say it again: TikTok and AI are the two greatest threats to information literacy in our current world, potential beneficial use cases notwithstanding. There is a widespread propensity for the public to misuse these tools when conducting research with them. When I say "TikTok" I mean "short-form in general."

This whole experience underscores why deliberate information literacy practices are more important than ever. Studies have shown that reliance on generative AI leads to decreased critical thinking skills, but the internet and digital resources always had negative aspects when it came to information reliability.

Stanford’s Web Literacy Experiment

Nine years ago, researchers at Stanford, led by Sam Wineburg, examined the tendency among students to rely on Google for information at the expense of critical thinking. They discovered that students were unable to distinguish between credible and unreliable information sources.

The study examined not only Google, but also digital iterations of news articles. Wineburg also looked at social media posts, comments, and other aspects of interactions between social media users.

He and his researchers found several concerning facts about how users, particularly young users, interact with online information sources.

1. Users consistently mis-labelled sponsored advertisements as actual news stories.

2. Users acted on the assumption that since a “search result” occurred higher in a search engine’s results list, it was more reliable than resources lower in the list or on a later page. Northwest University also conducted a study regarding this.

3. Users usually looked only at the appearances of the content, including the “about” page and the rhetoric or details of the content. They did not look at the origins, affiliations, or surrounding information ecosystems. They did not engage in “lateral research.”

Sam Wineburg, it seems, can indeed handle the truth, and he is trying to help others do the same.

We, all of us, need that help in finding good resources and search engines, while helpful, can be misleading if you do not know how to sift through sources and find reliable information.

Artificial intelligence tools (especially with the advent of sponsored ads and AI overviews in search engines) can be great helps at finding information resources for which you have specifically asked. However, there are some issues with these tools as well.

Issues with Plain AI Tools and Research

I think this article is the simplest and most direct explanation of why 1. AI tools on their own can never truly help in research, and 2. what is wrong with common search methods.

The Problem with “Perfect” Answers: GenAI and Academic Research Tools

Here is another post by the same newsletter. I was about ready to throw digital hands thinking that AAL was talking about SearchGPT (more human-driven, and deliberately at the mercy of the user). But this takedown of the Google “AI Overview” (more on that later) is phenomenal.

AI Search Will Never Work – Really

It reminds me of the “Situating Search” paper by Bender and Shah, which I have discussed in the “Perils of Insta-Research” article above.

The Purposes of Research

There are at least 16 “information seeking strategies” and 20 “search intentions,” and as Bender and Shah write, the current versions of search engines are not fulfilling many of them. In “Situating Search,” they reference work by Nicholas Belkin, Colleen Cool, and others to determine specific categories of search goals and strategies.

Researchers use search engines and other tools to fulfill the following purposes:

1. Search (know what they want already) or Scan (perusing lists of items)

2. Select (using criteria to handpick specific sources) or Learn (discovering aspects of a topic or item)

3. Specify (recalling previously known information) or Recognize (identify new items through association with previously known items)

4. Analyze Information (examine an actual item) or Retrieve Meta-Information (look at data records about an information resource).

According to Bender and Shah, the two most frequently-used strategies are:

1. Scanning through a list of items to Select relevant items based on criteria, Recognizing that they are consistent with previously known information, and gleaning new Information.

2. Searching for desired items or information that is already known and Selecting those specific resources, Specifying them based on previous experience with those resources, and retrieving already-known Information.

When you examine the purposes and strategies in Bender and Shah’s paper (I heartily encourage you to take a look at the entire paper), you eventually come to the conclusion that:

1. Search engines are not fulfilling the needs of researchers, and

2. Large language models, unaltered and minimally directed, are not much better.

Thus, we have a great need to develop, or entrench, our information literacy skills. This is as true today, when new content is created every second, as it was twenty years ago, when YouTube was released, and as it was when the first book was written thousands of years ago.

Caulfield’s Take

Mike Caulfield, a colleague of Sam Wineburg who created the SIFT Method with him, discussed the same ideas from an information professional standpoint in a critique of a site that had co-opted his (open) idea of the SIFT Method. News Literacy Project called their periodic online newsletter “The Sift,” no doubt hoping to capitalize on Caulfield’s then-new idea.

To his credit, Caulfield said nothing against them for stealing the idea. However, he did have an issue with how they portrayed his ideas and how they were communicating about information literacy. Caulfield was focused on “web literacy,” the central ideas of which I think are good for all information literacy (he may disagree, technically). NLP, on the other hand, said that some of these ideas were contradictory to “news literacy.”

To Caulfield, sacrificing info lit for “ideal news consumption” was problematic (and I agree). In his remarks, he reiterated what he viewed as the purpose of consuming information resources: to get to the truth as quickly and as simply as possible.

For more information on the SIFT method and the purpose of research, I would encourage you to read Wineburg’s and Caulfield’s book Verified.

How Can We Use Web-Enhanced AI Thoughtfully In Research?

So, we know the SIFT method. We know how to make cogent and powerful arguments. Library professionals know how to curate representative and well-evidenced lists. Educators, instructors, and trainers know (and teach their students) how to find the best resources and precedents for their respective fields.

How do we apply these principles when interacting with web-enhanced AI? This means large language models connected to the internet, as well as YouTube algorithms, Facebook Meta AI, and the AI chatbots integrated with search engines.

1. For starters, we can stop paying attention to the AI “summaries” and “overviews” in search engine interfaces. I have major concerns with the fact that the AI text generated at the top of all Google searches has the header "AI Overview." This implies that the AI bot has actively reviewed the articles below and ferreted out the keywords and their context. It has not done so. It is in no way related.

2. Another way is to fact-check the AI output in a similar way to how we would fact-check news stories or social media posts: lateral research. Take the keywords and main arguments of the posts (and any comments that you would like to include in your sources) and search them in databases and search engines).

3. Focus on links to sources. If you are using a web-enhanced LLM, look at the Sources list, which is associated with a button or is below the response. If you are using a search engine that has an AI “summary” at the top, either completely ignore it or view the sources used to create the Overview. It’s all about the sources (or the purported sources).

4. Look at some resources you have found without the use of AI. What do your library databases have to offer? What do professional resources and academic journals in the field have to offer? What professionals and colleagues have you talked to in person or through other forms of personal communication? AI and online resources are not the only worthwhile materials! Also, don’t forget printed books, textbooks, and records of other types (especially for history research).

5. If you put a document or link into a prompt and ask an AI tool to summarize it, please make sure that you verify that the keywords, subjects, or quotes (especially quotes) in the AI outputs are actually present in the document. This is especially important if you choose to copy-paste from an AI output (which I do not recommend at all, read “Deliberate Creation,”).

6. My last suggestion would be to use AI in non-acquisition ways. Use it to brainstorm about potential research topics. Use it to look at ways that different topics intersect. Then, you can use it to find resources if you want to. But, again, remember to use information literacy.

After I named this article, I realized that I included the word “deliberate” in the title. I looked into it, and it looks like I have used that word in about 1/4 of my articles. Maybe I should rename my blog “Deliberate InfoLit.” I hope that we are all deliberate, cautious, and responsible in how we research, with whatever technology we use.