Aw nuts. It’s sorry enough that we lean on computers to research and write stories.
It’s a given that much information is dredged from websites.
Now, we’re on to robots for fact-checking?
Sure, says Bill Adair, the Duke University professor who earlier helped birth the PolitiFact fact-checking operation at the Tampa Bay Times and in states around the country. The Austin American-Statesman’s PolitiFact Texas is among seven state-level PolitiFact franchises established nationally since 2010. All this happened after FactCheck.org debuted at the University of Pennsylvania and while The Washington Post launched its Pinnochio-popping FactChecker.
Globally, from 2014 to 2015, formal fact-checking efforts doubled to more than 60 projects, Adair reports. An international fact-checking group held its second gathering in London this summer.
In June, meantime, researchers at Indiana University revealed they’d devised a computer program to assign “truth scores” to excerpts from Wikipedia, the crowd-sourced online encyclopedia. Upshot: The computational fact-checker did well, consistently matching the assessment of human fact-checkers in certitude about the accuracy of statements.
A news release quoted Giovanni Luca Ciampaglia, a postdoctoral fellow at the Center for Complex Networks and Systems Research in the IU Bloomington School of Informatics and Computing, summing up: “We live in an age of information overload, including abundant misinformation, unsubstantiated rumors and conspiracy theories whose volume threatens to overwhelm journalists and the public. Our experiments point to methods to abstract the vital and complex human task of fact-checking into a network analysis problem, which is easy to solve computationally.”
Filippo Menczer, director of the Center for Complex Networks and Systems Research and a co-author on the study, said: “With increasing reliance on the Internet as a source of information, we need tools to deal with the misinformation that reaches us every day. Computational fact-checkers could become part of the solution to this problem.”
And now a University of Texas-Arlington team is getting closer to helping journalists determine which presidential debate statements warrant fact-checking. Still, I was kind of happy to hear, we’re not yet to the point of a computer churning out full-fledged fact checks.
Chengkai Li, a UT-Arlington associate professor, has enlisted university students and members of the public to hone his team’s compilation of potentially checkable claims made in the U.S. presidential debates from 1960 through 2012 as part of the ClaimBuster project, which Li describes as using “computational power to do tasks that are tedious and time-consuming for fact-checkers, such as finding claims by politicians that should be checked.”
By email, Li recently talked with the Statesman about the project.
Austin American-Statesman: How does the project work?
Participants … review transcripts from past presidential debates and identify sentences that should be fact-checked. These “labeled” sentences become training examples. Then, we use natural language processing (NLP) to extract features from the training examples. The features can include the words used in the sentences, the lengths of the sentences, the speaker’s attitude, and so on. We then use machine-learning algorithms to identify the characteristics of features in factual claims that are worth checking. The outcome of the machine-learning algorithm is called a model. With the help of this model, ClaimBuster examines any transcript to find sentences that should be fact-checked.
You’ve said your goal is to play a role in the 2016 presidential election. In what way?
During a live presidential debate, for each sentence coming from a candidate, ClaimBuster will give it a score which tells journalists and professional fact-checkers if the sentence should be fact-checked and how important it is. It will rank all the sentences by their scores. In this way, ClaimBuster will speed the process of finding important factual claims and leave the journalists with more time for assessing the validity of claims.
Can you envision a computer program that consistently evaluates factual claims?
The “Holy Grail” perhaps can never be completely accomplished.
There are mainly two fundamental challenges. One is to understand what one says. Computer scientists have made leaps and bounds in speech recognition and NLP and we now have wonderful tools such as [Apple’s] Siri in our hands. But these technologies are far from perfect.
The other lies in our capability of collecting sufficient evidence for checking facts. We are in the big-data era. A huge amount of useful data is accessible to us, and more is being made available at every second. But, what is being recorded is still tiny compared to the vast amount of information the universe holds. We will never be able to capture and record all such information.
A program can check with ease simple statements on related facts with no subtle implications, such as “the daily highest temperature in Arlington, Texas has been over 100 for 7 consecutive days.”
Of course, significant claims checked by professionals are usually complex and subtle. Even if a claim checks out at the surface level, it can be [rated] “Pants on Fire.” For instance, Mitt Romney said during a 2012 Republican presidential debate that “our Navy is smaller than it’s been since 1917.” To check this claim, first we need to figure out he was using number of military ships as the measure of U.S. Navy’s size. Furthermore, we need to understand he was suggesting that smaller Navy implies weaker Navy. Even though the U.S. Navy’s size in 2012 was indeed almost the smallest since 1917, it was hardly the weakest, since it makes little sense to compare modern battleships with those 100 years ago. For this and other reasons, PolitiFact gave his claim “Pants on Fire.”
Could a computer program eventually overtake human beings as fact-checkers?
Maybe, but not within many, many, many years.
We all have heard about the dream of artificial intelligence. We are much further than where we started merely 60 years ago. But it is still a dream.
“Traditional fact checking by expert journalists cannot keep up with the enormous volume of information that is now generated online. Computational fact checking may significantly enhance our ability to evaluate the veracity of dubious information.”
Abstract of research article, “Computational Fact Checking from Knowledge Networks,” PLoS ONE journal, June 17, 2015