• Fact Checking 🎸 Rocks!  
  • Fact checking baseline combining dense retrieval and textual entailment
    • Idea
    • Presentation
    • System description
      • Indexing pipeline
      • Search pipeline
      • Explain using a LLM
    • Limits and possible improvements
    • Repository structure
    • Installation
      • Entailment Checker node
      • Fact Checking 🎸 Rocks!

💡 This project aims to show that a naive and simple baseline for fact checking can be built by combining dense retrieval and a textual entailment task. In a nutshell, the flow is as follows:

• the user enters a factual statement
• the relevant passages are retrieved from the knowledge base using dense retrieval
• the system computes the text entailment between each relevant passage and the statement, using a Natural Language Inference model
• the entailment scores are aggregated to produce a summary score.

• 🍿 Video presentation @ Berlin Buzzwords 2023
• 🧑‍🏫 Slides

🪄 This project is strongly based on 🔎 Haystack, an open source NLP framework that enables seamless use of Transformer models and LLMs to interact with your data. The main components of our system are an indexing pipeline and a search pipeline.

• Crawling: Crawl data from Wikipedia, starting from the page List of mainstream rock performers and using the python wrapper
• Indexing
  • preprocess the downloaded documents into chunks consisting of 2 sentences
  • chunks with less than 10 words are discarded, because not very informative
  • instantiate a FAISS Document store and store the passages on it
  • create embeddings for the passages, using a Sentence Transformer model and save them in FAISS. The retrieval task will involve asymmetric semantic search (statements to be verified are usually shorter than inherent passages), therefore I choose the model msmarco-distilbert-base-tas-b
  • save FAISS index.

• the user enters a factual statement
• compute the embedding of the user statement using the same Sentence Transformer used for indexing (msmarco-distilbert-base-tas-b)
• retrieve the K most relevant text passages stored in FAISS (along with their relevance scores)
• the following steps are performed using the EntailmentChecker, a custom Haystack node
• text entailment task: compute the text entailment between each text passage (premise) and the user statement (hypothesis), using a Natural Language Inference model (microsoft/deberta-v2-xlarge-mnli). For every text passage, we have 3 scores (summing to 1): entailment, contradiction and neutral.
• aggregate the text entailment scores: compute the weighted average of them, where the weight is the relevance score. Now it is possible to tell if the knowledge base confirms, is neutral or disproves the user statement.
• empirical consideration: if in the first N passages (N<K), there is strong evidence of entailment/contradiction (partial aggregate scores > 0.5), it is better not to consider (K-N) less relevant documents.

• if there is entailment or contradiction, prompt google/flan-t5-large, asking why the relevant textual passages entail/contradict the user statement.

✨ As mentioned, the current approach to fact checking is simple and naive. Some structural limits of this approach:

• there is no statement detection. In fact, the statement to be verified is chosen by the user. In real-world applications, this step is often necessary.
• Wikipedia is taken as a source of truth. Unfortunately, Wikipedia does not contain universal knowledge and there is no real guarantee that it is a source of truth. There are certainly very interesting approaches that view a snapshot of the entire web as an uncurated source of knowledge (see Facebook Research SPHERE).
• Several papers and even our experiments show a general effectiveness of dense retrieval in retrieving textual passages for evaluating the user statement. However, there may be cases in which the most useful textual passages for fact checking do not emerge from the simple semantic similarity with the statement to be verified.
• no organic evaluation was performed, but only manual experiments.

While keeping this simple approach, some improvements could be made:

• For reasons of simplicity and infrastructural limitations, the retrieval uses only a very small portion of the Wikipedia data (artists pages from the List of mainstream rock performers). With these few data available, in many cases the knowledge base remains neutral even with respect to statements about rock albums/songs. Certainly, fact checking quality could improve by expanding the knowledge base and possibly extending it to the entire Wikipedia.
• Both the retriever model and the Natural Language Inference model are general purpose models and have not been fine-tuned for our domain. Undoubtedly they can show better performance if fine-tuned in the rock music domain. Particularly, the retriever model might be adapted with low effort, using Generative Pseudo Labelling.

• Rock_fact_checker.py and pages folder: multi-page Streamlit web app
• app_utils folder: python modules used in the web app
• notebooks folder: Jupyter/Colab notebooks to get Wikipedia data and index the text passages (using Haystack)
• data folder: all necessary data, including original Wikipedia data, FAISS Index and prepared random statements

💻

If you want to build a similar system using the EntailmentChecker, I strongly suggest taking a look at the node repository. It can be easily installed with

pip install haystack-entailment-checker

To install this project locally, follow these steps:

• git clone https://github.com/anakin87/fact-checking-rocks
• cd fact-checking-rocks
• pip install -r requirements.txt

To run the web app, simply type: streamlit run Rock_fact_checker.py