This repository contains a list of papers, datasets and leaderboards of the table reasoning task based on the Large Language Models (LLMs), which is carefully and comprehensively organized. If you found any error, please open an issue or pull request.

For more details, please refer to the paper: A Survey of Table Reasoning with Large Language Models, the overview of which is shown in the figure below.

In a table reasoning task, the inputs to the model include the table, optionally a text description of the table, and the user question that corresponds to variable tasks (e.g., table QA, table fact verification, table-to-text, and text-to-SQL), and the outputs are the answers of the task. Recent research has shown that LLMs exhibit compelling performance across NLP tasks, in particular, the ability of in-context learning without large-scale data fine-tuning dramatically reduces annotation requirements, which we call the LLM era. Considering the high annotation and training overheads of table reasoning, there has been a lot of work on applying LLMs to table reasoning tasks to reduce the overheads, which has become the current mainstream method.

In this part, we present leadboards of currect mainstream benchmarks of table reasoning with LLMs. Each benchmark is ordered by the performance. Type denotes the reasoning types:

• PLM-SOTA: the best performance of small-scale PLMs;
• LLM-fine-tuned: fine-tuning LLMs;
• LLM-few-shot: inference using LLMs with few-shot.

WikiTableQuestions serves as the initial benchmark in the table QA task, which has open-domain tables accompanied by complex questions.

TabFact, as the first benchmark in the table fact verification task, features large-scale cross-domain table data and complex reasoning requirements.

FeTaQA requires the model to generate a free-form answer to the question, with large-scale and high-quality data.

Spider is the first multi-domain, multi-table benchmark on the text-to-SQL task.

If you find our survey helpful, please cite as following:

@misc{zhang2024survey,
      title={A Survey of Table Reasoning with Large Language Models}, 
      author={Xuanliang Zhang and Dingzirui Wang and Longxu Dou and Qingfu Zhu and Wanxiang Che},
      year={2024},
      eprint={2402.08259},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}