Tabular data embedded within PDF files, web pages, and other document formats are widely used in various sectors, such as government, engineering, science, and business. These tabular datasets, known as human-centric tables (HCTs), have unique characteristics that make them valuable for deriving critical insights. However, their complex layouts and limited operational power at scale pose significant challenges for traditional data extraction, processing, and querying methods.
Current solutions in the field primarily aim to transform these tables into relational formats for SQL queries. While this approach has been helpful to some extent, it falls short when dealing with the diverse and complex layouts of HCTs. Consequently, querying such tables becomes a challenging task.
To address this challenge, the authors of this paper introduce HCT-QA, an extensive benchmark specifically designed to evaluate HCTs, natural language queries, and their corresponding answers. The benchmark dataset consists of 2,188 real-world HCTs along with 9,835 question-answer (QA) pairs. Additionally, the dataset includes 4,679 synthetic tables with 67.5K QA pairs.
While HCTs can potentially be processed by different types of query engines, this paper primarily focuses on assessing the capabilities of Large Language Models (LLMs) as potential engines for processing and querying such tables. LLMs, such as GPT-3, have shown remarkable advancements in natural language processing tasks and have the potential to handle the challenges presented by HCTs.
The HCT-QA benchmark provides an opportunity to evaluate the performance of LLMs in processing and querying complex HCTs. By assessing their ability to answer a wide range of questions posed in natural language, researchers can gain insights into the strengths and limitations of LLMs in this context. This analysis can inform the development of novel techniques and approaches that harness the power of LLMs to effectively process and query HCTs.
In conclusion, the HCT-QA benchmark and the focus on Large Language Models present an exciting avenue for advancing the field of tabular data processing and querying. By addressing the challenges posed by complex HCT layouts, researchers can unlock new possibilities for deriving insights from tabular data in various domains.