Beyond Edge Addition: A Dataset for
Information Extraction Incorporating New
Instances, Types, and Relations

Information extraction (IE) is the task of converting natural language text into structured triples comprising a subject, predicate, and object. Existing IE datasets often operate under the assumption that all entities (instances, properties, and classes) are already defined within a knowledge graph (KG), focusing solely on discovering the relationships between them. However, this assumption does not align with real-world scenarios, where many entities and relationships may be missing in the KG. Additionally, most current datasets do not provide a snapshot of the accompanying knowledge graph, leading to inconsistencies in evaluation,
as different systems may rely on different KG versions with varying degrees of completeness and labelling support. Such inconsistencies undermine fair benchmarking and reproducibility. In this paper, we introduce a novel information extraction dataset specifically designed to better reflect realistic KG incompleteness. Our dataset includes 20% missing classes and instances, along with 5% missing relations, requiring systems to not only add new links (edges) but also propose new instances, classes,
and relations. To ensure reproducibility and prevent leakage from pre-trained language models, we provide a heavily modified version of Wikidata where background knowledge cannot be exploited to trivially infer triples. This resource supports a more robust and comparable evaluation of IE systems in settings closer to real-world applications. We further present a strong baseline that employs large language models for extraction and disambiguation tasks, as well as encoder-based retrieval, to
integrate the background knowledge graph. It operates in several iterations, initially identifying a first set of triples, then progressively refining them by referencing the KG and generating new entities if necessary.

Dataset Description

There are two gzipped files uploaded. The first one contains the knowledge graph and the second one the actual dataset with the text and corresponding triples that should be extracted.

The files are password protected such that the dataset is not easily included in future LLM training.

The password can be found in the corresponding Github repository.