HERTy-Wiki

HERTy-Wiki (Hierarchical Entity Reasoning and Typing in Wikidata) is a human-verified benchmark for evaluating hierarchical reasoning in large language models (LLMs) for knowledge graph entity typing (KGET) using entities derived from Wikidata.

The dataset contains 8,767 multiple-choice questions derived from 3,776 Wikidata entities across five domains, with an optional multimodal extension that includes associated images. Each question requires models to select the most specific valid type from a set of semantically related candidates within the same subclass hierarchy, reflecting realistic Wikidata editing scenarios where contextual information may be limited or incomplete.

Baseline experiments on several state-of-the-art LLMs show that models frequently rely on memorised priors rather than hierarchical reasoning, achieving only modest improvements over a no-context baseline. These results highlight that current models still face significant challenges in reasoning over hierarchical knowledge structures, particularly in specialised or evolving domains.

The dataset includes machine-readable metadata in MLCommons Croissant format and accompanying documentation in the included README.

Code for running the benchmark:
https://github.com/nobretincheva/HERTy-Wiki

The benchmark is introduced in the paper “HERTy-Wiki: A Benchmark for Hierarchical Entity Reasoning and Typing in Wikidata”, accepted at the European Semantic Web Conference (ESWC).