AutoGraphQL turns a GraphQL schema into executable training and evaluation data, then supports model adaptation and cross-schema benchmarking end to end.
Scores from Tables 3–5 on the 6,403-instance public benchmark. The open-weight models are shown before and after fine-tuning on 9,118 AutoGraphQL training instances; Claude Sonnet 4.5 is the paper's frontier-model reference.
The corpus spans 173 schemas and seven query categories. The 9,118-instance training set and the 6,403-instance public benchmark are released under the MIT License at nl2graphql-bench/datasets. The 2,100-instance hidden benchmark is not released: it is what we score submissions against, so if it were public a model could be tuned on it and the score would mean nothing.
Files are gzipped: 310 MB and 388 MB uncompressed. Use the CleanSchema field for validation; see the dataset README for all fields.
Query categories per set (training / public benchmark):
The 173 source schemas span seven application domains, giving a realistic spread of entity types and query patterns rather than one narrow vertical.
Two paper runs can post similar overall scores for very different reasons. Pick up to four to overlay their category profiles.
Pick up to 4 · profiles are shown as % accuracy per category
Enterprise GraphQL APIs are schema-specific: each organization exposes its own types, fields, arguments, filters, and mutations. A model that works on one API may fail on the next. AutoGraphQL is designed around that reality: bring a schema, generate validated data for it, adapt a model, and evaluate whether the model generalizes to schemas it did not see in training.
Each instance is a validated triple — (schema, natural-language query, ground-truth GraphQL
query) — synthesized through schema-aware query generation, then checked for both syntactic
validity and successful execution before it's admitted to the corpus.
GraphQL is an open-source query language and runtime for APIs. Every service defines its own typed schema, so a generated query must use the fields, arguments, types, and relationships that the target API actually exposes. This schema dependence is the central challenge that AutoGraphQL measures.
An initial pipeline for translating natural-language requests into GraphQL queries.
The first large training and evaluation resource in this research program.
Schema structure and natural-language similarity guide the selection of demonstrations.
RGEval evaluates semantic equivalence beyond exact string matching.
A preliminary study covering 1,845 instances across 20 schemas.
The unified framework scales the resource to 17,621 instances across 173 schemas.
An illustrative instance from the corpus — a university-enrollment schema with a natural-language request that requires filtering, pagination, an alias, a conditional directive, a fragment, and a nested traversal all at once.
Papers each person contributed to, from the research lineage above.
@inproceedings{battu-etal-2026-autographql,
title = {{AutoGraphQL}: A Bring-Your-Own-Schema Framework for
Adapting Open-Weight {LLMs} to {GraphQL} {APIs}},
author = {Battu, Keya and Mehta, Parth and
Balagopal, Harikrishnan and Kesarwani, Manish},
editor = {Atanasova, Pepa and Haffari, Gholamreza and
Wang, Minghan},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods
in Natural Language Processing: Industry Track},
month = {oct},
year = {2026},
address = {Budapest, Hungary},
publisher = {Association for Computational Linguistics},
url = {https://aclanthology.org/2026.emnlp-industry.215/},
doi = {10.18653/2026.emnlp-industry.215},
pages = {3602--3613},
isbn = {979-8-89176-460-6}
}Recipient of an EMNLP 2026 Industry Track Best Paper Award. Official ACL Anthology record: aclanthology.org/2026.emnlp-industry.215.