Tool use over parsing: getting structured data out of an LLM in production

TL;DR: If you need fields out of messy text in production, stop asking the model to "return JSON" and parsing what comes back. Declare the output as a JSON Schema and let the provider enforce it, through tool use (function calling) or a strict structured-output mode, so a malformed shape cannot be generated in the first place. Then treat the schema as the floor: it guarantees shape, not truth, so a thin layer of deterministic code still normalizes values, clamps ranges, and refuses to let the model overwrite data you already know. Put the provider behind one small interface so that switching vendors, or moving to a cloud-hosted endpoint, is one new adapter instead of a refactor.

The problem: messy text in, database columns out

A lot of the AI in the products I run is unglamorous extraction. A long, inconsistent document goes in: a job posting, a profile, a web page. A row with typed columns comes out: a category from a fixed list, a seniority level, a list of technologies, a salary range, a work model, a handful of nullable fields. Nobody reads the model output directly. It lands in a database, feeds filters and matching, and renders on public pages.

That last part is what makes extraction a production problem rather than a prompt problem. A chat answer that is slightly off is a slightly worse answer. An extraction that is slightly off is a broken filter, a wrong badge on a public listing, or a row that fails to insert at three in the morning inside a batch job.

What are the options for getting structured output from an LLM?

There are roughly five, and I have used most of them at some point.

The first two are where most prototypes start and where they should not stay. JSON mode is a real improvement and the trap is that it feels like the finish line: the parse never throws, so you stop looking. My main extraction path runs on JSON mode, and the failure that taught me the difference was small and very visible. A nullable enum field, seniority level, sometimes came back as the string "null" instead of JSON null. The JSON was valid. The shape was valid. The public listing said "Null" in the place where a seniority badge should be. The immediate fix was a sanitizer that maps "null", "None", "undefined" and the empty string to a real null before assignment. The actual fix is not letting the model choose the representation at all. That older path still has the sanitizer in front of it, and moving it to a strict schema is on my list.

Wrapper libraries that validate and re-ask are a reasonable choice, and before providers enforced schemas natively they were the best option. Today the enforcement lives in the API, so for me the library would mostly add a dependency to audit and upgrade. I would rather own the thirty lines of glue.

What is tool use, and why does it beat parsing?

Tool use, also called function calling, is the API feature where you describe functions the model may call, each with a JSON Schema for its arguments, and the model responds with a structured call instead of prose. It was built for agents that take actions, but it is also the cleanest extraction interface there is. Define a single tool whose arguments are exactly the fields you want, tell the model to record its findings with it, and the arguments are your row.

Structured outputs are the same idea without the tool framing: you pass the schema as the response format, in strict mode, and the provider constrains generation so that only schema-valid output can be produced. Under the hood both rely on constrained decoding, where tokens that would break the schema are simply not available to the model. Which one you use depends on the provider and on whether the call also needs real tools. For pure extraction I use the response-format variant where it exists and a single strict tool where it does not.

// illustrative shape - a strict extraction schema
{
  name: "record_listing",
  strict: true,
  schema: {
    type: "object",
    properties: {
      category:  { type: "string", enum: ["engineering", "design", "data", "other"] },
      seniority: { type: ["string", "null"], enum: ["junior", "mid", "senior", null] },
      skills:    { type: "array", items: { type: "string" } },
      notes:     { type: "string" }
    },
    required: ["category", "seniority", "skills", "notes"],
    additionalProperties: false
  }
}

A few things in that shape carry weight. Strict mode generally wants every property listed as required, so optional becomes nullable: the field must be present, and absence is expressed as null inside a union type. Enums include null explicitly, which closes the door on the "null" string. And additionalProperties: false means the model cannot invent a field you then have to decide what to do with. The discovery extractors in my job-matching product were built this way from the start and have not needed one.

One provider-specific caveat: some APIs let you force the model to call a particular tool, others restrict that on certain models. I do not design around forcing. A strict schema plus a clear instruction to use the tool is portable; a forced choice is a knob that may not exist on the next model you switch to.

What a schema does not guarantee

This is the section I wish someone had written for me. Enforced shape removes one class of bugs and makes the remaining ones easier to see. The remaining ones are real.

• Ranges. An integer is an integer; strict mode does not stop a score of 140 on a 0 to 100 scale. Clamping stays in code.
• Truth. A perfectly typed field can still be wrong. The model can pick a plausible category that the source text does not support. Where the source already carries an unambiguous label, a deterministic mapping wins over the model, and I log every disagreement so I can see how often the model would have been overruled.
• Data you already have. The mistake that cost me the most cleanup in this area was letting the model return a field that also existed in the source and writing the model's version. The prompt asked it to clean up job titles by stripping modifier words. It also stripped legitimate words from real titles, truncating a multi-word role to its last word on 39 production rows. The rule since then: authoritative source fields are never overwritten by extraction. The model may still compute them internally for classification; the database keeps the original.
• Human corrections. If an admin fixes a field by hand, a later reprocess must not silently put the model's answer back. Reprocessing takes an explicit flag that preserves manual overrides.

Retries, idempotency and failure states

Structured output changes what you retry. Transport failures (rate limits, timeouts, server errors) are the SDK's job; the official SDKs already retry those with backoff. What matters is choosing the timeout per path. A nightly batch can live with long defaults. An interactive request where someone watches a spinner needs a tight bound and at most one retry, so it fails fast and cleanly instead of hanging for minutes.

Validation failures are different. Once shape is enforced by the provider, what is left to validate is meaning: a range check, a cross-field rule, a guard that checks the output does not contain something it must not. My rule for those is one retry with the specific findings appended to the request, then a hard failure. Never a second silent retry, and never persisting output that failed a check.

Around every extraction call sit three cheap guards that save more money than any model choice:

• A content hash. If the input has not changed since the last successful extraction, skip the call.
• A processing state. A record marked as in progress is not picked up by a second worker, and a failure is recorded as a failed state, not left half-written.
• A size limit and a cheap first pass. Oversized input is skipped before it costs anything, and a short gating call decides whether the record is worth the full extraction at all.

Should you abstract the LLM provider?

Yes, and thinly. The pattern is ports and adapters (also called hexagonal architecture): the application depends on a small interface it owns, and each vendor is an adapter that implements it. For an AI-visibility product I run, which queries several AI engines, the interface has two methods: whether the adapter is configured, and ask. Every adapter returns the same normalized result: the answer, citations, and token usage for cost accounting. Callers never import a vendor SDK.

// illustrative shape - the port the app depends on
interface Engine {
  id: string;
  enabled(env: Env): boolean;          // has its key/config
  ask(input: string, env: Env): Promise<{
    text: string;
    usage?: { inputTokens: number; outputTokens: number };
  }>;
}

The payoff is concrete. Switching the model for one job, trying a cheaper provider, or moving to a cloud-hosted endpoint such as Bedrock or Azure for data-residency reasons is a new adapter and a config change. A registry resolves configured adapters at runtime and skips the ones without keys, which is also how local development runs with a mock engine and no spend. Many providers now expose an OpenAI-compatible endpoint, which makes some switches as small as a base URL, but I would not rely on that alone: strict schema support varies between providers, and the adapter is where that difference gets handled.

The limit of the approach: an adapter hides call signatures, not behavior. Two models behind the same interface can extract the same document differently. That is why a provider switch is a measurement exercise as much as a code change. I stamp model-produced rows with the model that produced them, so the before and after can be compared on real data.

What I would tell someone doing this tomorrow

• Skip "return JSON" entirely. Start with a strict schema through tool use or structured outputs.
• Make optional fields nullable and enums include null. That single habit would have prevented my most visible extraction bug.
• Keep a deterministic layer after the model. Clamp ranges, map known labels, and log where code overrules the model.
• Never let extraction overwrite authoritative data. Not source fields, not human corrections.
• Leave transport retries to the SDK. Validation gets one retry with the findings, then a recorded failure.
• Hash, lock, and gate before you call. The cheapest model call is the one you skip.
• Own the interface, rent the vendor. One port, one adapter per provider, and the model name stored on every row.

The same measure-before-trusting habit applies to the retrieval side of the pipeline; I wrote about calibrating it in tuning pgvector similarity thresholds.