Why Your LLM Integration Returns Truncated JSON (Max Tokens vs Parsing Errors)
By Abhishek Patoliya · 2026-08-17
Your integration asks the model for a JSON object with a summary, a list of extracted entities, and a confidence score. It works fine in testing with short inputs, then in production it starts throwing Unexpected end of JSON input on a subset of requests — always the longer, more complex ones. That pattern is the signature of truncated JSON: the model was still writing the response when it hit its output token limit, and the response got cut off mid-structure.
Why Longer Inputs Produce Truncated JSON More Often
This isn't random — it correlates directly with output length. A document with more entities to extract produces a longer entities array; a more nuanced summary request produces a longer summary field. If your max_tokens setting was sized against your testing data (typically short, simple examples), it will be too low for the harder inputs that show up once real traffic arrives, and the model gets cut off exactly where the JSON is deepest — mid-array, mid-string, mid-object — which is why the resulting parse error is often something unhelpful like "unexpected end of input" rather than a pointer to what's actually wrong.
Fix 1: Size max_tokens for Your Worst Case, Not Your Average Case
Estimate output size from your longest realistic input, not your typical one, and add real headroom — 30-50% above your worst-case estimate is a reasonable starting point. Most providers charge only for tokens actually generated, so an unused ceiling costs nothing; a ceiling that's too low costs you a broken response.
const response = await anthropic.messages.create({
model: "claude-sonnet-5",
max_tokens: 4096, // sized for worst-case entity count, not the demo input
tools: [extractionTool],
tool_choice: { type: "tool", name: "submit_extraction" },
messages: [{ role: "user", content: documentText }],
});
Fix 2: Use Native Structured Output, Not Free-Text JSON
Asking a model to "respond with valid JSON" in a plain-text prompt is fragile even when max_tokens is sized correctly, because the model can still wrap the JSON in markdown fences, add explanatory text before or after it, or produce nearly-valid-but-not-quite JSON (trailing commas, unescaped quotes inside strings). Tool calling / function calling (Anthropic's tool use, OpenAI's structured outputs or function calling) sidesteps this entirely — the model's output is constrained to match a schema you define, and you parse the tool call's input object directly instead of parsing free text that happens to look like JSON.
const tool = {
name: "submit_extraction",
input_schema: {
type: "object",
properties: {
summary: { type: "string" },
entities: { type: "array", items: { type: "string" } },
confidence: { type: "number" },
},
required: ["summary", "entities", "confidence"],
},
};
This also makes truncation easier to detect programmatically: a tool call response that's missing required fields, versus a free-text response where you have to guess whether the JSON parser failed because of truncation or because of a genuinely malformed response.
Fix 3: Detect Truncation Explicitly, Don't Just Catch the Parse Error
Every major provider's API response includes a stop reason — check it before you even attempt to parse. Anthropic returns stop_reason: "max_tokens"; OpenAI returns finish_reason: "length". Checking this field directly tells you definitively whether truncation happened, instead of inferring it indirectly from a JSON parse failure that could have other causes.
if (response.stop_reason === "max_tokens") {
// Definitely truncated — retry with a higher limit, don't just
// attempt to parse and catch whatever error falls out
return retryWithHigherLimit(response);
}
Fix 4: Have a Retry Path, Not Just a Bigger Ceiling
Even with generous headroom, an unusually large input can still exceed your limit. Build a retry path that doubles max_tokens and tries again (capped at a sane maximum to avoid runaway cost on a genuinely malformed request), rather than letting the request fail outright. Log every truncation event with the input size that caused it — over time this data tells you whether your headroom estimate needs to move up permanently, or whether a specific input pattern (very long documents, unusually entity-dense text) needs its own handling path.
A Second, Sneakier Cause: Prompt-Level JSON Instructions Fighting the Schema
Even with tool calling and correctly sized max_tokens, truncated or malformed JSON can still show up if your system prompt gives the model conflicting instructions about output format — for example, asking it to "explain your reasoning, then provide the JSON" while also using tool calling, which expects the tool call to be the model's entire structured response. The model tries to satisfy both instructions, produces reasoning text, then runs out of budget before completing the tool call, or produces a tool call with reasoning awkwardly stuffed into one of the JSON fields where it doesn't belong. The fix is separating concerns cleanly: if you want reasoning, ask for it in a dedicated field within the schema (a "reasoning" string property), not as free text before or after the structured call, and never mix "explain in prose" instructions with a tool-calling constraint that expects the entire response to be the structured call.
Handling Truncation Gracefully When It Still Happens
No amount of headroom sizing eliminates truncation entirely — a genuinely unusual input can still exceed even a generous ceiling. Design your parsing layer to fail informatively rather than throwing a raw parse exception up to whatever called it. Catch the truncation case specifically (via the stop reason check above) and return a distinct, actionable error rather than a generic "invalid response" — one that a calling workflow can branch on to either retry with more budget, route to a human reviewer, or, for genuinely oversized inputs, split the input and process it in smaller pieces rather than trying to extract everything in a single call. A generic try/catch around JSON.parse that swallows the distinction between "truncated" and "the model produced genuinely malformed output despite structured tool calling" makes debugging every future occurrence slower than it needs to be.
Testing Truncation Handling Before It Happens in Production
Most teams discover their truncation handling is broken the first time it actually happens in production, because it's rarely exercised in normal testing where inputs are short and well-behaved. Test it deliberately: construct an input specifically designed to produce a long response (a document with an unusually high entity count, for example) and set max_tokens artificially low in a test environment to force a truncation, then confirm your stop-reason check fires, your retry path engages, and the eventual error — if the retry also fails — surfaces as the specific "truncated" case rather than a generic parse failure. This is a five-minute test that catches a class of bug that's otherwise invisible until a real user hits it with a genuinely large input.
Run that same test again whenever you change providers or switch models — max_tokens defaults, stop-reason field names, and even how aggressively a model trends toward longer outputs all vary enough between providers that a truncation-handling path validated against one can silently stop working correctly after a migration to another. Treat provider migrations the same way you'd treat any other dependency upgrade with behavioral differences: re-run the full test suite, including this one, before trusting the new provider's responses in production. A response format that "looked the same" in a quick manual comparison can still differ in exactly the field your truncation detection depends on.
Written by Abhishek Patoliya, Senior n8n Developer & Automation Architect, Surat, India. Verified n8n Creator.