Streaming tokens from an LLM response parse into malformed JSON mid-stream

Streaming structured output: client gets tokens as they arrive and tries to parse partial JSON progressively for UI updates (showing fields as they complete). 15-20% of streams produce unparseable intermediate states even though the final stream is valid. Current approach (trying JSON.parse on every token) fails on every partial stream.

context

OpenAI streaming completions, structured-outputs mode, schema with 8 top-level fields. Client is Next.js with server-sent events. Want to display each field as it completes to reduce perceived latency.

goal

Identify the right progressive-parsing strategy for streaming JSON. Options: streaming JSON parser library, field-boundary detection, partial-parse-and-repair, model-side instrumentation. Recommend one and explain why.

constraints

Stream must stay streaming (no buffering full response). Library dependencies are OK.