Streaming AI answers on Azure Static Web Apps: the one header that buffered everything

1. Why streaming matters for a sales chat

A visitor who asks a question on your website decides within a second or two whether the answer is worth waiting for. Before streaming, the guide on this site showed nothing but a typing indicator until the model had finished, and then the full text at once, after roughly 2.5 to 3 seconds. With streaming, measured on the live site, the first words appear after 1.2 to 1.5 seconds and the rest follows while the visitor is already reading.

Streaming doesn't make the model faster and it doesn't change the token bill. It changes when the visitor sees the first useful sentence, and for a chat that sells, that is the number that matters.

2. Server-sent events over a normal POST

For a chat answer you don't need WebSockets. The data flows in one direction, from the server to the browser, and server-sent events (SSE) do exactly that over plain HTTP, through every proxy and CDN that respects the protocol. The browser's EventSource only supports GET, but a chat request carries the conversation in its body, so the widget uses fetch with a POST and reads the response body as a stream.

Keep the event format small and explicit: an event named delta with a piece of text, done when the answer is complete, and error if something breaks. Each event is one JSON line after data:, followed by an empty line. That is the whole protocol, and it is easy to read in curl, in DevTools and in tests.

event: delta
data: {"text":"Yes. Start with a cost baseline "}

event: delta
data: {"text":"per resource, then right-size on p95."}

event: done
data: {"model":"gemini-3.5-flash"}

3. The function: v4 model and HTTP streams

Static Web Apps runs your API as managed Azure Functions. The older v3 programming model, with a function.json per endpoint, sets the response body once, so it can only answer in one piece. The v4 Node.js programming model has HTTP streams: you enable them once with app.setup({ enableHttpStream: true }) and return a ReadableStream as the body. Everything you enqueue goes out as it is written.

Moving this site's API from v3 to v4 meant registering the endpoint with app.http, pointing main in api/package.json to the file, and deleting function.json. I tested it on a separate preview environment of the same Static Web App before touching production: a test function that sent six events half a second apart, and they arrived half a second apart.

const { app } = require("@azure/functions");
app.setup({ enableHttpStream: true });

app.http("chat", {
  methods: ["POST"],
  authLevel: "anonymous",
  handler: async (request) => {
    const body = await request.json();
    const enc = new TextEncoder();
    const stream = new ReadableStream({
      async start(controller) {
        const send = (event, data) =>
          controller.enqueue(enc.encode(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`));
        try {
          for await (const text of askModel(body)) send("delta", { text });
          send("done", {});
        } catch {
          send("error", { error: "upstream" });
        } finally {
          controller.close();
        }
      },
    });
    // Exactly text/event-stream: no charset suffix (see step 4)
    return { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-store" }, body: stream };
  },
});

4. The header that held back every answer

With the real chat deployed, the test environment behaved strangely: the answer arrived as fifteen separate events, but all fifteen in the same millisecond, at the very end. The model was streaming, the function was streaming, yet the browser got one block. So I deployed a test build that wrote the server's send time into every event: the server sent over 1.3 seconds, the client received everything at once. Something between them was buffering.

To find what, I deployed eight variants of the same 300-millisecond ticker side by side, changing one thing at a time: GET or POST, the plain body or one wrapped in a Response object, Cache-Control no-store and no-transform, X-Content-Type-Options nosniff, X-Accel-Buffering no, and the Content-Type with or without "; charset=utf-8". Only one variable mattered. Every variant with the charset suffix arrived in one burst at the end; every variant without it arrived on time, 300 milliseconds apart.

The suffix adds nothing anyway: the HTML standard says event streams are always decoded as UTF-8 and there is no way to choose another encoding. So the fix was one line: send exactly text/event-stream.

5. The model side: stream, fall back, time out

Gemini streams through the streamGenerateContent endpoint; with alt=sse it answers as server-sent events, one JSON chunk per event. The function reads those chunks, skips parts that are marked as the model's internal reasoning, and forwards the text as delta events. Claude and OpenAI-compatible APIs stream the same way with their own event shapes, so the forwarding stays identical.

Fallbacks need one new rule. Without streaming you simply retry the next model when the first one fails. With streaming you may already have shown half an answer, and starting a second answer on top of it would confuse the visitor. So the next model is tried only while nothing has been sent yet. A timeout until the first token keeps a slow model from blocking the chat, and a longer overall cap ends a stream that stalls halfway.

// Try providers in order; switch only while nothing has been sent
for (const provider of providers) {
  let started = false;
  const ctrl = new AbortController();
  const firstToken = setTimeout(() => ctrl.abort(), provider.timeout ?? 15000);
  try {
    const out = await provider.stream(request, ctrl.signal, (text) => {
      if (!started) (started = true), clearTimeout(firstToken);
      send("delta", { text });
    });
    if (out.text) return send("done", { model: provider.model });
  } catch {
    /* fall through */
  } finally {
    clearTimeout(firstToken);
  }
  if (started) break; // half an answer is on screen: don't start another
}
send("error", { error: "upstream" });

6. The browser: render as it arrives, finish cleanly

In the browser, read the response body with a reader, decode the bytes, split on empty lines and handle each event. Render on the next animation frame rather than on every chunk, so long answers don't stutter. The answers of this guide end with markers for navigation buttons and follow-up questions; while the text streams, an unfinished marker would flash on screen, so everything from an open double bracket onward stays hidden until it is complete, and the buttons appear when the done event arrives.

Two details make it robust. Mark the growing message with aria-busy, so screen readers announce the finished answer once instead of every fragment. And keep the old path: if the server answers with JSON instead of a stream, because it is an older version or a host that can't stream, the widget shows that answer as before.

const res = await fetch("/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json", Accept: "text/event-stream, application/json" },
  body: JSON.stringify({ messages, stream: true }),
});
if (!res.headers.get("content-type")?.includes("text/event-stream")) return showJson(await res.json());

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  let i;
  while ((i = buffer.indexOf("\n\n")) >= 0) {
    const block = buffer.slice(0, i);
    buffer = buffer.slice(i + 2);
    const event = /^event: (.*)$/m.exec(block)?.[1];
    const data = JSON.parse(/^data: (.*)$/m.exec(block)?.[1] ?? "{}");
    if (event === "delta") appendText(data.text); // painted on the next animation frame
    if (event === "done") showButtons();
  }
}

7. Prove it before you trust it

A stream that is buffered somewhere looks exactly like a slow stream in the browser, so test with timestamps. curl -N disables curl's own buffering; print the arrival time of every event line and look at the spread. If all events share one timestamp, something buffers. Do this on a preview environment first, then on production, and once more with the production Content Security Policy applied.

The checks that run with every build of this site cover the rest: the package's own tests stream through each adapter, cut the connection after part of an answer and check that no second answer starts, and a check in the site's API confirms that the primary model streams and the events reach the client. The non-streaming JSON path stays tested too, because older clients still use it.

# Arrival time of every SSE event; all identical = something buffers
curl -sN -X POST https://<your-site>/api/chat \
  -H 'Content-Type: application/json' \
  --data '{"messages":[{"role":"user","text":"How do you start?"}],"stream":true}' \
  | perl -MTime::HiRes=time -ne 'BEGIN{$|=1; $t0=time} printf "%6.2fs %s", time-$t0, $_ if /^event:/'

8. Package it once, reuse it everywhere

All of this lives in one reusable package, the engine I built for this guide and now use on other sites: the widget, a handler with adapters for Azure Functions, Node, Vercel, Cloudflare Workers and other web-standard hosts, and the provider code for Gemini, Claude and OpenAI-compatible APIs. Streaming became one version bump for every site that uses it, and the charset fix one more patch release. Helptempo is the hosted version of the same engine, for teams that want the chat without touching their own stack.

If your chat waits for the whole answer today, the change is smaller than it looks: one setting in the function, one header, a reader in the browser and a fallback rule. Measure before and after; on this site it cut the time to the first visible word roughly in half.