Voice / ML Platform
Go services for streaming audio and ML integrations.
Interactive architecture
Streaming, cancellation and trace context.
Audio arrives in chunks
Input arrives incrementally as audio chunks, rather than waiting for the complete recording.
- Turn
- illustrative turn A
- Buffer policy
- bounded
- Output
- not started
A conceptual caller. Both input and output remain open over a streaming transport.
Read every step as text
Follow one illustrative turn from incremental input to incremental output.
Input arrives incrementally as audio chunks, rather than waiting for the complete recording.
The receiver holds a bounded queue. If downstream processing slows, apply a deliberate backpressure or discard policy rather than grow memory indefinitely.
VAD classifies incoming chunks as speech or silence before endpointing the current utterance.
The ASR stream can produce a partial hypothesis before a final result. Synthetic example: “Let’s…” → “Let’s discuss the system.”
The orchestrator owns a cancellable context for this turn. Final and partial results have different semantics; do not treat a partial as an irrevocable command.
A conceptual LLM call produces response content. Context, deadlines and failure boundaries propagate with the request.
TTS creates output chunks for the current turn. The output queue remains bounded; already completed work cannot be retroactively cancelled.
Bidirectional streaming lets output travel while the input side remains open. Trace context connects service boundaries without exposing audio or transcript data.
Record spans, cancellation causes and queue pressure. No fake production traces or millisecond budgets are emitted by this demonstration.
Less work per chunk. Faster audio.
I profiled the processing path, removed bottlenecks and reduced audio chunk processing from 200 to 30 ms.
Audio chunk processing
Both bars use the same 200 ms scale. Reduction: (200 − 30) / 200 = 85%.
less processing time per audio chunk
lower API latency after SQL and inter-service optimizations
200 / 30 = 6.67×. Less processing time leaves more room in the audio pipeline’s latency budget.
I improved intent classification by 0.10 absolute Macro F1.
I improved entity extraction by approximately 0.10 absolute F1.
I increased the share of responses with valid JSON structure by 5–6 percentage points.
Reduced negation error frequency fourfold. The comparison is normalized to four units before and one after.
Overview
I build Go backend services for distributed audio processing and ML integrations. My work covers streaming interfaces, profiling, database queries and service reliability.
The problem
Streaming interactions and asynchronous ML jobs have different lifecycles. Interfaces need to make cancellation, backpressure, failures, and completion understandable across those boundaries.
My contribution
I developed and maintained Go services for distributed audio processing at Zuzex. Contributed to architecture and code reviews, profiled bottlenecks, optimized SQL and inter-service operations, and independently maintained CI/CD pipelines with Docker, unit tests, and golangci-lint.
Decisions & trade-offs
- Use explicit service contracts for streaming and asynchronous boundaries.
- Carry context through service boundaries and connect traces with OpenTelemetry.
Streaming improves incremental interaction, but requires careful lifecycle and backpressure handling. Asynchronous processing decouples execution at the cost of delayed completion and additional failure states.
Streaming across service boundaries.
At Zuzex, I develop Go audio-processing services, profile processing paths, optimize SQL and maintain CI/CD. Streaming interfaces must handle slow consumers, interrupted requests and partial results without leaking work.
A bidirectional gRPC stream separates input and output lifecycles. Closing one direction, cancelling the whole context and reaching a deadline are different events; service contracts must distinguish them.
Incremental input and partial recognition.
The illustrative input delivers chunks through a bounded buffer. VAD and streaming ASR can emit activity and partial recognition before a final transcript exists. Partial and final results need different event types; treating a mutable hypothesis as a final instruction creates subtle application bugs.
An orchestrator owns the current turn, invokes response generation and feeds incremental TTS output back over an open stream. Transport-level ordering is not enough: turn identity must also distinguish current work from delayed chunks of a cancelled response.
Cancelling an interrupted response.
If the caller speaks while output is playing, a new turn takes ownership. Cancel outstanding synthesis and any still-running cancellable downstream work, invalidate the old output queue and ignore late chunks tagged with the old turn. A completed LLM call cannot be undone.
Cancellation is cooperative and crosses service boundaries imperfectly. In Go, propagate a context, select on cancellation around blocking operations and define what closing a send/receive direction means. Tests should distinguish a graceful half-close, deadline expiry, peer disconnect and domain cancellation.
Backpressure is a product decision too.
An unbounded queue hides overload until latency or memory becomes unacceptable. A bounded queue forces a policy: block upstream, drop stale chunks, reject work or degrade gracefully. The correct policy depends on whether losing a chunk is safer than delivering very old speech.
Batch request/response is simpler but delays useful partial output. Streaming offers incremental feedback while introducing more lifecycle states, backpressure and cancellation races. A hybrid can retain synchronous control while moving long work to an asynchronous boundary.
Test the overlaps, not only the transcript.
The proposed matrix delays ASR partials, cancels during synthesis, delivers a stale chunk after a new turn begins, slows an output consumer and disconnects one side of the stream. Assert queue bounds, cancellation propagation and absence of old-turn playback.
I test cancellation at each boundary, including after a new turn has started. The key invariant is that a late chunk from a cancelled response must never reach audio playback.
A latency budget needs a critical path.
Capture, buffering, recognition, generation and synthesis can overlap. I trace the critical path rather than add concurrent spans as sequential work. Queue time, first usable output, completion and cancellation need separate markers.
I propagate trace and turn context across services, keeping audio and transcript payloads out of logs. Queue saturation, cancellations, deadline outcomes and stale-chunk rejection help explain where a stream stalled.
Results & next steps
I reduced audio chunk processing from 200 to 30 ms — an 85% reduction in processing time. Optimizing SQL and inter-service operations also reduced API latency by 15–20%.
The takeaway
Design for the boundaries: ownership, cancellation, recovery, and observability matter as much as the service implementation.