NOSSLETTER.techLIVE
back to archive

[Perf][Frontend] Decode streamed custom-tool input incrementally

Language: Python+129 -4performance
concepts demonstrated
Incremental parsingString slicing and manipulationPerformance optimization techniquesUnit testing for performance and correctness
The Issue & Context

Previously, every streamed delta of a custom tool argument triggered a full re-parse of the entire accumulated input string, causing CPU cost per delta to grow linearly with input size.

Root Cause Analysis

The partial_json_parser was re-run over the entire concatenated argument string on each delta update in responses/streaming_events.py, leading to O(N) work per delta where N is the input length.

Solution Strategy & Architectural Pattern

The patch modifies the decoding logic in responses/utils.py and streaming_events.py to incrementally decode only the newly appended tail of the input string after each delta, avoiding repeated parsing of the entire input. This preserves streamed event semantics while drastically reducing CPU usage.

Key Takeaway for Contributors

Incremental parsing of streamed inputs can prevent costly repeated work and improve scalability, especially when input size grows large and updates are small deltas.