Turn-level error handling discards all partial work (text, reasoning, tool calls)

Summary

When a Hermes session encounters a provider error, credential failure, network drop, or any exception mid-turn, the WebUI discards all accumulated partial work — assistant text, reasoning steps, and tool call results — from the session sidecar. The user sees only a generic error message. This contrasts with the user-initiated cancel path (cancel_stream), which correctly snapshots and preserves partial content.

For a turn where the agent worked for 20 minutes executing complex tools, a final-step provider error throws away all intermediate progress from the WebUI's perspective. The core agent transcript (state.db) does preserve messages, but the WebUI sidecar does not, causing context desynchronization on subsequent turns.

Root Cause

Two error-handling paths in api/streaming.py clear the session's pending state and append a terminal _error=True block without first harvesting accumulated partial content from the streaming memory buffers:

Path 1: Silent Failure (no assistant response returned)

api/streaming.py ~line 6795:

_materialize_pending_user_turn_before_error(s)
s.active_stream_id = None
s.pending_user_message = None
s.pending_attachments = []
s.pending_started_at = None
_error_message = {
    'role': 'assistant',
    'content': f'**{_err_label}:** {_error_payload.get("message") or _err_label}\n\n*{_err_hint}*',
    'timestamp': int(time.time()),
    '_error': True,
}
s.messages.append(_error_message)
s.save()

Path 2: Unhandled Exception

api/streaming.py ~line 7744:

_materialize_pending_user_turn_before_error(s)
s.active_stream_id = None
s.pending_user_message = None
s.pending_attachments = []
s.pending_started_at = None
_error_message = {
    'role': 'assistant',
    'content': f'**{_exc_label}:** {_error_payload.get("message") or err_str}' + (f'\n\n*{_exc_hint}*' if _exc_hint else ''),
    'timestamp': int(time.time()),
    '_error': True,
}
s.messages.append(_error_message)
s.save()

Why Partial Content Is Lost

1. No buffer snapshot: Both paths skip reading STREAM_PARTIAL_TEXT[stream_id], STREAM_REASONING_TEXT[stream_id], and STREAM_LIVE_TOOL_CALLS[stream_id] before clearing state.

2. Safety net bypassed: Both paths clear s.active_stream_id and s.pending_user_message before saving. The finally block's _last_resort_sync_from_core() (line ~7790) checks for these fields and skips recovery when they're None.

3. Self-heal disabled: _apply_core_sync_or_error_marker() in api/models.py:1838 bails early when pending_user_message is not set, so subsequent page loads cannot recover the content either.

Contrast: The Cancel Path (Correct Behavior)

cancel_stream() at line 7952 correctly:

• Snapshots STREAM_PARTIAL_TEXT, STREAM_REASONING_TEXT, STREAM_LIVE_TOOL_CALLS under lock (prevents race with agent interrupt)
• Builds a _partial=True assistant message with the snapshot content
• Inserts it into s.messages before the _error=True cancel marker
• The partial content IS included in subsequent conversation history sent to the LLM

Impact

• Wasted token costs: 20+ minutes of tool executions and reasoning are discarded from the WebUI on a final-step provider error
• Context desynchronization: The agent's core transcript (state.db) retains messages, but the WebUI sidecar does not. On the next turn, the LLM conversation history sent to the provider will be missing the failed turn's tool calls and results, causing the model to be out-of-sync with the actual workspace state
• Duplicate work: The agent may re-execute tools or re-read files because it doesn't see prior results in its context
• User confusion: No visibility into what the agent accomplished before the error

Suggested Fix

Adopt the same preservation strategy used in cancel_stream() for both error paths:

1. Snapshot streaming buffers: Before clearing pending state, read STREAM_PARTIAL_TEXT[stream_id], STREAM_REASONING_TEXT[stream_id], and STREAM_LIVE_TOOL_CALLS[stream_id] under STREAMS_LOCK (same pattern as cancel_stream lines 7998-8021).

2. Build _partial message: If any partial content, reasoning, or tool calls exist, construct an assistant message with _partial: True containing the accumulated content. Insert it into s.messages before the error block.

3. Append error block: Then append the _error: True block as before.

4. Preserve the existing finally safety net: Do NOT clear active_stream_id and pending_user_message until after the partial content has been harvested.

Minimal Diff Sketch

# In both silent-failure and exception handlers, BEFORE clearing pending state:
from api.streaming import STREAMS_LOCK, STREAM_PARTIAL_TEXT, STREAM_REASONING_TEXT, STREAM_LIVE_TOOL_CALLS

with STREAMS_LOCK:
    _snap_text = STREAM_PARTIAL_TEXT.get(stream_id, '')
    _snap_reasoning = STREAM_REASONING_TEXT.get(stream_id, '')
    _snap_tools = list(STREAM_LIVE_TOOL_CALLS.get(stream_id, []) or [])

if _snap_text or _snap_reasoning or _snap_tools:
    import re as _re
    _stripped = _re.sub(r'<think(?:ing)?\b[^>]*>.*?</think(?:ing)?>', '', _snap_text, flags=_re.DOTALL | _re.IGNORECASE).strip()
    _stripped = _re.sub(r'<think(?:ing)?\b[^>]*>.*', '', _stripped, flags=_re.DOTALL | _re.IGNORECASE).strip()
    _partial_msg = {
        'role': 'assistant',
        'content': _stripped,
        '_partial': True,
        'timestamp': int(time.time()),
    }
    if _snap_reasoning:
        _partial_msg['reasoning'] = _snap_reasoning.strip()
    if _snap_tools:
        _partial_msg['_partial_tool_calls'] = _snap_tools
    s.messages.append(_partial_msg)

# Then clear pending state and append error block as before

Existing Test Coverage

Test File	What It Covers
tests/test_issue1361_cancel_data_loss.py	User cancel preserves partial text/reasoning/tools
tests/test_issue893_cancel_preserves_partial.py	Partial text saved with _partial=True on cancel
tests/test_cancel_interrupt.py	Cancel calls agent.interrupt() correctly

Gap: No tests verify that provider errors or exceptions preserve partial streamed content.

Suggested Test

def test_provider_error_preserves_partial_work():
    """When a provider error occurs mid-turn, partial assistant text,
    reasoning, and tool calls should be preserved in the sidecar."""
    # 1. Start a stream that will produce partial content
    # 2. Inject a provider error after some tokens + tool calls are streamed
    # 3. Assert that the sidecar contains a _partial=True assistant message
    #    with the accumulated content before the _error=True block
    # 4. Assert that subsequent conversation history includes the partial work