Skip to content

fix(runtime-core): report failed agent-added code blocks via onBlockDone#369

Merged
tkislan merged 1 commit into
tk/agent-vscode-abstractfrom
jhobbs/agent-block-quick-fixes
Apr 30, 2026
Merged

fix(runtime-core): report failed agent-added code blocks via onBlockDone#369
tkislan merged 1 commit into
tk/agent-vscode-abstractfrom
jhobbs/agent-block-quick-fixes

Conversation

@jamesbhobbs

@jamesbhobbs jamesbhobbs commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Chained on top of #342. Quick fixes from the review of that PR.

What this changes

When the agent inserts a code block via addAndExecuteCodeBlock, the runtime tracks the outcome of that kernel execution and surfaces it through onBlockDone. Previously:

  • A kernel.execute resolution with success: false (e.g. SyntaxError) was reported via onBlockDone with success: true — the success flag was dropped between blockOutputs and the post-loop.
  • A kernel.execute rejection was never reported via onBlockDone at all — the catch path returned the error string to the agent but never pushed to blockOutputs, so the post-loop skipped it. The block was already inserted into notebook.blocks, leaving consumers with an inserted-but-never-completed block.
  • When executeAgentBlock itself threw partway through, the post-loop never ran, so even successfully-completed added blocks weren't reported.

The fix

  • Track success: boolean on each blockOutputs entry.
  • Push an entry on kernel.execute rejection (with a synthesized error output via the existing createErrorOutput helper) so the failed block still gets onBlockDone.
  • Wrap the executeAgentBlock call in try/finally so added blocks are drained even when the agent throws.
  • Pass bo.success through to onBlockDone instead of hardcoded true.
  • Drop the redundant collectedOutputs.set in the post-loop (the per-block set inside addAndExecuteCodeBlock already covers all paths now).
  • Type blockOutputs.outputs as IOutput[] to drop the two as IOutput[] casts.
  • Tighten the add_code_block tool description (it claimed "stdout, stderr, and execution results" but actually returns a single formatted string).

Tests

  • Augmented surfaces addAndExecuteCodeBlock failures when kernel execution rejects to assert the failed code block now gets onBlockDone with success: false and a synthesized error output.
  • Augmented returns failure when kernel execution fails to assert success: false propagates for the resolved-failure path too.

All 204 tests in runtime-core pass; typecheck and biome clean.

Out of scope (deliberately)

From the review:

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Enhanced error handling and reporting for code blocks executed by agents. Callbacks are now reliably emitted even when execution fails, and each block's success/failure status is accurately tracked and reported.
  • Tests

    • Added tests to verify proper error reporting and callback emission when agent-executed code blocks encounter execution failures.

Track success on blockOutputs entries and push on kernel.execute
rejection so blocks the agent inserted always get an onBlockDone event
matching their actual outcome. Wrap the agent call in try/finally so
added blocks are drained even when executeAgentBlock throws.

Also tighten add_code_block tool description and drop the redundant
post-loop collectedOutputs.set / unknown[] casts.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@coderabbitai

coderabbitai Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 94be64bf-6a6c-49bc-af43-fbbb94a42595

📥 Commits

Reviewing files that changed from the base of the PR and between cbe4e74 and a5b8b14.

📒 Files selected for processing (3)
  • packages/runtime-core/src/agent-handler.ts
  • packages/runtime-core/src/execution-engine.test.ts
  • packages/runtime-core/src/execution-engine.ts

📝 Walkthrough

Walkthrough

The changes enhance error tracking for agent-inserted code blocks. The execution engine now records success/failure status per added block, generates error outputs on kernel failures, and moves callback reporting into a finally block to fire completion handlers even when agents fail mid-execution. A tool description was clarified to reflect the revised output contract.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant ExecutionEngine
    participant Kernel

    Agent->>ExecutionEngine: Add code block
    ExecutionEngine->>Kernel: Execute block
    
    alt Execution succeeds
        Kernel-->>ExecutionEngine: outputs + success
        ExecutionEngine->>ExecutionEngine: Record success: true
    else Kernel error/rejection
        Kernel-->>ExecutionEngine: error/exception
        ExecutionEngine->>ExecutionEngine: Generate error output<br/>Record success: false
    end
    
    ExecutionEngine->>ExecutionEngine: Finally block:<br/>Fire onOutput, onBlockDone
    ExecutionEngine-->>Agent: Block complete<br/>(success reflects actual status)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Updates Docs ⚠️ Warning No documentation files were modified. Agent block API behavior changed regarding success callback reporting. Verify if documentation exists for agent block API and onBlockDone callback, then add clarification about failed blocks now correctly reporting success: false.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Title accurately summarizes the main change: fixing how failed agent-added code blocks are reported via onBlockDone callback.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Review rate limit: 4/5 reviews remaining, refill in 12 minutes.

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented Apr 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.80%. Comparing base (cbe4e74) to head (a5b8b14).
⚠️ Report is 2 commits behind head on tk/agent-vscode-abstract.

Additional details and impacted files
@@                    Coverage Diff                    @@
##           tk/agent-vscode-abstract     #369   +/-   ##
=========================================================
  Coverage                     82.80%   82.80%           
=========================================================
  Files                           144      144           
  Lines                          5867     5869    +2     
  Branches                       1141     1141           
=========================================================
+ Hits                           4858     4860    +2     
  Misses                         1009     1009           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tkislan tkislan marked this pull request as ready for review April 30, 2026 16:38
@tkislan tkislan requested a review from a team as a code owner April 30, 2026 16:38
@tkislan tkislan merged commit 4c32bb0 into tk/agent-vscode-abstract Apr 30, 2026
21 checks passed
@tkislan tkislan deleted the jhobbs/agent-block-quick-fixes branch April 30, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants