Agentic Coding Tools for Data Analysis

Executive summary

Agentic coding tools split into two classes for analytical work, and as of mid-2026 the split is converging on a bridge rather than a winner. Terminal and IDE agents — Claude Code, Codex, Cursor — treat a Jupyter notebook as a file to edit, manipulating .ipynb JSON with no kernel access [1]. Notebook-native agents — Hex, Databricks, Deepnote, Runcell — operate inside the kernel, seeing live variables and rendered plots [2][3]. The convergence point is the Model Context Protocol: a Jupyter MCP server is how a terminal agent reaches inside the kernel, and practitioners now recommend it over file-only editing [4].

The central risk is a sharp accuracy cliff between simple and multi-step analysis. In benchmarks run on 2024–mid-2025 models, single-step, closed-form questions were largely solved — InfiAgent-DABench put GPT-4 at 78.99% [5], DABstep's Easy split reached 76.39% [6] — while realistic multi-step analysis collapsed: DA-Code 30.5% [7], DSBench 34.12% [8], DABstep Hard 14.55% [6]. No re-runs on current-generation models surfaced in this wave's sources, so the absolute scores are dated; the durable finding is the size of the easy-to-hard gap. The dangerous failure mode is the silent semantic error: a query that executes successfully and returns a plausible but wrong number, undetected until someone acts on it [9].

Two structural levers move the failure mode more than they move raw accuracy. Routing agent queries through a governed semantic layer reached 98.2–100% accuracy versus 84–90% for text-to-SQL on the same modeled data, and — more to the point — fails with an error message instead of a plausible wrong answer [10]. For charts, letting the agent see its own rendered image is the decisive affordance: ablating multimodal visual feedback drops chart accuracy 10–15% across models [11], and tools split on whether the agent can view what it drew.

The bottleneck has moved from producing analysis to reviewing it. The 2025 Stack Overflow survey found trust in AI accuracy fell to 29% from 40%, even as 66% of developers report spending more time fixing "almost-right" code [12]; the 2025 DORA report found higher AI adoption raises delivery throughput and instability together [13]. Within Claude Code specifically, writing and data analysis roughly doubled as a share of sessions — from about 10% to 20% between October 2025 and April 2026 [14].

Introduction

This snapshot covers how data teams use agentic coding tools — tools built around agents and subagents, hooks, skills, and MCP, with Claude Code and OpenAI Codex as the prime examples — for analysis work as of July 2026. Four concerns run through it: how Jupyter notebook workflows shift when an agent joins the loop, how agents produce charts and stakeholder-facing artifacts, when agents catch analytical errors versus introduce them, and the trade-off between fast iteration and reviewable analysis. Underneath all four is a structural question: does agentic analysis tend toward an agent inside the notebook or an agent that produces notebooks, and does the answer differ by tool.

Analysis is not software engineering, and the difference sets the stakes. For a data team, code is a means to an end — a number in a board deck, a chart a stakeholder acts on — not the deliverable itself. An agent that produces plausible-looking but subtly wrong analysis is more dangerous than one that produces buggy software, because the error is silent and surfaces only after someone has acted on it [9]. This is not a niche use: within Claude Code, writing and data-analysis sessions roughly doubled between October 2025 and April 2026 [14], making analysis one of the fastest-growing uses of a tool built for coding.

The snapshot is based on a deep-research run of 58 sources spanning official documentation, vendor engineering blogs, academic benchmarks, and practitioner accounts, weighted to the last 12–18 months. It covers the two-architecture split and the MCP bridge; the easy/hard correctness cliff and how analysts verify; the semantic layer as a correctness lever; chart production and multimodal feedback; the shift of the bottleneck to review; the four cross-tool primitives; and the tool-specific affordances that materially change what analysts can do.

Two architectures: agents that produce notebooks and agents inside the kernel

The clearest structural pattern in 2026 is a split by tool class, not by vendor. Terminal and IDE coding agents were built to edit files in a repository, and they treat a Jupyter notebook as exactly that. Claude Code's built-in NotebookEdit tool can insert, replace, or delete cells, but it has no kernel access, so it cannot "support the real workflow: writing code, running it, inspecting outputs, fixing errors" [1]. It also writes each cell's source as a single JSON string with literal newline escapes rather than Jupyter's list-of-strings format, so every edit surfaces in a git diff as a whole-cell rewrite [1]. A widely-referenced feature request argues that "working with .json file for .ipynb is like working with assembly code, error prone, waste lots of tokens," and proposes a .py-first workflow where a paired Python script is the source of truth and the notebook is generated [15].

OpenAI's Codex has effectively conceded the point. Its data-analysis guidance instructs the agent to "prefer scripts and saved artifacts over one-off notebook state," positioning notebooks as one artifact among many, created "for exploratory analysis, experiments, and reusable walkthroughs" rather than driven live [16]. This is the purest form of agent-that-produces-notebooks: the notebook is an output, not a workspace. Cursor sits in between — its agent "only generates and edits the notebook cells but doesn't appear to be able to run them," so the human still executes [17].

The opposite architecture is where the notebook-native platforms concentrate. Hex's Notebook Agent "lives in your Hex project with access to your code, existing files, and warehouse tables," chaining Python, SQL, Markdown, pivot, and chart cells with cell-level diff review [2]. Databricks' Assistant Data Science Agent is "fully integrated with Databricks Notebooks and the SQL Editor," running code and iteratively fixing its own errors [3]. Posit is explicit about the mechanism: "Posit AI closes the loop by embedding the chatbot in RStudio, letting it see what data and packages you have loaded and run code on your behalf" [18]. Runcell runs "inside JupyterLab on your existing .ipynb files," reading live variables, DataFrames, and image outputs, and foregrounds cross-session memory — a problem the file-editing agents ignore because they hold no persistent kernel state [19].

The difference matters for correctness, not just ergonomics, because of the agentic loop. An agent that cannot execute a cell cannot see that its code raised an exception, produced an empty DataFrame, or returned a nonsensical aggregate; it reasons blind. The bridge that closes this gap for terminal agents is MCP. Connecting Claude Code to a Jupyter MCP server "gives it full IPython kernel access," enabling a loop where the agent "can self-evaluate its code, fix errors in a continuous loop until everything is working"; the practitioner recommendation is to "skip NotebookEdit" and "use the Jupyter MCP Server instead" [1]. Jupyter AI, an official JupyterLab extension, generalizes this into a standard, connecting external agents — Claude, Codex, Copilot, Gemini, and others — to notebooks over the open Agent Client Protocol, with agents acting through a built-in Jupyter MCP server under a permission system that requires approval before writes or command execution [4].

The convergence is that the agent-inside-the-notebook and the agent-that-produces-notebooks are becoming the same agent, distinguished only by whether an MCP or ACP server has granted kernel access. Databricks states the reason data teams push toward the embedded end plainly: "for data teams, code is merely a vehicle to manipulate and understand the underlying data" [20]. A terminal agent without a kernel bridge is a fast code writer that cannot check its own analytical work, and the correctness burden falls entirely on the analyst.

The easy/hard correctness cliff

The most important quantified pattern for analytical trust is the drop in accuracy as tasks gain steps. On closed-form, single-question benchmarks, agents looked strong: on InfiAgent-DABench (257 questions over 52 CSV files), GPT-4 scored 78.99%, "the best of all models tested but still highlighting significant room for improvement" [5], and on DABstep's Easy split the best agent reached 76.39% [6]. Realistic multi-stage work collapsed. On DA-Code (500 real data-science coding tasks across wrangling, EDA, and modeling), even GPT-4 "achieves only a 30.5% score" [7]. On DSBench (466 analysis tasks from ModelOff and Kaggle), "the best agent [solved] only 34.12% of data analysis tasks" [8]. On DABstep's Hard split of 450+ real financial-analytics tasks, the best agent managed 14.55% against 76.39% on Easy, leading the authors to conclude that "tasks requiring multiple steps of reasoning remain largely unsolved" [6]. These benchmarks were run on 2024–mid-2025 models — GPT-4 generation for the oldest — and no re-runs on current-generation models surfaced in this wave's sources; read the absolute numbers as dated, while the cliff between single-step and multi-step accuracy is the pattern that persists across benchmarks.

The failure modes behind the numbers are specific. DA-Code found that "agents often make incorrect assumptions about the environment, such as presuming file names" rather than checking them [7]. DABstep agents "miss necessary intermediate calculations, fail to consult required documentation at the right point, or hallucinate incorrect analysis plans," and are "considerably more prone to fail" when a task requires following rules stated implicitly in context [6]. In the query layer, agents "guess input fields based on standard naming conventions — they might assume your database uses user_id when your schema actually requires customer_uuid" [9]. This is schema hallucination, and it produces the most dangerous class of analytical error: the silent semantic error, where "a semantically wrong query runs successfully and returns a result that looks reasonable — you do not know it is wrong until someone makes a decision based on bad data" [9]. The pattern recurs in reported figures — a model stating "the Philippines' GDP growth in 2024 was 6% when the official figure... was 5.7%," right dataset, wrong number, full confidence [21].

Silent errors are dangerous because the human verification step is weak precisely where it needs to be strong. A CHI study of how analysts verify AI-assisted analysis captured an analyst who, "without fully understanding the code, deems the answer to be correct," discovering the mistake only by chance against a colleague's independent result; the study warns that AI analysis code "can be misaligned with the analyst's intent or be seemingly correct but lead to incorrect conclusions," and that even experts are susceptible to automation bias [22]. Plausibility is the trap. As a 2026 code-review analysis puts it, "AI does not write bad code — it writes plausible code" [23].

The catch-error side of the ledger is real but weaker than vendors imply. Agents do catch errors through execution feedback and self-generated tests, but self-testing has a structural flaw: a study revisiting self-debugging found that "a flawed program might pass all test cases (false positive label)," so an agent's own validation suite can silently endorse wrong analysis [24]. The self-correction loop also has diminishing returns — "noisy or verbose error feedback can confuse later self-fix iterations, diminishing returns beyond two attempts" [25]. The reliable catch mechanism remains human-in-the-loop with domain expertise. Anthropic's usage research offers the archetype: "an accountant who has never used Python, but tells Claude exactly which reconciliation rules a Python script must enforce and catches the edge case it mishandles at month-end close" [14]. The agent writes the code; the domain expert catches the edge case.

An older data point contextualizes all of this: notebooks were unreproducible before agents arrived. A 2019 study of 863,878 executions of valid notebooks found "only 24.11% executed without errors and only 4.03% produced the same results" [26]. Agents that write notebook cells inherit this substrate — hidden kernel state and out-of-order execution mean an agent-produced notebook can look valid and be silently wrong for reasons unrelated to the model.

The semantic layer as a correctness lever

The most effective structural defense against the plausible-but-wrong number, across the sources reviewed, is routing agent queries through a governed semantic layer instead of letting the agent author raw SQL. The evidence is unusually crisp because dbt Labs ran a head-to-head benchmark. In its April 2026 update, querying through the dbt Semantic Layer reached 98.2% accuracy with Claude Sonnet 4.6 and 100% with GPT-5.3-codex on in-scope questions, versus 90.0% and 84.1% respectively for text-to-SQL on the same modeled data [10]. The mechanism is the point: "the Semantic Layer's deterministic query generation means the LLM can't produce subtly wrong results" [10].

The deeper shift is in failure mode, not accuracy rate. As dbt frames it: "with text-to-SQL, failure looks like a plausible but incorrect answer. With the Semantic Layer, failure looks like an error message. For anything going to a board deck, an auditor, or a company KPI dashboard, that difference is everything" [10]. Raw text-to-SQL fails closed and quiet; the semantic layer fails open and loud. Cube makes the same argument from consistency: pointed at raw tables, "an LLM re-derives joins and metric logic on every prompt, so the same question returns different numbers," whereas over a semantic layer "it selects from certified definitions instead" [27]. Run-to-run non-determinism in the numbers is corrosive to stakeholder trust, and the semantic layer removes it by construction.

The pattern is being productized across warehouses, and governance travels with it. dbt made its remote MCP server generally available and launched dbt Agents at Coalesce 2025; the agent "uses your dbt project context to generate SQL directly from your models, then executes it in your warehouse and returns verified results with definitions, tests, and lineage" [28]. Snowflake launched managed MCP servers in public preview in October 2025, exposing Cortex Analyst and Cortex Search as tools while enforcing "the same trusted governance policies, from role-based access to masking, for the MCP server as you do for your data" [29]. Google's fully managed remote BigQuery MCP server (preview as of January 2026) is connectable from "LangGraph, Claude code, Cursor IDE, or other MCP clients," with access governed by Cloud IAM [30]. The consistent theme: warehouse vendors are not just exposing query tools, they are making the agent inherit the same permission and masking model as a human analyst, so an agent physically cannot read what its operator cannot.

Two guardrail patterns round out the toolkit. Statement-level SQL guarding: the Snowflake-Labs MCP server lets analysts configure which statement types execute, where those "marked as False will be stopped before execution" — a way to hold an agent to read-only SQL regardless of what it generates [31]. And procedural skills that encode analytical discipline: dbt's agent skills (February 2026) "transform generalist coding agents into highly capable data agents" and are explicitly designed to "protect against plausible but incorrect output," for instance instructing the agent to "preview the data before writing tests to ensure that the output matches the real data in your warehouse" [32]. The through-line is that the most reliable way to make an agent's analysis correct is not to make the model smarter but to narrow what it is allowed to author, from free-form SQL toward selection from certified definitions.

Chart production and multimodal visual feedback

Chart generation exposes the widest capability gap between tools, and the axis that matters is whether the agent can see what it drew. The academic evidence is direct. PlotGen, a multi-agent visualization system, includes a Visual Feedback Agent that "uses GPT-4V to observe the draft figure similar to how a human 'sees' the figure and provides textual feedback" on colors, layout, and aesthetics [11]. Ablating that one component is catastrophic — removing it causes "a severe performance drop (10-15%) across all code LLMs" [11] — while keeping it yields "a 4-6 percent improvement on the MatPlotBench dataset" over strong baselines [33]. A second system, CoDA, builds a Visual Evaluator that "scores rendered outputs and routes issues back to upstream agents," outperforming baselines "by up to 41.5%" on MatplotBench [34]. For charts, the render-see-critique loop is the largest single source of accuracy.

This reframes the tool comparison. Claude Code closes the loop natively — after creating a visualization it "automatically examines the resulting image and provides initial feedback," acting as both author and critic [35]. Cursor works blind: "Cursor can't 'see' the chart, but it can analyze the code and suggest enhancements (e.g., adding labels or changing styles)" [36]. Codex CLI accepts user-supplied PNG/JPEG images as reference input, but the documentation gives no indication it can visually inspect its own rendered output [37]. An analyst asking Cursor or Codex to improve a chart is asking a blind agent to improve a picture it has never seen — editing the code by pattern, not the chart by observation. This is a durable structural distinction, not a passing feature gap.

The library the agent reaches for also shapes reliability. Chart libraries with declarative JSON configuration are easier for an LLM to emit correctly in one shot than imperative ones: "an LLM can write D3 code, but the error surface is enormous" [38]. This favors matplotlib/seaborn and vega-lite/altair-style grammars for agent work, and explains why agents often succeed on a first-pass matplotlib chart but flail on hand-rolled D3. A quieter problem is that agents optimize for what they can see and neglect what they cannot: agent-generated charts routinely treat accessibility "as an afterthought," producing outputs that "look visually correct but fail screen readers, lack keyboard navigation, have insufficient contrast, or rely on color alone to convey meaning" [38]. For stakeholder-facing artifacts subject to accessibility requirements, this is a review burden the agent will not flag on its own.

The artifact target is also expanding beyond static PNGs. Anthropic's 2025 Claude Code Artifacts update "generates live HTML pages that pull from multiple data sources and update in real time," letting a data team "point a live data source at a Claude Code session and hand a monitoring URL to a non-technical stakeholder" — work that "would otherwise require a dedicated BI tool and deployment pipeline" [39]. The shift is in what an analyst can ship directly: not just a chart in a report but a live dashboard produced from a coding session. The correctness caveat still applies — a live dashboard built on a silently-wrong query is a silently-wrong dashboard, updated in real time.

Reviewability as the emerging bottleneck

Agentic tools are demonstrably good at producing stakeholder-facing artifacts end to end. A practitioner analyzed "78 years of London weather data and produced a comprehensive report with charts without writing one line of code," noting "it all ran in the background — I didn't have to look at the code at all" [40]. Another walkthrough has Claude Code "combining the text from [a markdown analysis] and all saved .png charts into a single .docx file" as the final deliverable [41]. MIT Sloan gives the pattern a name: "vibe analytics collapses this chain into an improvisational dialogue with data using generative AI tools, much like vibe coding transforms programming" [42]. The speed is real and uneven across the workflow: one practitioner estimate holds that the analysis phase accelerates roughly 5x while hypothesis generation and diagnosis barely move — "a particular weakness of the current generation of AI products" — and writing up findings resists acceleration because "writing is thinking, and that piece won't get much faster with AI" [43].

The same speed relocates the work rather than eliminating it. The 2025 Stack Overflow Developer Survey captured the tension: trust in AI accuracy fell to 29%, "down from 40%" the prior year, even as adoption climbed toward 80% of developers, and "66% of developers say they are spending more time fixing 'almost-right' AI-generated code" [12]. The 2025 DORA report generalizes this to delivery outcomes, finding that "higher AI adoption is associated with an increase in both software delivery throughput and software delivery instability" — AI as an amplifier that raises output and risk together unless strong practices are in place [13]. For analysis, the failure mode is the reviewability gap from the weather-data account: when the analyst never reads the code, no one has verified that the numbers underlying the polished report are right. The towards-AI walkthrough makes the risk concrete — without schema context, the agent "made guesses everywhere, and was forced to constantly fix its errors" [41], which is only safe because a human is watching.

The organizational response taking shape is to gate agent output through the same review machinery used for production code. Holistics' agentic-BI pattern is representative: whatever the agent changes, "it's typed AML grounded in your semantic layer, versioned in Git, and reviewable through pull requests before it reaches production" [44]. Reproducible reporting reinforces this — Quarto's July 2025 parameterized-reports feature lets teams "create a single document template, then render it multiple times with different parameter values," pairing agent-driven analysis with a deterministic, re-runnable artifact [45]. Auditability is emerging as a governance requirement because the worst agentic failures are the untraceable ones: McKinsey's 2026 trust analysis warns that "the scariest failures are the ones that cannot be reconstructed because the workflow wasn't logged" [46]. The pattern across these responses is consistent — keeping fast analysis trustworthy means making it reviewable after the fact: grounded in governed metrics, versioned, gated by human review, and logged.

Cross-tool primitives: subagents, hooks, MCP, and skills

Beneath the tool-specific branding, four primitives recur across agentic tools and account for most of what makes them useful — and safe — for analysis. The first is subagents: isolated agents "with their own context windows... each with its own system prompt, tool permissions, and even a different model" [47]. For analysis this enables a read-only data-exploration subagent that cannot mutate anything, or offloading a heavy profiling task so it does not pollute the main context. Codex offers the same primitive, with "custom agents [defined as] standalone TOML files under ~/.codex/agents/ for personal agents or .codex/agents/ for project-scoped agents" [48], showing the concept has generalized beyond any one vendor.

The second is hooks — "scripts that fire automatically at lifecycle events... where you define the event, the matcher, and the action" [47]. Hooks are the deterministic backbone of agent safety in data work because they run regardless of what the model decides. A PreToolUse hook can block a destructive statement outright: "with hooks, dangerous commands are physically blocked at the execution layer. The agent cannot bypass it, forget it, or reason its way around it" [49]. This is categorically stronger than a natural-language instruction in a CLAUDE.md file, which the model may ignore or rationalize past. For an analyst pointing an agent at a production warehouse, a hook that blocks DROP/DELETE/UPDATE is the difference between a guardrail and a suggestion.

The third primitive is MCP, covered above: it is simultaneously the kernel bridge that lets a terminal agent execute notebook cells [4] and the governed pipe to the warehouse that enforces role-based access and masking [29]. The fourth is skills — portable, reusable bundles of procedure. Anthropic's Agent Skills "run across Claude.ai, Claude Code, and the API without needing to modify for each platform" and compose automatically for multi-step workflows [50]. The dbt agent skills are the analytical application, encoding "preview the data before writing tests" as a reusable instruction the agent applies every time [32]. Codex's dataset guidance functions similarly, telling the agent to "profile the merge before performing it" by checking "uniqueness for candidate keys" and running "trial joins and report match rates" [16] — procedural knowledge that catches join errors before they happen.

The four primitives are where analytical reliability is engineered. The model provides raw capability; subagents scope it, hooks constrain it, MCP grounds it, and skills teach it discipline. A tool with all four can be made safe for real data work; one missing hooks or MCP cannot. Anthropic's usage data indicates this is not niche — within Claude Code, "writing and data analysis roughly doubled, from about 10% to 20% of sessions between October 2025 and April 2026" [14].

Tool-specific affordances

Several affordances are concentrated enough in specific tools to change what is possible. Claude Code is distinguished for analysis by native multimodal chart critique [35], the full skills/subagents/hooks/MCP stack [47], and live Artifacts dashboards [39]; its main structural weakness is the notebook-file problem, addressable via a Jupyter MCP bridge. A boundary between products matters here: the Claude.ai chat Analysis tool "has no Python runtime inside the chat UI: the sandbox uses JavaScript; Python requires API or Claude Code" [51] — genuine pandas/Jupyter work requires Claude Code, not the chat assistant.

OpenAI Codex brings two affordances for scale and safety. Its sandbox "applies to spawned commands, not just to Codex's built-in file operations," so when it runs "git, package managers, or test runners, those commands inherit the same sandbox boundaries" [52] — analysis code executes inside enforced limits by default. And Codex cloud "can work on tasks in the background (including in parallel) using its own cloud environment" [53], letting an analyst fan out several independent data tasks at once. Cursor made in-notebook editing first-class in Cursor 1.0 (June 2025): the "Agent will now create and edit multiple cells directly inside of Jupyter" [54], though the execution and blind-chart limitations above still apply. Aider is the auditability specialist: it builds a repo map via "a graph ranking algorithm... where each source file is a node and edges connect files which have dependencies" [55], and it "makes a commit for every request" [56] — a granular, revertible git history of every AI-made change.

The notebook-native tools each enable something the terminal agents cannot easily match. Google's Colab Data Science Agent (broadly available March 2025) "creates notebooks... removing tedious setup tasks like importing libraries, loading data, and writing boilerplate" from a natural-language goal, and self-reported placing "4th on the DABStep... benchmark, ahead of ReAct agents based on GPT 4.0, Deepseek, Claude 3.5 Haiku, Llama 3.3 70B" [57]. Hex's Notebook Agent chains "SQL and Python for anything from a simple query to a complex multi-step analysis" with warehouse-table context and native chart cells [2]. Deepnote's Auto AI "will not only add blocks but also execute them and self-correct its way to the desired outcome" [58]. Databricks' Genie Code (March 2026) "generates a complete Spark Declarative Pipeline with ingestion, transformations, and data quality expectations built in" [20]. The pattern: notebook-native tools win on the inside-the-kernel loop and warehouse-context grounding; terminal agents win on portability, the four primitives, git-native auditability, and integration with a broader engineering workflow. The choice follows the dominant need — reviewable, governed, engineering-integrated analysis favors the terminal agents plus MCP; fast interactive exploration against a governed warehouse favors the notebook-native agents.