A chart can be technically valid and analytically wrong. The source query may omit rows, the Python may aggregate at the wrong grain, a correlation may be described as causation, missing values may disappear, a forecast may ignore seasonality, or a visual may hide the unit and time range that make the result meaningful. A sandbox does not detect these business failures.
Microsoft's preview Code Interpreter tool gives Fabric Data Agent a secure, sandboxed Python environment. The agent queries its governed sources, passes retrieved results to the tool, and generates Python for analysis, calculations, forecasting, or visualizations. Run steps expose the generated code, inputs, outputs, and charts. That evidence makes systematic validation possible, but only when each layer has an explicit contract and independent control.
Can an analyst reproduce the chart from the exact rows, columns, filters, and Python shown in run steps? If not, the result is not ready to influence a business decision.
Treat the result as a multi-stage analytical pipeline
Define when Code Interpreter should be used, which source must answer first, what data must be transferred, what analysis is allowed, and when the agent should refuse or escalate. Microsoft currently does not support tool-specific instructions; use agent-level instructions to guide invocation, context, and final formatting, then test whether those instructions work across adjacent questions.
| Layer | Evidence | Failure mode | Control |
|---|---|---|---|
| Intent and route | User question, conversation state, chosen source and tools, and invocation order. | Python is used when a governed measure or simple query should answer directly. | Use-case contract, routing tests, clarification, and no-tool cases. |
| Source query | Generated SQL, DAX, KQL, GQL, filters, permissions, result size, and freshness. | Incomplete or unauthorized input reaches a mathematically correct script. | Independent source query, persona tests, completeness check, and snapshot. |
| Python input | Rows, columns, data types, units, missingness, ordering, sampling, and truncation. | The script analyzes a partial, mislabeled, coerced, or stale dataset. | Input manifest, row reconciliation, schema assertions, and limit warnings. |
| Generated code | Libraries, transformations, formulas, parameters, random seed, and exceptions. | Wrong denominator, leakage, unstable sampling, silent coercion, or unsupported model. | Code review, deterministic reference implementation, and adversarial fixtures. |
| Output and visual | Computed values, chart type, axes, labels, units, scale, sorting, confidence, and image. | Misleading encoding or narrative overstates the evidence. | Analytical rubric, approved chart, accessibility, and causality rules. |
| Final response | Summary, caveats, source attribution, freshness, limitations, and recommended action. | Correct output becomes an overconfident or incomplete business conclusion. | Response contract, human review, and prohibited-decision boundary. |
Build ground truth for calculations, forecasts, and visuals
Create representative tasks for descriptive statistics, ratios, rankings, correlations, segmentation, anomaly detection, time-series transformations, forecasting, and charting. For each task, preserve the source snapshot, expected input, reference Python or analytical control, expected result, tolerance, chart specification, required caveats, user persona, latency target, and severity.
| Test family | Risk | Pass condition | Required challenge |
|---|---|---|---|
| Aggregation | Wrong denominator, duplicate rows, weighted versus unweighted mean, or total mismatch. | Intermediate and final values reconcile with the reference calculation. | Duplicates, zero denominator, nulls, negative values, and mixed grain. |
| Correlation | Pairwise deletion, outliers, nonlinear relationship, or causal language. | Method, population, missing-data rule, coefficient, and limitation are correct. | Confounding, constant column, sparse segment, and Simpson's paradox. |
| Forecast | Leakage, insufficient history, ignored seasonality, unstable extrapolation, or false precision. | Training window, method, horizon, uncertainty, backtest, and limitations are disclosed. | Regime change, missing periods, structural break, and naive-baseline comparison. |
| Visualization | Wrong chart, truncated categories, misleading scale, missing units, or hidden filter. | Encoding matches the data and question; labels and caveats preserve meaning. | Long tail, negative values, dual scale, unsorted time, and inaccessible colors. |
| Permission | Author testing hides consumer RLS, CLS, source denial, or filtered result. | Input and output match each persona's effective access and fail closed. | Allowed, restricted, denied, and permission-revoked identities. |
| Reproducibility | Nondeterministic code, random sampling, library change, or context-dependent result. | Approved tolerance holds across clean sessions and controlled repeated runs. | Repeat run, new chat, reordered rows, runtime change, and seed variation. |
| Failure behavior | Timeout, empty input, code exception, unsupported analysis, or blocked source. | Agent states the limitation and does not fabricate a chart or conclusion. | Empty, malformed, oversized, stale, and out-of-scope inputs. |
Score source correctness, input completeness, code correctness, numeric correctness, visual fidelity, explanation fidelity, permission behavior, latency, and abstention separately. Preserve run steps with every result. A final-answer score alone cannot tell whether to fix routing, source configuration, data shape, Python, instructions, or the user-facing response.
Define what the sandbox does and does not control
Sandboxing constrains Python execution, but the business risk still depends on which governed source results enter the tool and what the generated output reveals. Test with actual identities and verify that source permissions, RLS, CLS, Purview restrictions, and data-agent sharing behavior remain effective before the tool receives the result.
Classify allowed analytical tasks and data domains. Exclude or require approval for sensitive attributes, small cohorts, protected-class analysis, high-impact forecasts, compliance decisions, automated personnel or credit decisions, and outputs that could expose restricted values through derived statistics. Define minimum cohort sizes, output suppression, human review, and escalation paths.
Review cross-geo processing and storing settings, interaction auditability, diagnostics access, retention, eDiscovery, support evidence, and preview risk with security and legal owners. Do not infer Fabric Code Interpreter's exact isolation or retention behavior from a different Microsoft 365 Code Interpreter product; validate the controls documented and configured for the Fabric deployment.
Control preview changes, latency, capacity, and client differences
Code Interpreter is a preview feature. Maintain a published baseline, versioned instructions, task suite, run-step evidence, and rollback procedure. Compare runtime, tool, model, source, and configuration changes with identical inputs. Monitor tool invocation rate, source-query failures, Python exceptions, wrong-chart feedback, latency, capacity, repeated retries, and high-cost task patterns.
Data Agent consumption includes AI Query input and output tokens, while generated source queries consume the corresponding engine separately. Treat Code Interpreter latency and capacity as an empirical workload dimension: measure end-to-end response time and Fabric utilization for representative tasks rather than estimating from prompt length alone.
Keep native Fabric Data Agent visual responses separate from Code Interpreter visuals. Microsoft documents native chart types and a 200-row visualization limit in the Fabric Data Agent experience, while Code Interpreter follows a different tool path and can also be used through Microsoft 365 Copilot. Test the exact client, input path, and output limit that users will experience.
Run a three-to-five-week Code Interpreter assessment
- Select one Data Agent, target clients, analytical tasks, data domains, personas, decision owners, security owners, and prohibited uses.
- Map source routes, query tools, permissions, input transfer, instructions, Code Interpreter run steps, client behavior, preview dependencies, capacity, and current failures.
- Create deterministic source, Python, numeric, visual, explanation, permission, latency, and abstention ground truth with production-shaped snapshots.
- Run baseline, missing-data, truncation, duplicate, outlier, leakage, causality, prompt-ambiguity, permission, exception, timeout, and repeated-session tests.
- Classify every failure by route, source query, input, generated Python, output, visualization, response, permission, platform, or test control.
- Refine agent instructions and use-case boundaries, rerun the full suite, compare draft and published behavior, and prove rollback.
- Deliver the task contract, risk register, evaluation suite, Python and chart scorecards, permission matrix, monitoring, release gates, runbooks, and go, limit, or stop recommendation.
Frequently asked questions
What does Code Interpreter do in Microsoft Fabric Data Agent?
Code Interpreter gives a Fabric Data Agent a secure, sandboxed Python environment. The agent queries its connected sources, passes retrieved results to the tool, and generates Python for analysis, mathematical calculations, forecasting, or visualizations. Generated code, inputs, outputs, and charts are visible in run steps.
Can Fabric Data Agent Code Interpreter be given its own instructions?
Microsoft currently says instructions can't be added directly to the Code Interpreter tool. Agent-level instructions can guide when the agent should invoke it, what context to pass, and how the final response should be formatted.
How should generated Python and charts be validated?
Capture the selected source, input rows and columns, generated Python, library behavior, calculations, chart encoding, output, and narrative. Compare each layer with an independently approved query or analytical control and test missing values, truncation, units, filters, grain, outliers, and unsupported causal claims.
Does the Code Interpreter sandbox make the analysis automatically safe?
No. Sandboxing limits the execution environment, but teams must still validate effective source permissions, sensitive inputs, generated analysis, misleading outputs, retention and audit requirements, preview changes, latency, capacity, and which decisions require human review.
How long does a Fabric Data Agent Code Interpreter assessment take?
A focused assessment commonly takes three to five weeks for one agent and a bounded set of analytical tasks. It covers route and input contracts, Python and visualization ground truth, permission personas, adversarial tests, latency and capacity, instructions, monitoring, release gates, and rollback.
Official implementation references
- Microsoft Fabric Data Agent Code Interpreter guidance
- Fabric Data Agent visual responses and limitations
- Fabric Data Agent operation, governance, and limitations
- Fabric Data Agent source permissions and sharing
- Fabric Data Agent consumption and monitoring
Start with the chart or forecast users are most likely to trust without checking. Datrick can reconstruct its source-to-Python evidence, build deterministic controls, test security and failure behavior, and convert preview functionality into a bounded production decision.