A creator assistant can generate polished instructions and runnable example queries faster than a team can agree on the business meaning of revenue, active customer, incident, or fiscal period. The dangerous failure is not malformed SQL or KQL. It is a syntactically valid configuration that encodes the wrong grain, join, time rule, status, source priority, or exception and then improves confidence in the wrong answer.
Build agent with AI is a preview mode for SQL and Eventhouse sources. It can explore schema, inspect query history when the creator has permission, generate four configuration artifacts, and execute read-only queries. Microsoft explicitly recommends reviewing suggestions before acceptance because accepted updates replace existing configuration blocks. Treat it as a configuration accelerator inside an evidence-based build process.
Which critical questions improved, regressed, or remained unproven after the AI-generated configuration? Review that evidence before adding more instructions.
Define what the assistant may change
Establish the supported Data Agent domain, source types, selected schema, user cohort, critical questions, business owners, current baseline, evaluation threshold, and release owner. Build agent with AI does not select data sources or schema, publish the agent description, or configure semantic models, ontology, Graph, or unstructured data. If the Data Agent includes an unsupported source, the current preview mode does not work.
| Artifact | Assistant contribution | Required human evidence | Failure to prevent |
|---|---|---|---|
| Agent Instructions | Propose routing, global terminology, response rules, and source priority. | Approved cross-source decision rules and prohibited behavior. | Global instruction overrides the intended source-specific meaning. |
| Data Source Instructions | Describe tables, columns, joins, filters, grain, time, and query patterns. | Data contract, semantic map, authoritative queries, and owner sign-off. | Plausible join or filter becomes a durable wrong-answer rule. |
| Data Source Description | Summarize content and intended use to support source routing. | Domain boundary and authoritative source catalogue. | Overbroad description routes unrelated questions to the source. |
| Example Queries | Generate natural-language and SQL or KQL pairs from schema and patterns. | Question intent, approved query, result, grain, literals, and conflict review. | Few-shot teaches a conflicting metric or hidden exception. |
| Configuration update | Replace accepted configuration blocks in the current preview. | Before state, diff, draft test, regression result, approval, and rollback. | Useful existing behavior disappears after an unreviewed acceptance. |
Control schema and query-history evidence
Select only the tables and columns required for the question set before invoking the assistant. Microsoft recommends focused domain agents and a limited source scope; Build agent with AI does not perform schema selection for you. Clear names and descriptions, stable keys, explicit grain, date roles, status values, units, joins, exclusions, and source ownership produce better suggestions than a broad ambiguous schema.
Query history can reveal common joins, filters, and aggregation patterns, but history is evidence of usage, not correctness. It can contain exploratory queries, incidents, deprecated logic, bypasses, one-off fixes, or access patterns that should not become product behavior. Review query authorship, recency, source version, performance, business approval, and exceptions before turning a recurring pattern into an instruction or few-shot.
Access to query history requires the corresponding source permission. When history is unavailable, Microsoft says the assistant falls back to schema-driven recommendations. Record which evidence mode generated each proposal. A schema-only suggestion should not inherit the confidence of a pattern backed by approved production queries.
Review every proposed configuration change
Capture the current draft configuration before the session. Ask the assistant to explain the evidence, source entities, intended questions, generated query pattern, assumptions, and known exceptions behind each proposal. Reject generic instructions, duplicated rules, negative-only guidance, contradictions, unsupported source behavior, and examples that do not match real user language.
For SQL few-shots, use Microsoft's evaluation utility where available to inspect Clarity, Relatedness, Mapping, and conflict detection. The utility is currently documented for SQL examples, not KQL or other query types. Independent deterministic checks remain necessary: execute every query, compare results and grain with approved ground truth, and inspect whether literals, date windows, joins, filters, and aggregations match the question.
Accept changes only in the draft version, then compare draft with the published baseline. Because accepted suggestions can replace configuration blocks, keep a versioned before state and a tested rollback. Do not rely on chat history as the system of record for why a business rule changed.
Measure configuration impact end to end
Build a representative question set before applying suggestions. Include common questions, high-risk metrics, ambiguous terms, unsupported requests, no-data periods, rare values, permission boundaries, prompt manipulation, and known failure cases. Preserve the expected source, query, result, answer, clarification, refusal, latency, and severity.
| Dimension | Test | Pass condition | Remediation |
|---|---|---|---|
| Instruction fidelity | Business terms, source purpose, positive rules, exceptions, precedence, and contradictions. | Configuration encodes the approved contract without overlap. | Split global and source rules, simplify, or remove unsupported assumptions. |
| Few-shot quality | Clarity, relatedness, literal mapping, schema validity, query result, uniqueness, and conflicts. | Each pair teaches one correct representative pattern. | Rewrite the question/query, add comments, remove overlap, or reject the example. |
| Source routing | Questions that could map to multiple SQL or Eventhouse sources. | Agent consistently chooses the authoritative source or asks for clarification. | Source descriptions, routing instructions, or domain separation. |
| Query correctness | Join, grain, time, filter, aggregation, null, distinct, value, and performance. | Generated query and result match deterministic ground truth. | Schema, instruction, few-shot, governed view/function, or use-case boundary. |
| Security | Creator permissions, query history, consumer identities, restricted schema, and adversarial prompts. | Assistant evidence and published answers remain within approved access. | Permission correction, source separation, logging control, or stop rule. |
| Regression | Published baseline versus changed draft over the full question suite. | Critical accuracy improves without unacceptable new failures. | Revert the block, isolate the change, or refine and retest. |
| Operations | Latency, errors, drift, source changes, feedback, release, and rollback. | Configuration has an owner, threshold, monitor, and recovery path. | Release gate, scheduled review, runbook, or managed support. |
Classify every failure by source data, schema selection, instruction, description, few-shot retrieval, generated query, permission, runtime, or final answer. This prevents the team from adding more prompt text to a source or identity problem. Track which suggestions were accepted, their evidence, evaluation delta, approver, release, and rollback version.
Run a two-to-four-week configuration review
- Select one SQL or Eventhouse Data Agent, business domain, user cohort, critical question set, owners, and measurable accuracy target.
- Capture the published and draft configuration, selected schema, permissions, source contracts, current failures, and baseline evaluation.
- Review schema and query-history quality; identify authoritative patterns, deprecated logic, conflicts, sensitive evidence, and unsupported sources.
- Use Build agent with AI to explore, learn, generate, and execute while preserving each proposal, evidence, assumption, and proposed diff.
- Validate instructions and descriptions; execute and score few-shots; run deterministic query, permission, routing, answer, and adversarial tests.
- Apply approved changes only to draft; compare with the published baseline, classify regressions, remediate, and prove rollback.
- Deliver the reviewed configuration, few-shot catalogue, evaluation scorecard, decision log, release gate, runbook, and prioritized backlog.
Frequently asked questions
What does Build agent with AI do in Microsoft Fabric Data Agent?
Build agent with AI is a preview creator assistant that can explore supported SQL or Eventhouse sources, inspect query history when permissions allow, propose Agent Instructions, Data Source Instructions, Data Source Descriptions, and example queries, and execute read-only queries during configuration.
Which data sources support Build agent with AI?
Microsoft's current preview documentation limits Build agent with AI to SQL and Eventhouse data sources. It doesn't support semantic models, ontology, Graph, or unstructured data, and it doesn't work when unsupported sources are added to the same Data Agent.
Does Build agent with AI validate business correctness?
It can run read-only queries and help inspect schema and query patterns, but a successful query doesn't prove the intended metric, grain, time rule, join, permission behavior, or natural-language answer. Business owners and data engineers still need deterministic ground truth and end-to-end regression tests.
Can Build agent with AI overwrite existing Data Agent configuration?
Microsoft warns that accepted suggestions replace the applicable existing configuration blocks during the current preview. Capture the before state, review the proposed diff, test in draft, and keep a rollback path before accepting changes.
How long does a Build agent with AI configuration review take?
A focused review commonly takes two to four weeks for one Data Agent, one supported business domain, and a representative question set. It covers source readiness, permissions, current configuration, generated suggestions, few-shot quality, regression testing, release evidence, and an improvement backlog.
Official implementation references
- Build agent with AI capabilities and limitations
- Example-query validation and conflict detection
- Data Agent configuration best practices
- Source-specific configuration support
- Draft, published, sharing, and permission behavior
Start with the current configuration, the questions it must answer, and the wrong answers it already produces. Datrick can turn the assistant's proposals into reviewed changes, measurable regressions, and a controlled release decision.