A semantic model can contain certified measures and still produce a wrong conversational answer. The agent may choose an overlapping measure, apply the wrong date, miss a relationship, aggregate a numeric column implicitly, retrieve an obsolete verified answer, or generate valid DAX at the wrong business grain. Fluency does not reveal which failure occurred.
Fabric Data Agent uses an orchestrator to select the semantic model and a specialized DAX tool to generate, validate, and execute the query. Microsoft states that this DAX tool relies on model metadata and Prep for AI configuration, not semantic-model instructions written at the Data Agent level. Accuracy therefore depends on the model, AI Data Schema, Verified Answers, AI Instructions, generated DAX, user identity, and release process working as one controlled system.
Can your team show the DAX and deterministic expected result behind every high-value agent answer? If not, start with a bounded question set before adding more users.
Test every layer of the NL2DAX path
Separate source routing from DAX accuracy. First prove that the orchestrator selected the intended semantic model. Then inspect the generated DAX, execute an independently approved control query against the same snapshot and identity, compare results, and verify that the final response preserved filters, units, freshness, grain, uncertainty, and truncation limits.
| Layer | Evidence to capture | Typical failure | Control |
|---|---|---|---|
| Route | Selected source, competing sources, agent instructions, runtime, and conversation state. | A raw SQL source wins over the governed semantic model. | Explicit source contract and route-level expected result. |
| Model context | Tables, columns, measures, relationships, descriptions, synonyms, formats, and report visual metadata. | Similar measures or ambiguous dates lead the tool toward the wrong object. | Business naming, explicit measures, star schema, focused metadata, and dependency review. |
| Prep for AI | AI Data Schema, AI Instructions, Verified Answers, trigger questions, filters, and model version. | Schema and agent selection diverge, or instructions conflict with verified behavior. | Versioned configuration with single-purpose changes and regression gates. |
| Generated DAX | Query text, referenced measures, filters, date context, grouping, blank behavior, and execution time. | Valid DAX returns a plausible value at the wrong grain or time period. | Independent control DAX and component-level reconciliation. |
| Effective access | User identity, model Read permission, RLS role, CLS restrictions, and result differences. | Testing under an author identity hides a consumer-specific denial or data leak. | Persona matrix with positive, negative, and role-boundary tests. |
| Answer | Value, units, labels, freshness, limitations, latency, source attribution, and conversation history. | The result is correct but summarized, relabeled, or qualified incorrectly. | Answer rubric and clean-session versus follow-up comparison. |
Configure Prep for AI where the DAX tool can use it
Define the Data Agent's supported questions first, then configure a focused AI Data Schema in Prep for AI. Select only relevant tables, columns, and explicit business measures, including required dependent objects. When adding the semantic model to Data Agent, select the same tables. A smaller schema reduces ambiguity and latency; an incomplete schema can silently remove the objects needed by a measure.
Use Verified Answers for common or complex questions with stable, approved interpretations. Microsoft recommends multiple natural trigger variations and supports limited filter flexibility. In Data Agent, the verified visual is not returned directly; its question and visual properties guide DAX generation. Test exact prompts, semantic paraphrases, near-misses, stale field names, changed filters, and questions that must not match.
Add AI Instructions only after baseline testing shows a reproducible gap. Keep them specific: approved metric preference, fiscal period definition, default grouping, terminology, and bounded analysis rules. Instructions are interpreted rather than guaranteed, so test them. Conflicting or complex guidance can reduce predictability and increase latency.
Do not duplicate semantic logic in Data Agent instructions. Microsoft documents that the semantic-model DAX tool ignores those instructions. Reserve agent-level instructions for cross-source routing, shared abbreviations, response format, tone, and behavior that applies to every attached source.
Build a ground-truth suite for DAX and business meaning
Start from real analyst and executive questions, not synthetic prompts alone. For each case, record intent, persona, model version, data snapshot, expected measure, dimensions, filters, date interpretation, expected DAX or deterministic result, acceptable phrasing, prohibited behavior, severity, and latency target. Add paraphrases only after the base case passes.
| Test family | Example risk | Pass condition | Remediation layer |
|---|---|---|---|
| Metric selection | Revenue, net sales, gross sales, and bookings all appear relevant. | Approved explicit measure is used or the agent asks a useful clarification. | Model naming, AI schema, instruction, or verified answer. |
| Date and comparison | Calendar versus fiscal period, order versus ship date, partial period, or prior-year baseline. | Generated DAX uses the approved date and comparison logic. | Date model, measure, instruction, verified answer, or question scope. |
| Grain and filter | Customer, account, product, region, hierarchy, Top N, or hidden default filter. | Rows, totals, filters, and ranking match the control query. | Relationships, metadata, schema, visual grounding, or DAX measure. |
| Verified Answer retrieval | Paraphrase misses, unrelated prompt matches, renamed field, or outdated visual filter. | Expected questions match; near-misses do not; generated DAX remains correct. | Triggers, filters, visual, field references, or verified-answer redesign. |
| Instruction adherence | Business term, preferred measure, default period, or conflicting guidance. | Target behavior improves without regressions in unrelated cases. | Simplify, relocate, split, or remove the instruction. |
| Security | Two users ask the same question under different RLS or CLS boundaries. | Each answer matches that user's effective model access and fails closed. | Permission, role, model security, source sharing, or rollout scope. |
| Conversation state | A follow-up inherits the wrong date, metric, filter, or summarized 25-row context. | Context is preserved only when intended; a new chat resets the result. | Clarification, conversation design, answer limit, or usage guidance. |
| Performance | Large model, inefficient measure, high-cardinality field, or generated DAX timeout. | Correct answer meets the approved latency and capacity envelope. | Model optimization, schema reduction, DAX tuning, or use-case limit. |
Score route accuracy, DAX validity, numeric correctness, semantic correctness, security correctness, answer fidelity, latency, and abstention separately. A single pass rate hides whether remediation belongs in the model, Prep for AI, the Data Agent, access design, or the user experience. Preserve generated DAX and run evidence so each failure has an accountable owner.
Prove consumer permissions and controlled release
Current Microsoft documentation says Data Agent users need Read permission on the semantic model, not Build or workspace access, and that RLS and CLS remain enforced. Build or Write is required for model changes and Prep for AI authoring. Test with actual consumer identities because author permissions can conceal empty results, authorization failures, or excessive access.
Keep a published baseline and an isolated draft. Change one layer at a time, compare the same question suite, inspect generated DAX, and block promotion when a critical metric, permission boundary, or latency target regresses. Include model refresh, deployment pipeline, renamed objects, verified-answer updates, runtime changes, and report visual metadata changes in release impact analysis.
Monitor wrong-answer feedback, no-answer rate, verified-answer match rate, DAX execution failures, authorization errors, latency, capacity, model freshness, route changes, question drift, and configuration releases. Preserve rollback to the last approved model and Data Agent version. A correct semantic model is not a permanent guarantee of correct NL2DAX after metadata, configuration, data, or runtime changes.
Run a three-to-five-week NL2DAX accuracy assessment
- Select one semantic model, one Data Agent, target users, business owners, critical metrics, and a representative production question set.
- Map model design, measures, metadata, relationships, refresh, report visuals, AI Data Schema, AI Instructions, Verified Answers, permissions, runtime, and current failures.
- Define deterministic controls for route, expected measure, dimensions, filters, date logic, DAX result, security persona, answer wording, and latency.
- Run baseline tests before remediation; inspect generated DAX and classify failures by route, model, Prep for AI, query, permission, answer, or performance.
- Refine the smallest responsible layer, rerun the full suite, and compare draft with published behavior under real consumer identities.
- Test paraphrases, ambiguity, unsupported questions, permission boundaries, stale data, model changes, conversation history, timeouts, and runtime changes.
- Deliver the model-readiness findings, Prep for AI configuration, NL2DAX scorecard, ground-truth suite, permission matrix, release gates, runbooks, and go, limit, or stop recommendation.
Frequently asked questions
How does Fabric Data Agent query a Power BI semantic model?
Fabric Data Agent routes a question through its orchestrator to a specialized DAX generation tool. That tool uses semantic model metadata, Prep for AI configuration, and conversation context to generate, validate, and execute DAX before the agent formats the result.
Where should Power BI semantic model instructions for Fabric Data Agent be configured?
Microsoft says semantic-model-specific instructions must be configured in Power BI Prep for AI. The DAX generation tool ignores semantic-model instructions placed at the Data Agent level. Agent instructions should cover only cross-source routing and behavior that applies to the whole agent.
Do Verified Answers work with Fabric Data Agent?
Yes. Verified Answers are stored on the semantic model and can guide the DAX generation tool when a Data Agent question exactly or semantically matches a trigger. The agent does not return the original Power BI visual; it uses the question and visual properties to influence DAX generation.
What permissions are required to query a semantic model through Fabric Data Agent?
Current Microsoft documentation states that Read permission on the semantic model is sufficient for Data Agent queries and workspace or Build permission is not required for interaction. RLS and CLS still apply. Build or Write permissions are needed to modify the model or configure Prep for AI.
How long does a Fabric Data Agent NL2DAX accuracy assessment take?
A focused assessment commonly takes three to five weeks for one semantic model and a representative business question set. It covers model readiness, Prep for AI configuration, generated DAX, deterministic reconciliation, user security, latency, draft-versus-published comparison, and release controls.
Official implementation references
- Microsoft semantic model best practices for Fabric Data Agent
- Power BI Prep data for AI overview
- Power BI AI Data Schema guidance
- Power BI Verified Answers guidance
- Power BI AI Instructions guidance
- Fabric Data Agent semantic model permissions
Start with the questions where the generated DAX is valid but the business answer is wrong. Datrick can connect semantic-model remediation, Prep for AI configuration, deterministic controls, consumer security, and release evidence into one measured assessment.