Copilot can generate a fluent answer from a semantic model that is technically valid but operationally unfit for AI. Measures have ambiguous names. Similar KPIs disagree. Business definitions live in slide decks. Technical columns are exposed. Relationships produce surprising totals. Data freshness is unknown. RLS is incomplete. Users ask questions the model cannot answer and mistake a plausible response for an approved metric.
Datrick treats Copilot readiness as a data-product and evaluation problem, not a tenant-toggle exercise. AI can help document models, generate candidate instructions, propose synonyms, expand test questions, and classify failures. Accountable model owners approve business meaning, deterministic queries establish expected results, and controlled evaluation decides whether a use case is ready.
Are executives asking to enable Copilot while model owners cannot predict which answers will be correct? Start with one governed domain and a measured question set.
Define the Copilot readiness contract
Choose the users, business domain, semantic models, reports, approved question types, decisions, sensitive data, prohibited topics, languages, regions, workspaces, capacities, and rollout period in scope. Define what a correct answer requires, when Copilot should abstain, which responses need citations, who can approve verified answers, and which decisions must never rely on generated output alone.
| Readiness domain | Evidence | Question to answer |
|---|---|---|
| Business use | User personas, decisions, question catalogue, expected value, current workflow, risk, owner, and acceptance criteria. | Is Copilot solving a bounded business need or being enabled without an accountable outcome? |
| Semantic model | Measures, names, descriptions, relationships, formats, hierarchies, synonyms, hidden fields, refresh, lineage, and ownership. | Can the model express the approved business meaning consistently and unambiguously? |
| AI preparation | AI data schema, AI instructions, verified answers, excluded fields, examples, terminology, and version history. | Does the model provide relevant context without treating instructions as a correctness guarantee? |
| Security and privacy | Workspace roles, RLS/OLS, Build, sharing, labels, user groups, tenant settings, geographic processing, and Purview controls. | Can each user ask only permitted questions about data they are authorized to access? |
| Quality evaluation | Golden questions, expected queries and results, data snapshot, scoring rubric, failures, reviewers, and regression history. | Does answer quality meet the threshold for the intended decision and persona? |
| Operations and cost | Capacity prerequisites, CU consumption, telemetry, incidents, support, change, training, feedback, and rollout groups. | Can the organization operate and improve the experience without uncontrolled risk or capacity cost? |
Prepare the semantic model before writing AI instructions
Correct ambiguous measures, duplicate metrics, unclear tables, missing relationships, inconsistent formats, exposed keys, and unreliable refresh first. Use explicit measures for approved business calculations. Add human-readable names and descriptions, organize fields, hide technical artifacts, define hierarchies, and ensure model owners can explain grain, filters, time logic, currency, exclusions, and reconciliation.
Configure the AI data schema so Copilot focuses on the fields relevant to the use case. Add AI instructions for organizational terminology, analytical priorities, definitions, and important context that the model structure cannot express. Microsoft notes that AI instructions guide Copilot but might not be respected in every authoring scenario; treat them as guidance, not access policy or deterministic logic.
Use verified answers for recurring questions that require an approved response path. Define the question variants, underlying visual or query, owner, model version, data expectations, and review date. A verified answer can improve consistency for known questions but does not validate every paraphrase or adjacent analysis.
Align tenant, capacity, and compliance decisions
Review Copilot tenant settings, allowed security groups, workspace access, delegated capacity settings, supported capacities, home region, and whether data may be processed outside the capacity's geographic region, compliance boundary, or national cloud instance. Record the legal, privacy, security, and data-owner decision rather than assuming a default toggle is approved.
Microsoft states that Copilot consumes Fabric capacity units rather than a separate per-user Copilot license. Establish a usage and capacity baseline, attribute demand where possible, and monitor whether Copilot workloads affect business-critical refreshes or interactive reporting. Define budget, pilot limits, overload response, and scale criteria.
Evaluate Purview options for securing and retaining Copilot interactions when required. Decide who can access prompts and responses, how long evidence is retained, which sensitive information must be prevented or investigated, and how employee monitoring and privacy obligations are handled.
Build a golden-question evaluation harness
| Evaluation dimension | Test method | Failure example |
|---|---|---|
| Calculation correctness | Compare measures, filters, period, grain, totals, rounding, and result with an approved deterministic query. | Correct narrative built on the wrong revenue measure or date context. |
| Grounding and traceability | Verify referenced report, visual, fields, model, data cutoff, and explanation support the answer. | Answer cannot identify which source or model result supports its claim. |
| Security | Run the same question across RLS personas, elevated roles, prohibited topics, and attempted prompt manipulation. | Restricted user receives a total that reveals another region or customer. |
| Relevance and interpretation | Test business synonyms, ambiguous questions, follow-ups, unsupported requests, and expected clarification. | Copilot chooses bookings when the business term “sales” means recognized revenue. |
| Uncertainty and refusal | Ask questions outside model scope, with missing periods, stale data, contradictory instructions, or no approved measure. | Confident answer instead of clarification, limitation, or safe abstention. |
| Reproducibility and regression | Repeat questions against a fixed model and snapshot, then rerun after model, instruction, verified-answer, or service changes. | Material answer changes without data or approved business-logic change. |
| Utility | Have target users score clarity, actionability, time saved, correction effort, and decision confidence. | Answer is technically accurate but too vague or verbose to support the workflow. |
Version every test with persona, prompt, expected interpretation, approved measure, filters, date, data snapshot, expected result, acceptable narrative, prohibited content, and scoring threshold. Separate deterministic facts from subjective answer quality. AI can expand paraphrases and classify errors; it must not grade its own output as the only evaluator.
Investigate failures at the correct layer
An incorrect answer can originate in source data, refresh, transformation, semantic logic, metadata, AI data schema, instructions, verified answers, user ambiguity, security, or service behavior. Preserve the prompt, persona, model and configuration version, data cutoff, generated query or cited visual when available, response, expected result, and reproduction steps.
Fix the durable layer. Do not compensate for an incorrect revenue measure by writing longer prose instructions. Do not expose a restricted field to improve answer quality. Do not create hundreds of verified answers when the model lacks consistent business measures. Track recurrence and regression after every change.
Roll out by use case and risk
- Enable only named pilot groups and approved workspaces after tenant, region, privacy, security, and capacity decisions.
- Train users on supported questions, data cutoff, citations, limitations, sensitive data, feedback, and decisions requiring independent verification.
- Provide an in-product or operational route to report wrong, unsafe, stale, or confusing answers with the necessary context.
- Monitor usage, CU consumption, model coverage, answer failures, security incidents, corrections, adoption, time saved, and unsupported demand.
- Review instructions, schemas, verified answers, golden questions, owners, sensitive fields, and user groups on a defined cadence.
- Expand only when quality, security, support, and capacity thresholds are met for the next domain.
Do not use adoption volume as the only success metric. A lower-volume use case that saves analysts hours with measurable correctness can be more valuable than broad experimentation that creates verification work and decision risk.
Run a three-to-five-week readiness assessment and pilot
- Define target users, decisions, business domain, models, questions, risks, prohibited uses, stakeholders, and measurable outcomes.
- Review tenant, capacity, workspace, region, security, privacy, Purview, data handling, and support prerequisites.
- Assess semantic models for measure quality, metadata, relationships, security, refresh, ownership, lineage, and AI preparation.
- Remediate a bounded model and configure AI data schema, instructions, and representative verified answers.
- Create deterministic golden questions, security negatives, ambiguity tests, abstention cases, and user utility scenarios.
- Run the evaluation against fixed data and personas, classify failures, and implement one controlled improvement cycle.
- Pilot with a named user group while monitoring quality, capacity, feedback, incidents, and unsupported demand.
- Deliver the readiness scorecard, model backlog, evaluation suite, governance decisions, rollout gates, training, support runbooks, and scale recommendation.
Frequently asked questions
How do you prepare a Power BI semantic model for Copilot?
Start with a correct, governed semantic model: clear business names and descriptions, explicit measures, valid relationships, hidden technical fields, appropriate formats, synonyms, security, ownership, and reliable refresh. Then configure the AI data schema, AI instructions, and verified answers for supported scenarios. Test representative business questions and prohibited questions with known expected results before rollout.
What is included in a Power BI Copilot readiness assessment?
The assessment covers business use cases, semantic-model quality, measures and metadata, AI data schema, AI instructions, verified answers, data freshness, security, tenant and capacity settings, regional and compliance decisions, Purview options, user groups, answer-quality evaluation, cost and capacity consumption, training, support, monitoring, and phased rollout controls.
Do AI instructions guarantee correct Power BI Copilot answers?
No. AI instructions provide context and guidance, but they are not a deterministic policy or correctness guarantee. Important answers should be evaluated against approved business definitions and deterministic queries. Use verified answers for known questions where appropriate, preserve citations and query evidence, and require users to validate material decisions.
How do you test Power BI Copilot answer quality?
Create a versioned golden-question set with expected measures, filters, time periods, security personas, acceptable answer forms, prohibited content, and abstention criteria. Run it against a fixed model and data snapshot, score factual and calculation correctness, grounding, security, relevance, reproducibility, and safe uncertainty, investigate failures, and repeat after model or configuration changes.
How long does a Power BI Copilot readiness assessment take?
A focused assessment and controlled pilot can often be completed in three to five weeks for one business domain when the semantic model, owners, security roles, target questions, capacity, and tenant decisions are available. Remediation takes longer when measures are inconsistent, metadata is weak, access is excessive, models are duplicated, data is unreliable, or business definitions lack accountable owners.
Official implementation references
- Microsoft Power BI guidance for AI instructions
- Microsoft overview of Copilot for Power BI reports and semantic models
- Microsoft guidance for enabling Copilot in Fabric
- Microsoft overview of how Copilot in Fabric works
- Microsoft Purview guidance for Copilot in Fabric interactions
Start with one governed semantic model and the questions users already ask analysts. Datrick can prepare the model, build the evaluation suite, run a controlled pilot, and define rollout gates.