With Microsoft’s flagship AI tool Copilot now integrated into Power BI, users can request analysis, navigate reports and generate visuals using natural language, all without relying on an analyst. That raises a practical question for many teams: should we be using it, what are the limitations right now, and what do we need in place to get value from it?

Thorogood recently worked with a client, a global consumer health company, to evaluate Copilot in Power BI as an extension of a comprehensive solution we have developed and is in active use at the customer. The project gave us early insights into where Copilot performs well today, where it still struggles, and how to think about the familiar trade-off in AI between out-of-the-box offerings and more bespoke solutions.

Testing analytics, visuals and navigation

The goal of the project was to assess Copilot’s performance in three areas: analytical Q&A (using natural language to retrieve insights quickly), navigation (helping users locate relevant content within a report), and visual generation (creating charts dynamically based on the available data).

We tested Copilot in two configurations: a base setup and an enhanced setup with AI instructions. In the enhanced version, we introduced instructions iteratively alongside prompt refinements and light data model simplification. Across both configurations, we assessed performance through the lens of accuracy, consistency and usability.

Asking the right questions

One of the biggest advantages of out-of-the-box tools is speed: you can start quickly, and Copilot in Power BI is no exception. When queries were clearly framed, it could provide accurate summaries, point users to relevant pages, and generate simple visuals. But the client found it still struggled with understanding the intent of requests and a lack of citations also reduced trust. It’s crucial to provide guidance on how to ask effective questions and what Copilot can, and cannot, reliably do today.

Overcoming current limitations

Copilot in Power BI still has inherent limitations. Repeatability is a key issue – the same query can produce different answers. We also saw inconsistencies in how Copilot handled time periods and slicers, which affected its responses and the visuals it generated. Understanding hierarchical data was another common challenge.

However, we were able to improve performance with some iterative, custom pre-prompting using the tool’s native features. By introducing these targeted AI instructions to clarify ambiguous terms (for example, what “sales” means in a specific context), apply consistent time logic, and better detect navigational intent, we saw a meaningful improvement in quality and reliability. Simplifying parts of the data model also helped Copilot interpret the available measures and relationships more consistently.

When to go bespoke

So, should your organization be using Copilot?
In many cases, yes. Copilot can add immediate value, provided users are shown how to request information effectively and the data model and AI instructions are thoughtfully applied. But current limitations mean it is not yet fully reliable, and it lacks many of the features needed for broad, productionized use.

This is where a more bespoke, API-based solution may be worth considering. A tailored approach can address common shortcomings by supporting follow-up questions and conversational context, learning from how users interact with the tool over time, and giving teams better visibility through usage tracking and feedback loops that drive targeted improvements.

Scalability is also a factor. An API-based solution can integrate multiple data sources, optimize response times, and provide a custom interface aligned to how your team’s work.

While more of an aesthetic preference, bespoke solutions also bring with them the advantage of custom design elements; presently Copilot for Power BI has a standard look and feel that cannot be altered, it exists in one form within the Power BI application.

Copilot in Power BI is best seen as a jumping-off point: useful for simple requests today, and a practical way to build organizational familiarity with natural language interfaces. For teams looking to deliver a more robust, scalable and user-centric experience, a bespoke solution can take those capabilities further and align more directly with a long-term Data and AI strategy.