Building GPT-boosted chatbots with Copilot Studio

The recent drama at OpenAI highlighted one thing: the enormous bet Microsoft has made on GPT and its derivatives, using them to power its growing family of Copilots. The generative AI capabilities of GPT and similar large language models (LLMs) are key components of these natural language interfaces, providing language understanding and summaries, as well producing user-centric text outputs that can be tuned to user needs and preferences.

Now that Microsoft has rolled out an army of Copilots across its various properties, the next question is clearly how do we get to build these into our own code? Last week I looked at one option, the prompt flow-based Azure AI Studio, which provides direct access to Microsoft’s growing library of machine learning models. This week I look at a second, Microsoft’s low-code AI development tool, Copilot Studio.

AI for the rest of us

Azure AI Studio is clearly a tool for experienced application developers and for data scientists. But what about the rest of us? What do we use when we want to go beyond using canned models? When we want to build LLM-powered applications that work with our own data and our own business requirements?

Low-code and no-code development tools like Microsoft’s Power Platform are the modern equivalent of Visual Basic, designed to deliver applications that work with your business data. Power applications run on desktop or mobile clients, or on the web. They can be used as process automation tools, providing ad hoc links between applications. The Power Platform’s library of connectors and actions includes support for many of Microsoft’s machine learning services, including a basic customer service chat platform that can be quickly trained and added to your web presence.

These Power Virtual Agents have had a significant upgrade, building on Microsoft’s move to use OpenAI’s generative AI models. That upgrade is paired with the launch of a new AI authoring environment for Power Platform, Copilot Studio. Announced at Ignite 2023, Copilot Studio is available as both a web application and as a Teams app. Quickly expanding beyond the original US-only launch, Microsoft has already added support for several other geographies.

Complementary Copilot Studios

Each version of Copilot Studio targets a different audience. The web application supports more advanced development models, and is suitable for building externally facing chatbots, by development teams and IT admins. The Teams app allows users to build their own internally facing tools, and to rapidly prototype new ideas. It’s a pairing of tools with a purpose: Once a chatbot has been proven internally, it can be handed over to a development team to be refined and shared with external users, and incorporated into customer service workflows.

The resulting process builds on concepts explored in Power Platform’s Fusion Teams, where different users work together to build enterprise IT apps quickly. There are other differences between the web development platform and Teams, for example managing access controls, supporting multiple input channels, and working with existing Bot Framework capabilities.

Build your first generative AI chatbot

Getting started with Copilot Studio requires a Copilot Studio account. You can get a trial account for free if you have a Microsoft 365 account, so you can try out the service before committing to another subscription.

Much of what goes into building a Copilot Studio chatbot should be familiar. Your bot is trained on topics, using your documents to provide the underlying knowledge. The new toolkit builds on these capabilities, providing what Microsoft calls a GPT-powered “boost” to summarize additional data based on user intent, delivering the results in natural language. You don’t need to train your chatbot on this data; all you need to do is provide a link to the source. When answers can’t be found in the training data, this additional information can be used to generate new responses.

This hybrid approach to AI takes advantage of lower-cost, low-compute approaches when appropriate, and draws on more complex and more resource-intensive queries when the initial response tier fails. We’ve seen this model before, for example, when using cloud infrastructure to extend on-premises resources.

At the heart of Copilot Studio is the idea of “topics,” which are collections of related information that are labelled and used to train the underlying AI model. For example, if you’re building a chatbot to provide first-level triage for an IT support team, you might have a list of software versions linked to available updates, labelled “Supported applications.” When a user asks if a particular tool is supported, your chatbot will offer them links to the self-service update instructions.

You define the conversation flow the bot uses to get to that result, including the rules that control what questions are asked, drilling down to the specific software version, and if it’s covered by existing IT policies. Similar flows could guide users to other self-help tools, such as for resetting passwords or setting up email clients, allowing the support team to concentrate on more complex problems.

Boosting chat with GPT

Developing a chatbot using Copilot Studio remains relatively simple, building on the techniques developed for Power Virtual Agents. It also turns out to be the easiest way to get access to Azure OpenAI’s GPT models, albeit within a strict framework that avoids many of the associated risks.

To build a boosted chatbot, begin by creating a new bot from the Copilot Studio home page. A boosted chatbot requires a name and the URL that will be used for fallback queries. This is most likely to be your own property, but could include useful but unstructured information sources, like a user discussion forum or an internal support knowledgebase that wouldn’t ordinarily be exposed to external queries.

You’re not limited to a single fallback site; additional fallback sites and document sources can be added later. Because Copilot Studio is a Microsoft 365 service, you’re able to use SharePoint and derivative services like OneDrive as part of your boosting strategy. You can even use Power Automate to extract data from line-of-business applications and services, formatting it as JSON data and using it as a custom data source.

That’s all you need to do to add GPT support to a chatbot. You can then continue to build the underlying service, adding topics and rules as before. The GPT-based service only runs when your bot can’t find data in its pre-defined set of resources and needs access to external data. The underlying techniques are much like those used when building AI applications with Prompt Flow or Semantic Kernel.

The external data sources used as fallbacks can be thought of as the semantic memories used to ground the LLM, using its summarization and embedding features to limit the search space used by the model. This is done by providing a more structured prompt that can be used to wrap search answers from your data sources in natural language, with a much lower risk of hallucination. That output can then be delivered through your choice of chatbot channels, from inside Teams to using generative speech techniques for voice outputs.

There’s a copilot in my copilot!

There’s an interesting twist to Copilot Studio. Not only can you use it to build your own copilots, but it’s a copilot in its own right. A Create With Copilot option gives you an AI-guided approach to building a chatbot. Simply tell the system what you want, describing what you intend to deliver to your users, and it will generate an appropriate topic for you.

With this approach Microsoft is taking the low-code model even further away from any form of design surface. There’s no need for an editor, or an IDE, or even a graphical programming model like the one used to build Power Automate flows. Instead, the copilot uses user intent to drive development, perhaps finally delivering on the long-promised intentional development model that has been part of Microsoft’s research strategy for as long as I can remember.

It will be interesting to watch how this approach evolves. The Power Platform’s connections to line-of-business applications and to Microsoft’s own Dynamics 365 platform could well make Copilot Studio the simplest way to quickly add intelligence to your business, delivering more than Q&A-driven chatbots.