Summarizing user feedback with generative AI

Summary: This blog post explores a case study where content designers used generative artificial intelligence (Gen AI) to help analyze over 93,000 pieces of user feedback, leveraging insights to improve content on Canada.ca.

Contributors:

• Strategy and Engagement - DDPD (StratEngageDCDS-StratEngageSCCN@cra-arc.gc.ca)

On this page

• Discovering user needs and optimizing content
• Using page feedback to improve content
• Case study: using Gen AI to analyze MyAccount page feedback
  • The challenge: overwhelmed by volume
  • The solution: simplifying the process with GenAI
  • The outcome: improving the user experience with AI insights
• AI at the CRA

Discovering user needs and optimizing content

At the CRA, we follow the user-centred design (UCD) process to ensure that digital content meets user needs. One key way we do this is through content optimization projects (COPs), where content designers, web developers, program experts, data analysts, and UX researchers collaborate to improve some of the top pages on Canada.ca.

During a COP, teams follow the UCD process to ensure that user needs remain the focus when making content changes. This approach makes it easier for users to find information and complete tasks on our website without needing to contact a call centre for help.

A key phase of this process is the Discover phase. It involves research activities to identify which web content may need improvement.

Using page feedback to improve content

When updating web content, reviewing real user feedback is crucial—it helps identify user challenges and is a key activity in the Discover phase of the UCD process.

Each CRA web page includes a feedback form at the bottom, asking users, "Did you find what you were looking for?" If users select "no," they can provide details about their challenges. This feedback plays a crucial role in creating accessible and effective content.

Analyzing this data helps teams:

• Understand the specific issues users face
• Identify content gaps and usability issues
• Provide evidence-based justification for content changes
• Measure the impact of content updates after implementation

Case study: using Gen AI to analyze MyAccount page feedback

When the COP team began optimizing the My Account, My Business Account, and Represent a Client pages on Canada.ca, they turned to Generative AI during the Discover phase to help them uncover pain points that they could address.

The challenge: overwhelmed by volume

The page feedback tool collects a vast amount of data for the CRA—within just 6 months, we can receive up to 500,000 comments across our pages.

When investigating user issues on the identified pages, the COP team had to analyze over 93,000 comments. A strategist spent 50 hours manually reviewing the comments and identifying issues. While they were able to broadly understand the problems with the content, this manual review carried the risk of missed insights and was not an efficient use of resources during the project.

The solution: simplifying the process with GenAI

The team decided to leverage a GenAI tool to speed up this process and uncover more insights. They aimed to:

• Provide design and program teams with clear, summarized insights into the challenges users encounter on the pages
• Experiment with using GenAI in the Discover phase

Content strategists screened all comments to be 100% sure that there was no personal or sensitive information included. Next, they categorized comments with machine learning techniques. Then they prompted the GenAI tool to review, classify, and summarize large sets of feedback, focusing on:

• Identifying major pain points
• Clustering related themes for deeper analysis
• Summarizing key user challenges
• Proposing actionable recommendations

The outcome: improving the user experience with AI insights

The AI analysis identified 10 problem areas in the content, with 2 standing out as the most reported: users struggled with signing in and registering for their CRA account. For example, while the existing content provided detailed registration instructions, users struggled to find a link or button to start the process.

Using this insight—along with user testing, collaboration with call centre colleagues, and other data-gathering activities—content designers redesigned the registration page, adding a more prominent registration button. This optimization helped the CRA achieve:

• 51% decrease in negative feedback on registration
• 42% reduction in calls to the CRA per 100 page visits, freeing up valuable call centre resources

As a result of this effort, users can now register for CRA account more easily. Our COP team was also able to demonstrate the potential for AI to streamline parts of the UCD process.

AI at the CRA

The CRA’s guidance on AI use is evolving, but this case is an important first step in understanding how AI can assist with content optimization. By harnessing AI to analyze feedback, identify patterns, and generate actionable insights, we can make informed, user-driven decisions for content design on Canada.ca more efficiently.

If you’re interested in learning more about using AI for content and design work, stay tuned for updates from the Digital Design and Production Directorate.

Before using AI in any CRA work, be sure to consult the Guidelines for Employee Use of Online Generative Artificial Intelligence. Additional resources, guidance, and information about AI can be found on InfoZone.