ICAEW’s digital and IT teams originally looked at areas where generative AI technology could be quickly applied with a meaningful and positive impact to our members and students. The team drew inspiration from OpenAI’s early case study on working with Khan Academy to develop an AI-powered learning assistant using GPT-4.   

The team wanted to focus on a use case that could be developed in isolation with limited dependency on system integration and internal resources to reduce interference with the ongoing projects. While cost-cutting was not an isolated motivation, it was another helpful driver.  

As a result, ICAEW’s digital and IT teams initially assessed four use cases for the application of generative AI:  

• Evolving the ICAEW chatbot Mia.
• Creating a conversational experience for technical advisory (TAS) helpsheets.
• A CPD regulations assistant.
• A learning assistant. 

During the period in which ICAEW’s teams were assessing where this technology could add value, the external generative AI environment was evolving exceptionally quickly, with new tools and products coming to market on a daily basis. This meant ICAEW needed to consider which development paths may become obsolete once resources would be available. For this reason, the concept of a learning assistant, most likely the longest to develop, was descoped and the other three were taken forward.  

Shortly thereafter it also became clear that generative AI would not easily and immediately add value to members attempting to understand the new CPD regulations. The generative AI technology would need to learn information from a decision-tree that had been developed to guide members on their CPD pathway. However, the IT team noticed that it was unable to interpret information from the decision tree in the right format. As such this use case was also descoped, leaving two use cases centred on the adaptation or creation of conversational bots. 

The goal then became to create a conversational, value-first, engagement bot that had a personality, boundaries, and tight connectivity in the underlying LLM model.  

ICAEW’s original conversational chatbot Mia was originally launched across the website in 2019 to aid students, as well as current and prospective members, with enquiries regarding their studies, membership and finding relevant content for their needs. Since its launch, Mia handled around 27% of all inbound enquiries, many of which have been outside of customer support hours. 

In its original form, Mia is a technical and mechanical bot fulfilling an important business capability. However, there are several pain points:  

• While it offered reliable help in the first instance, it could often be slow from a user experience point of view. 
• The chatbot was also only able to answer a small subset of frequently asked questions and therefore a significant proportion of queries still relied on human-led support from the Member Services team. 
• From an operational perspective, updating the content in the chatbot often required copy-paste work, and it could be time-consuming to correct misunderstandings. 
• In its original form, it is unable to customise the user experience without large investment and evolution.  

ICAEW considered the potential gains in productivity by evolving Mia to become a chatbot powered by generative AI. A generative AI powered chatbot would use natural language processing (NLP) to generate human-like text responses to input questions improving customer experience and providing users with faster access to a wider range of information across ICAEW’s website. Automation and links to the underlying large language model (LLM) would also allow continuous updates as documents are updated, providing real-time accurate information.  

The conversational MiaPlus bot uses content from icaew.com as its primary repository from which all answers are grounded. Grounding is the process of providing contextually relevant data to the large language model (LLM). Using the website as the source to derive responses largely reduces the risk of factually incorrect statements being generated by the bots, as it’s frequently updated and any genuine mistakes can be rectified immediately.  

Using the website to ground the models does however introduce some complexity. During the development process, ICAEW opted to use OpenAI’s GPT 3.5 turbo model for a number of reasons, significantly because it, and its data, resides on ICAEW’s existing Microsoft infrastructure/tenancy. However, this, and other models, come with limitations to the number of characters and words that can be inputted. As content on icaew.com has been designed for presentation on a graphical user interface the length and format often breached these limits. So, to allow the grounding process to occur, lengthy content had to be “chunked”. In doing so a question within the conversation may not be sufficiently answered by one chunk. This then required the team to overlap chunks by breaking them into smaller chunks, often determined by the existence of headings or subheadings in the HTML.  

ICAEW was at an advantage in having a legacy Mia bot which has been built, trained, and adapted over several years, allowing the identification of common intents that users have for a customer service-oriented bot. These intents were used to conduct initial training of the MiaPlus bot and training was then extended by colleagues asking a broader range of questions that could be answered from website content.  

However, if the pre-existing Mia chatbot didn’t exist, there are a number of services, including services from OpenAI utilising ChatGPT+, which enable the creation of generative AI powered chatbots that reference to a specific body of content. These can bypass a lot of the technical and human effort involved.  

In addition to the grounding risks identified above, several others have been, identified: 

• Data confidentiality – was identified as a risk early on in the development process. This risk has been mitigated in two ways:  
  • Firstly, the chatbot has not been integrated with the system that identifies a user (ie, the site login) and this information is not collected or processed by necessity to use the chatbot. This means that MiaPlus can be used anonymously.  
  • Secondly, the bot has been deployed on the ICAEW Microsoft tenancy (a dedicated instance of Microsoft 365 and our organisational data which is stored within a specified location). This means that if the user decides to input personal information into the chatbot (against the terms of use), the data remains within ICAEW and only authorised individuals monitoring the conversations will have access to it, protecting the data privacy of the individual. 
• Intellectual property and copyright infringement – the development team worked closely with the internal legal team and third-party legal support to draft a set of terms and conditions of use, a fair usage policy and an amended website privacy notice to incorporate the use of MiaPlus. The team also have the ability to remove website content from the database the bot learns from if necessary. 
• Perception risk – around the time the team were completing testing and communicating the risks of going live to the senior leadership team, DPD had an unfortunate incident that attracted bad publicity. The negative perception of the technology could have jeopardised the project at a critical time. However, there was no negative change in perception amongst the leadership at ICAEW after the chatbot and the risks were evaluated. This is an ongoing risk as views on social media and updates in cyber security will not only impact ICAEW’s approach and strategy, but also the way generative AI will be used in the long-term. 
• Running costs – borne out of the GPT model these costs are typically charged on a Pay-As-You-Go basis but can be projected and capped. As more models become available the ability to switch between models can reduce costs at a trade-off for performance. 
• Reproducibility – identified during training was the notion that each response is often worded differently, even if the question asked is identical. This supports a human-like conversational experience but makes testing and measuring progress challenging. There are parameters in the models that can control the degree of variation, however, the sensitivity of changes does trade off against performance and experience. While this mostly refers to variation in performance, there are a small number of instances where the response can vary with a small tweak to the original question making the response completely unhelpful. This is an area of focus in evaluating the performance of the new MiaPlus chatbot. 
• Hallucinations – the impression that generative AI bots can hallucinate by responding with factually incorrect information was known early on. The process of grounding goes a long way to mitigating this and is further quashed by prompt engineering in the backend by developers (a process that provides instruction to the model as to how a conversation should be structured. For example, the tone to use or the verbosity of response).

Despite the risks, shifting to a generative AI based conversation chatbot offers a wide range of experiential and operational benefits: 

• Website users can benefit from an “Ask me anything” customer interaction. There is also a deeper level of engagement through retained conversation history. Users can interact with the chatbot using intuitive text-based conversations, as opposed to the navigational hierarchy, providing a more personalized experience. There is also a smoother transition to human-led support wherever additional help is required. 
• Operationally, with zero-touch updates to the chatbot, internal teams can spend more time on refining the customer experience. The chatbot can be refined through prompt engineering and ICAEW’s digital and IT teams can spend more time on other transformation projects.
• For ICAEW, the biggest benefit would be an improved experience of the website for members and students. This impact will be measured through the conversational capabilities of the chatbot and the experiential outcomes. For example, answers provided by the chatbot are evaluated and scored based on relevancy, accuracy, clarity, and tone. In terms of experience, customer experience scores, the number of enquiries successfully handled, the chatbot’s ability to interrogate ICAEW content and channel availability will all be important to measure success and to continue to enhance and improve MiaPlus. 

ICAEW has been engaged with a technology partner on this project since the spring of 2023. At the time of writing, the MiaPlus chatbot was live in beta mode.