Case study: The Alcohol and Drug Foundation
In the face of overwhelming amounts of information being made readily available, accessing health information, and more importantly, correct health information, can be difficult. Add in the stigma of accessing information related to drugs and alcohol, and you quickly come up against barriers of entry. This is the issue that Dotahn Caspi, Digital Manager at the Alcohol and Drug Foundation (ADF) wanted to solve, and by doing so help more people access correct and relevant health-related information when they need it.
While working at the ADF and studying for a master’s degree during Melbourne lockdowns, Caspi noticed the rise of large language models and the effect they could play in the ability of people to access information online.
‘While I was doing that kind of research and thinking around my master’s, at the same time the whole digital world was being disrupted’ says Caspi. ‘So, I kind of had to pivot, following along with the rise of large language models and thinking around what does this mean for the way people access information?’
Interesting questions started to surface in relation to how people could access health information. In the case of the ADF, some of the information that people may be looking for to help them, may be considered illicit in nature, due to it being drug or alcohol related. If the information isn’t readily accessible, it could delay people from finding it when they need. The stigma around drug use also serves as a deterrent for people accessing relevant information.
“Most people delay seeking information or help for a very long time, usually up to a decade, and do not seek help unless they’re in crisis or there’s a significant life event which causes them to do so.”
These barriers in mind, Caspi developed a business case for the ADF to create an AI chatbot that would become dib. One of the main factors for Caspi was the idea of keeping things anonymous for users, ‘One of the big values that I saw with developing with AI is that idea of anonymity and creating a channel which is a safe space to ask questions without another human involved.’ dib provides an anonymous channel where users can ask questions without fear of judgement. The information provided by dib is sourced directly from the Alcohol and Drug Foundation website and is built in a way that only allows it to source information from the website, not the wider web, using retrieval-augmented generation (RAG). This allows dib to take the information from the website and translate it to users in a way that follows the flow of conversation.
‘We’ve done a lot of work to get it to the point where now we feel it is quite safe and it is providing accurate information.’
When asked about the decision to be upfront about dib being AI, Caspi says that the decision was unanimous. ‘There was a pretty wide agreement that we should be up front, very clear in articulating to people who are using the system.’ As a part of the original business case, Caspi employed an ethical framework for working with AI, looking at the human impacts to workforce, risks around accuracy of information, and providing a structure for handling risk. ‘It was in the process the whole way through. The ethical framework was great for exploring those risks and ethics of using AI.’
The project started by setting up an advisory group which included people working in the clinical space, academics and people with lived experience. The group explored the risks from each of their unique perspectives, which gave Caspi confidence in the ethical framework they developed. The framework also includes effective monitoring and assessment to catch and fix any inaccuracies.
Benefits
Since the public launch of the chatbot in March, the ADF has seen roughly 200 conversations a day, with users able to access the information they are looking for, or delving deeper into the information on the website. ‘Often with web content, (users) will jump into a page and jump out again, whereas we’re finding the conversations with the chatbot are encouraging a deeper exploration of the information, which I think is really valuable and hopefully an indicator that we are reducing some of those barriers to access.’
To the surprise of Caspi and the ADF, their chatbot has not only brought benefit to end users of the website, but also benefits to the ADF as well. ‘For quality improvement, it’s really, really been helpful,’ Caspi says. ‘It’s providing us with real-time insight into the gaps in our information, and insights into the information needs of people, the type of language people are using, the specific queries and what topics are of most interest.’ This type of quality insight has helped the ADF keep their website up to date, and helps to address the needs of the community more effectively.
Advice
Caspi credits the success of dib to the work that had been done early in development. When asked what advice he would give to not-for-profits looking to implement AI, he said to focus on guardrails. ‘I am thankful everyday for the amount of time we spent in building safeguards into the system. Make sure you get them right, because someone’s going to come in and break the system just with language.’ Guided by the ethical framework from development to implementation and beyond, ensures that the ADF is using AI in an ethical and safe way, while keeping abreast of the wider advances in AI technology. ‘We’ve implemented an internal AI policy and are continually having this conversation both with staff across the organisation, but with the board as well. You need to have those conversations early because it’s important to understand the risks.’