Overview:
The ACROSS Virtual Assistant enables the end user to use the ACROSS platform through the Citizen Web App not only in the conventional mode of interaction (i.e. using keyboard and pointing device) but also through conversational interfaces, i.e. natural language.
Role and description:
In a dialogue that can take place via text chat or through speech, the VA “remote-controls” the CWA according to the user’s utterances, provides directions for the user on how to proceed (i.e. when a form needs to be filled step-by-step), or answers specific user questions, providing background information for the ongoing session. The VA supports multi-lingual operation, and the prototype provides partial support for the ACROSS use-case languages (German, Latvian and Greek) as well as for English.
The Virtual Assistant’s purpose is to improve ease of use and accessibility (in particular, for users with disabilities) for the ACROSS Citizen Web App. This goal is pursued by extending the CWA with dedicated conversational interfaces (i.e., multi-lingual textual chat and speech interfaces) for controlling user interface features of the ACROSS Web and Mobile App by natural language. In contrast to voice-centric solutions in the consumer sector such as Amazon Alexa, the existing web interface of the CWA always remains visible and usable, so the user can easily check whether e.g. form fields have been filled correctly by the VA. Once invoked, the VA dynamically extends the CWA with a chat&voice sidebar through which the conversation can be controlled. Switching forth and back between traditional and conversational interaction, or between voice- and chat-based conversation is possible at any time. Similarly, the conversation language can be changed in sync with the CWA display language.
The Virtual Assistant is implemented by building upon modern web standards and open-source software and integrating it both with the other ACROSS subsystems and with AI-based background technology from Fraunhofer FOKUS using an innovative, minimally invasive integration approach, utilizing a state-of the art service-based architecture. The specialised AI back-end components of the VA (e.g. Automatic Speech Recognition and Machine Translation) are deployed together with the other ACROSS subsystems on the ACROSS Kubernetes infrastructure but could also be used e.g. in private clouds or on-premises deployment. Fraunhofer FOKUS is currently working on integrating generative AI models into the VA in order to improve its flexibility, especially w.r.t. understanding the users.
Developed by:
Fraunhofer FOKUS