CARESSES: smart and friendly robots for the elderly

An international partnership where SBRE puts forward Pepper to answer user specific needs
caresse header

How can technology support the worldwide issue of the ageing populations and the complexity of human connections? SBRE committed into the CARESSES program to integrate a system that plans the robot's actions according to a relevant cultural knowledge base.

When innovation embraces cultural background to ensure robots' performances and acceptability

It’s easy to picture that nursing staff who better understand their patients will dispense a greater quality of care. But the needs of patients can vary widely, according to a multitude of factors such as age, sex, and other background information.

Part of this background information is culture. The EU has understood that our ageing population is bringing with it a diversity of patients and a diversity of needs and backgrounds. And they anticipate that in an ever more connected environment, there is a real potential for robots to contribute. So it has launched a programme called CARESSES, which stands for Culture Aware Robots and Environmental Sensor Systems for Elderly Support. As its name implies, it addresses not only how robots can be culturally aware, but also how they can further answer user specific needs by managing sensor equipment such as lighting temperature etc, of a user’s environment.

This project thus combines smart with friendly.

Logo caresses
Logo caresses

Logo Caresses, a drawning where a robot offers multicolored flower petals

With their humanoid shape and their ever more complex communication with people, Pepper robots are particularly suited to this type of task. But challenges remain of adapting human-robot interaction (HRI) based on a complex operation of software, hardware to perceive and accurately react to the social and cultural background as well as personal needs, of people around the world.

SBRE is committed to a 3-year international project ending in January 2020, collaborating with European and Japanese Universities and Institutes that not only specialize in robotics, but AI and HRI too, and also in social psychology and health care technology.

In the CARESSES programme, these commercially available robots learnt to become culturally competent and were able to autonomously reconfigure their choice of actions and words to offer suitable services to match the culture, customs and etiquette of the person they attend to.

Partners studied how to represent the cultural model, how to adapt and use these models in sensing, planning and acting, and interacting with people and on IoT. Development and testing involved 3 robots, 3 different cultures, a number of elderly volunteers and their caregivers in 3 countries.

How did they do it? Let’s have a look.

CARESSES, the Flower that Taught Robots about Culture, Promotional video: online video; Youtube, 2:44

An International Collaboration of expert institutions

Partners around the world

The project is funded by the European Commission and the Ministry of Internal Affairs and Communications of Japan and involved European and Japanese Universities and Institutes specialized in robotics, AI, HRI, social psychology and health care technology.

EU Partners

  • Coordinator: University of Genova, Italy (Robotics, Artificial Intelligence)
  • Advinia Healthcare Limited, UK (Network of Residential and Nursing care homes)
  • Middlesex University, UK (Transcultural Nursing, Culturally Competent Healthcare)
  • Orebro University, Sweden (Robotics, Artificial Intelligence)
  • Softbank Robotics Europe, France (Robotics company)
  • The University of Bedfordshire, UK (Evaluation of Health- and Wellbeing-related technology)

The map of the EU CARESSES partners
The map of the EU CARESSES partners

CARESSES is an International collaboration of expert institutions including EU universities and a network of residential and nursing care homes.

Japanese Partners

  • Japan Coordinator: Japan Advanced Institute of Science and Technology, Japan (Human-Robot Interaction, Smart Home Automation)
  • Chubu University (Human-Robot Interaction, Human Behaviour Analysis)
  • Nagoya University (Social Psychology, Human-Robot Interaction)

The map of the japanese CARESSES partners
The map of the japanese CARESSES partners

Japanese universities are invloved in the CARESSES project

Common objective and approach

CARESSES aims to design care robots that are designed to be sensitive to the culture-specific needs and preferences of elderly clients while offering them a safe, reliable and intuitive system, specifically designed to support active and healthy ageing and reduce caregiver burden.

The objective of CARESSES is to build culturally competent care robots, able to autonomously reconfigure their way of acting and speaking, when offering a service, to match the culture, customs and etiquette of the person they are assisting.

From the user’s perspective, culturally appropriate behaviour is the key to improving acceptability; from the commercial perspective, it will open new avenues for marketing robots across different countries.

CARESSES teams adopted the following approach:

  1. Study how to represent cultural models, how to use these models in sensing, planning and acting, and how to acquire them.
  2. Consider three (physically identical) commercial robots on the market (i.e. our dear little Pepper with NAOqi 2.5) and integrate cultural models into them.
  3. Test the three robots, each one customized to a different culture, in the EU (two cultural groups) and Japan (one cultural group), on a number of elderly volunteers and their caregivers.

A schema of partners skills
A schema of partners skills

The mapping of CARESSES partners skills: Technical development, robot technology, health and user evaluation, testing facility

The Global Architecture: 3 modules and a web semantic communication protocol

The global architecture is split into 3 modules:

  1. The CKB(Cultural Knowledge Base)
  2. The CSPEM (Culturally Sensitive Planner and Execution Module)
  3. The CAHRIM (Culturally Aware Human-Robot Interaction Module)

Software architecture of the Caresses project
Software architecture of the Caresses project

Software architecture of the Caresses project

These three software components interact with each other using the universAAL framework.

The UniversAAL is an open standard platform for interoperable AAL solutions, with a communication protocol for semantic web technologies. It is the Enabler for Cross-domain Open Distributed Systems of Systems (e.g. IoT) made by the Fraunhofer IGD (Institut für Graphische Datenverarbeitung).

A rich source of information: Cultural Knowledge Base (CKB)

To make the Cultural Knowledge Base (CKB) which is the culture-generic knowledge layer containing general information about cultures, there are two stages.

The first is the setup stage, where the Culture-Generic Knowledge (in blue) is copied for a specific user to make the Culture-Specific Knowledge (see the yellow brick in the schema below). From then on, the Culture-Generic Knowledge is no longer needed.

Then through interactions, the robot gathers Person-Specific Knowledge (in orange) and which modifies the Culture-Specific Knowledge.

It is the principle of managing exceptions. In this way, leaning is going to be faster than building up knowledge from scratch.

The culture-specific knowledge enriched with a person-specific information

Let’s take a previous example to illustrate that process. Pepper interacts with an Englishman, let’s say Mr Hutchinson. The cultural base holds the following information: “An English person might enjoy football”. Pepper engages Mr Hutchinson and brings the topics of sports. The robot asks “Do you like football?” but Mr Hutchinson replies no. This interaction enriches the knowledge with that exception: Mr Hutchinson does not like football. Culture-Specific Knowledge is updated with this piece of information. As we can see, it becomes more and more specific.

Layers of the Cultural Knowledge Base
Layers of the Cultural Knowledge Base

Visual representation of the culture-generic layer, and the culture-specific/person-specific layer

This allows for more personalized interaction. The Culture-Specific Knowledge is directly connected to a dialogue tree, containing the possible subjects of conversation. This dialogue tree is updated with each Culture-Specific information, thus adapting weight to conversation topics. From the example previously given, when the robot updates the sports preferences of Mr Hutchinson, the “football” topic of conversation will have its weight reduced. A Bayesian network is associated with the dialogue tree so that when a topic is updated, that update is propagated to related topics of discussion.

Visual representation of the interactions between the culture-specific/person-specific layerand the dialog tree
Visual representation of the interactions between the culture-specific/person-specific layerand the dialog tree

Visual representation of the interactions between the culture-specific/person-specific layerand the dialog tree

Architecture of the Cultural Knowledge Base
Architecture of the Cultural Knowledge Base

Architecture of the Cultural Knowledge Base

A smart decision maker and planner

In robotics, a planner is a decision making state machine. For CARESSES, it is a specific module called Culturally Sensitive Planner and Execution Module (CSPEM).

This decides what and when to sequence, based on what is stored in the CKB and also on the feedback from the CSPEM.

For instance, having recognized that it is in front of a real person, i.e. processed information from the camera, the robot will engage in a particular conversation, i.e. using that person’s specific information.

In the previous example, the personal comfort zone when approaching people varies according to culture. Since Mr Hutchinson has a British culture the motion to approach that person will be adapted to give him a greater comfort zone.

This Planner uses SpiderPlan, a constraint-based planning framework. SpiderPlan includes different types of knowledge for automated planning. It also features execution via a Robot Operating System (ROS) interface.

Architecture of the Culturaly Sensitive Planner and Execution Module
Architecture of the Culturaly Sensitive Planner and Execution Module

Architecture of the Culturaly Sensitive Planner and Execution Module

The CARESSES’ planner is composed of multiple reasoning agents, allowing the system to handle many sub-problems at the same time. The agents interact through a constraint network, holding a shared representation of facts, goals, plans, and relations among them. This constraint network outputs actions to be executed and parameters associated with those actions.

The human-robot interaction module

Acting as robot interface, the Culturally Aware Human-Robot Interaction Module (CAHRIM) executes the actions to the person (or the connected devices) as requested by the Planner, sends back the state of the robot and the person’s requests.

It also exchanges between the robot and the other two modules, sending updates to the Cultural Knowledge Base. As in “This is what I was asked to do, and this is what happened”.

Sensory hubs constantly retrieve various sensory inputs from the states of the robot and person (e.g. position, emotion, gestures, activity, etc.).

CAHRIM ensures the accuracy of the CKB with the collected information and identifies the person’s habits and personality traits.

A conversation module retrieves the person’s vocal requests (those requests may be completed by information on the touch sensors).

Actuation hubs execute different actions (going close to the person, greeting them, etc.). These actions are adapted given the cultural and personal context of the situation. The actions are not only executed by the robot but could also be executed by other connected devices of IoT (lamps, connected switches, thermostats, etc.) or i-house (a smart house facility that belongs to the Japan Advanced Institute of Science and Technology - JAIST).

Architecture of the Culturaly Aware Human Robot Interaction Module
Architecture of the Culturaly Aware Human Robot Interaction Module

Architecture of the Culturaly Aware Human Robot Interaction Module

Culturally aware care robots are likely to have a greater acceptance from both the elderly and their caregivers

As populations are ageing across the world are placing health systems are coming under increasing pressure. Elderly care robots can be a means to relieve that pressure in hospitals and care homes, as well as a way to improve care delivery at home and promote independent living for the elderly. Care robots that are culturally aware and competent are likely to meet with greater acceptance from both the elderly and their caregivers.

Visual representation of the messages exchanged by the Caresses software modules
Visual representation of the messages exchanged by the Caresses software modules

Visual representation of the messages exchanged by the Caresses software modules

The project started on the 1st of January 2017 and is due to last 37 months. A successful demo was given in March 2019 at a residential care home in Luton (UK).

NAO
Edouard LAGRUE
Innovation Manager/Project Manager
NAO
Maxime BUSY
Innovation Engineer/TechLead

Related content