Smart Queries: ask Pepper about monsters and get accurate replies

Handling complex criteria

So, Pepper can now list monsters by color. That’s nice, but not very exciting. It’s worth stepping back and thinking about what kind of questions we want to ask; remember, we wanted to be able to ask about:

  • More complex criteria (“Show me the one-eyed flying purple monsters”)
  • Different kinds of questions (“How many red monsters have horns?”)

In this section, we’ll learn about the first: how to filter monsters by number of eyes, presence of horns, etc. and, more importantly, how to map that onto natural sentences people might say.

Step back and think of architecture

This section is mostly theoretical, we’ll discuss sample code, but no need to implement it in the app yet.

Let’s go back to our list of questions:

  • Show me red monsters
  • Show me blue monsters with horns
  • Show me monsters with one red eye

… and think of how we could make a pattern that recognizes all of these; for example, it could be:

concept:(color) [red blue yellow green orange purple pink brown black grey white]

u:(Show me _~color monsters)
   ^execute(clearScreen) Let me see ^execute(showMonsters, $1) here they are

u:(Show me _~color monsters with horns)
   ^execute(clearScreen) Let me see ^execute(showMonstersWithHorns, $1) here they are

u:(Show me monsters with _[1 2 3 4] _~color [eye eyes])
   ^execute(clearScreen) Let me see ^execute(showMonstersByNumberAndColor, $1, $2) here they are

We quickly notice a problem: the number of custom executors we need to create increases exponentially with the number of characteristics we want to ask about!

Another problem would arise if we ever wanted to make this app work in another language. Our executors are receiving English color names, but the same app in French would receive French names, and then the executor would have to be able to deal with that. Taken together, this can easily make the app very complex.

So, we’re going to use a very useful trick: assigning variables in concepts. Let’s say we redefined our ~color concept to be:

concept: (color) {
   "$qColor=red red"
   "$qColor=blue blue"
   "$qColor=yellow yellow"
   "$qColor=green green"
   "$qColor=orange orange"
   "$qColor=purple purple"
   "$qColor=pink pink"
   "$qColor=brown brown"
   "$qColor=black black"
   "$qColor=grey grey"
   "$qColor=white white"

What this means is that anytime one of those patterns is matched anywhere, this concept is used, and the corresponding variable is assigned to qColor.
We can then use this concept in a rule:

u:(Show me ~color monsters)
   ^execute(clearScreen) Let me see ^execute(showMonsters, $qColor) here they are

Notice that we’re not using $1 and the underscore any more. A nice advantage of this way of doing things is that we could redefine the concept in the French version of the topic file:

concept: (color) {
   "$qColor=red rouge"
   "$qColor=blue bleu"
   "$qColor=yellow jaune"
   "$qColor=green vert"
   "$qColor=orange orange"
   "$qColor=purple violet"
   "$qColor=pink rose"
   "$qColor=brown marron"
   "$qColor=brown brun"
   "$qColor=black noir"
   "$qColor=grey gris"
   "$qColor=white blanc"

… and the qColor would still be assigned the “standard” English one, so that the kotlin code would never have to worry about what language we’re in.
Another important advantage here is that this allows us to define some words as synonymous; for example, in French, both “marron” and “brun” are fairly common terms for shades of brown, but here we just mapped them both to “brown”. The same could be done in English, e.g. by mapping “scarlet” and “crimson” to “red” (but don’t go overboard with this, there a lot of color names and it’s quite unlikely that someone would actually ask Pepper for a “celadon monster”; it’s better to stick to words that are likely to appear in common usage; but if we were talking about hair color, including “blonde” as a synonym of yellow would be wise); we can therefore have both a standard and controlled vocabulary and understand very flexible sentences.
But let’s get back to our monsters. We could define equivalent concepts for eye color (defining a different concept that maps to a different variable!), number of eyes etc. and define a rule using concepts like these:

u:(Show me ~color monsters with ~eyeCount ~eyeColor [eye eyes])
   ^execute(clearScreen) Let me see ^execute(showMonsters, $qColor, $qEyeCount, $qEyeColor) here they are

…but this could quickly become unwieldy, and a bigger problem is that it doesn’t cover all combinations yet: even in this single case, the sentence will only be triggered if one specifies the color AND the eye count AND the eye color.
We could solve this by resetting variables at the beginning of user rules for example, but it would still remain unwieldy, especially as we add and remove variables.

There is a simpler solution: instead of passing the variables as parameters to the executor, we’ll just have the executor directly access the variables, so the QiChat can stay light...

Enough Theory, back to implementation

So, we’ll define some qiChat patterns that can recognize all of these patterns.

Let’s define the variables these are going to define; in all cases, if the value is not assigned, that means we accept any value:

  • qColor, a color name
  • qEyeColor, a color name (note that we can’t use the same concept as for color!)
  • qEyeCount, a number
  • qHorns, that can be “y” or “n” depending on whether we want horns or lack of horns
  • qHornCount, the specific number of horns we want
  • qTentacles, qWings and qFur, same principle as qHorns

First, define some common concepts:

concept: (bodyColor) {
   "$qColor=red red"
   "$qColor=blue blue"
   "$qColor=yellow yellow"
   "$qColor=green green"
   "$qColor=orange orange"
   "$qColor=purple purple"
   "$qColor=pink pink"
   "$qColor=brown brown"
   "$qColor=black black"
   "$qColor=grey grey"
   "$qColor=white white"

concept: (eyeColor) {
   "$qEyeColor=red red"
   "$qEyeColor=blue {sky} blue"
   "$qEyeColor=yellow orange" # Special
   "$qEyeColor=golden orange"
   "$qEyeColor=green green"
   "$qEyeColor=orange orange"
   "$qEyeColor=purple purple"
   "$qEyeColor=pink pink"
   "$qEyeColor=brown brown"
   "$qEyeColor=black black"
   "$qEyeColor=black dark"
   "$qEyeColor=grey grey"
   "$qEyeColor=white white"

   "$qEyeCount=2 [2 two]"
   "$qEyeCount=3 [3 three]"
   "$qEyeCount=4 [4 four]"

concept:(fur) [fur hair hairs]

Now, to use those concepts, we can distinguish two ways of specifying the monster’s traits:

  • Adjectives, that come before (“a two-eyed monster”)
  • Attributes, that come afterwards (“a monster with two eyes”)

Adjectives are fairly straightforward:

   "$qEyeCount=1 [one-eyed "one eyed"]"
   "$qEyeCount=2 [two-eyed "two eyed"]"
   "$qEyeCount=3 [three-eyed "three eyed"]"
   "$qEyeCount=4 [four-eyed "four eyed"]"
   "$eyeColor eyed"
   "$qHorns=y horned"
   "$qHornCount=1 one-horned"
   "$qHornCount=2 two-horned"
   "$qHornCount=3 three-horned"
   "$qTentacles=y tentacled"
   "$qWings=y [winged flying]"
   "$qHair=y [hairy furry]"

   "~adjective ~adjective"

Note that we allow between zero and two adjectives (the curly braces mean that this concept is optional), we could allow more but it is quite unlikely that someone could ask for a “one-eyed horned flying purple monster”.

Note also that we’re using “winged” and “flying” as synonymous.

Now to attributes; as for adjectives, we first define a singular attribute, and then ways of combining them:

       "~eyeColor eyes"
       "~eyeCount eyes"
       "~eyeCount ~eyeColor eyes"
       "$qEyeCount=1 [a an one 1] {~eyeColor} eye"
       "$qHorns=y horns"
       "$qHorns=n no horns"
       "$qHornCount=1 one horns"
       "$qHornCount=2 two horns"
       "$qHornCount=3 three horns"
       "$qHornCount=4 four horns"
       "$qTentacles=y tentacles"
       "$qTentacles=n no tentacles"
       "$qWings=y wings"
       "$qWings=n no wings"
       "$qFur=y ~fur"
       "$qFur=n no ~fur"

       "with ~attribute"
       "with ~attribute and ~attribute"
       "with ~attribute, ~attribute and ~attribute"
       "$qHorns=n without horns"
       "$qTentacles=n without tentacles"
       "$qFur=n without ~fur"
       "$qWings=n without wings"

Note how earlier, we didn’t cover the number “1” in the ~eyeCount concept, because it only appears in patterns where “eyes” is plural; in the ~attribute concept, we’ll add the pattern for having a single eye.

We can now define the ~monsters concept, and the corresponding query:

   "~adjectives monsters ~withAttributes"

u:(Show me ~monsters)
Do you like them?

So much for the QiChat part, now code

So, now we need to update our showMonster executor so that it uses this.

First, we need to understand how our QiChat variables translate to actual criteria. It’s important to know that QiChat variables are always strings; so first, let’s add helper extensions that make it easy to convert QiChat variable into kotlin nullable types; create a new kotlin file called QiChatExtensions.kt containing:

import com.aldebaran.qi.sdk.`object`.conversation.QiChatVariable

// returns the value, or null if it's an empty string
fun QiChatVariable.toStringOrNull() : String? {
   return when {
       value.isNotEmpty() -> value
       else -> null

// returns the value cast as int if possible, otherwise null
fun QiChatVariable.toIntOrNull() : Int? {
   return toStringOrNull()?.toIntOrNull()

// returns the value interpreted as boolean (y or n), otherwise null
fun QiChatVariable.toBooleanOrNull() : Boolean? {
   return when(value) {
       "y" -> true
       "n" -> false
       else -> null

... note that this matches the convention seen in QiChat above of using “y” and “n” for true and false for boolean values.

We can now use these to create a CriteriaManager class, which is able to filter the list of monsters to only keep those that match the criteria the user asked for. It will do this by creating an immutable “Criteria Snapshot” object, that contains the criteria as typed values instead of strings

Create a CriteriaManager.kt file, containing this class:

import com.aldebaran.qi.sdk.`object`.conversation.QiChatbot

class CriteriaManager(qiChatbot: QiChatbot) {
   private val colorVariable = qiChatbot.variable("qColor")
   private val eyeColorVariable = qiChatbot.variable("qEyeColor")
   private val eyeCountVariable = qiChatbot.variable("qEyeCount")
   private val hornsVariable = qiChatbot.variable("qHorns")
   private val hornCountVariable = qiChatbot.variable("qHornCount")
   private val tentaclesVariable = qiChatbot.variable("qTentacles")
   private val wingsVariable = qiChatbot.variable("qWings")
   private val furVariable = qiChatbot.variable("qFur")

   // "Frozen" criteria at a given point in time, can be used to check valid monsters
   inner class CriteriaSnapshot() {
       private val color = colorVariable?.toStringOrNull()
       private val eyeColor = eyeColorVariable?.toStringOrNull()
       private val eyeCount = eyeCountVariable?.toIntOrNull()
       private val horns = hornsVariable?.toBooleanOrNull()
       private val hornCount = hornCountVariable?.toIntOrNull()
       private val tentacles = tentaclesVariable?.toBooleanOrNull()
       private val wings = wingsVariable?.toBooleanOrNull()
       private val fur = furVariable?.toBooleanOrNull()

       fun isValid(monster : Monster) : Boolean{
           return when {
               color != null && monster.color != color -> false
               eyeColor != null && monster.eyeColor != eyeColor -> false
               eyeCount != null && monster.eyes != eyeCount -> false
               horns == false && monster.horns > 0 -> false
               horns == true && monster.horns == 0 -> false
               hornCount != null && monster.horns != hornCount -> false
               tentacles == false && monster.tentacles > 0 -> false
               tentacles == true && monster.tentacles == 0 -> false
               wings == false && monster.wings > 0 -> false
               wings == true && monster.wings == 0 -> false
               fur != null && monster.fur != fur -> false
               else -> true

   // Resets all QiChat variables, so that they are not "remembered" in next utterance
   fun clearVariables() {
       colorVariable?.value = ""
       eyeColorVariable?.value = ""
       eyeCountVariable?.value = ""
       hornsVariable?.value = ""
       hornCountVariable?.value = ""
       tentaclesVariable?.value = ""
       wingsVariable?.value = ""
       furVariable?.value = ""

   // Gets list of monsters fulfilling the actual criteria
   fun filterMonsters(monsters: List<Monster>) : List<Monster> {
       val snapshot = CriteriaSnapshot()
       return monsters.filter(snapshot::isValid)

Let’s see what this would mean if the user asks for “three-eyed red monsters with horns”

Matched concept

QiChat variable

Variable in CriteriaSnapshot

$qColor=red red


color = ”red”



eyeColor = null

"$qEyeCount=3 three-eyed"


eyeCount = 3

“$qHorns=y horns”


horns = true



hornCount = null



tentacles = null



wings = null



fur = null

If you look at the lines in isValid() in such a CriteriaSnaphot() object, this:

               eyeColor != null && monster.eyeColor != eyeColor -> false

… will never trigger because eyeColor is null, whereas this

               horns == false && monster.horns > 0 -> false

...becomes equivalent to

monster.horns > 0

Notice also that the condition on hornCount doesn’t trigger (because hornCount is null), just like eyeColor; there are actually two QiChat variable (“qHorns” and “qHornCount”) that map to the same monster attribute, the first is for one we only care about whether a monster has horns, the second for when we care about the exact number.

Now that we are able to filter monsters, update our MonsterDialogue class to include this criteriaManager, and :

class MonsterDialogue(qiContext: QiContext,
                     val allMonsters: List<Monster>,
                     val showMonsterList: (List<Monster>) -> Unit) {
   val TAG = "MonsterDialogue"

   private val topic = TopicBuilder.with(qiContext).withResource(R.raw.monsters).build()
   private val qiChatbot = QiChatbotBuilder.with(qiContext).withTopic(topic).build().apply {
       executors = hashMapOf<String, QiChatExecutor>(
               "showMonsters" to ShowMonstersExecutor(qiContext),
               "clearScreen" to ClearScreenExecutor(qiContext),
   private val criteriaManager = CriteriaManager(qiChatbot)

   private val chat = ChatBuilder.with(qiContext).withChatbot(qiChatbot).build() 

   fun run(): Future<Void> {
       return chat.async().run()

   // Executors

   inner class ShowMonstersExecutor(context: QiContext?) : BaseQiChatExecutor(context) {
       override fun runWith(params: MutableList<String>) {
           val queryMonsters = criteriaManager.filterMonsters(allMonsters)

       override fun stop() {}

   inner class ClearScreenExecutor(context: QiContext?) : BaseQiChatExecutor(context) {
       override fun runWith(params: MutableList<String>) {
       override fun stop() {}

Note that in each of the executors, we make sure to call criteriaManager.clearVariables(), otherwise the previous value will be remembered and will be added to the next utterance.

You can now test the app, and see if Pepper can answer correctly!