Understanding Pepper Focus and Android Lifecycle

A solution to properly handle rest and sleep modes, using MediaPlayer and a dance animation as an example
Understanding Pepper Focus and Android Lifecycle

The purpose of this article is to explain how to correctly interrupt and restart an activity according to the specific states of the robot: alive, rest and sleep modes. It is based on a simple activity that plays music through Android MediaPlayer and makes Pepper dance by executing a QiSDK animation.

Prerequisites

The content of this article is for advanced users.

Recommended readings:

Assets:

  • Pepper animation headbang_a001.qianim
  • Synthwave Vibe by Meydän provided with the demo

The app was tested on:

  • Pepper QiSDK 2.9.3 using Robot SDK API 5
  • Pepper QiSDK 2.9.5 using Robot SDK API 7

Introduction

Making your robot dance while playing music seems like a nice feature to implement on your Pepper, right? This is easy to achieve as you may have seen using the Getting Started tutorials. You would play music and run an animation action and stop them when the application loses the focus. Easy!

But it may not be as simple as that in all situations. As you may know, it is possible to change the state of the robot from alive mode to sleep mode or to rest mode (also referred to as "disabled robot state"). This interrupts the execution of your application in a peculiar way that needs to be dealt with based on complementary information presented here.

First, let's introduce the elements involved in the process described in this article:

Robot Lifecycle and Android Lifecycle

On an Android Pepper QiSDK robot, the robot focus and Android application focus are two separate but linked processes. You need to understand what is the Android lifecycle on one side and what is the Pepper QiSDK robot lifecycle on the other side.

In addition to Android lifecycle callbacks, your application needs to deal with a connection to NAOqi processes that are located in the head of the robot and that will give access to actions (listen, talk, motion, ...) or services (people detection, mapping, ...).

Android/Naoqi communication
Android/Naoqi communication

To achieve that on Pepper QiSDK your app has to implement 3 callbacks that handle the Robot Lifecycle (onRobotFocusGained, onRobotFocusLost, onRobotFocusRefused). These are called if the robot gets registered or unregistered or if it cannot register to the head.

Android and Robot Lifecycle schema
Android and Robot Lifecycle schema

This is documented in the Getting Started and Mastering focus & robot lifecycle articles.

Live Robot modes: alive, rest and sleep

When your robot is turned on, it should be in alive mode. It is the standing posture after the boot process has been performed. The robot is now aware of the environment, "breathing", reacting to stimuli (hearing noise and voice, touching sensor contacts, seeing people, detecting movements around him). The tablet is turned on and displays the Android desktop or the application that is running.

From this Standing Posture, you can reach two other modes called rest and sleep mode, also referred to as Safe Posture.

Standing and Safe Pepper postures
Standing and Safe Pepper postures

The rest mode or disabled robot state is achieved by pressing twice on the chest button when Pepper is turned on, as described here. Doing so, the robot will then go to the Safe Posture. The Android application is still running but the connection to the head is interrupted. Any QiSDK process is interrupted.

The sleep mode is triggered by covering the top head camera and the touch sensors with your hand as described here. Doing so, the robot will then go to the Safe Posture with the shoulder and the eye leds lit in purple. (Why purple? I can't say, but it's possibly because Pepper has fancy dreams!) The Android application is still running and the connection to the head is interrupted as in rest mode. A difference to previous mode is that a black screen will cover the tablet view when the robot reaches Safe posture and you then have no access to the drawer menus, etc. Any QiSDK process is interrupted, but the app is still registered.

MediaPlayer, an Android resource

MediaPlayer class can be used to control playback of audio/video files and streams. This resource is from Android proper, and therefore independent from the robot lifecycle.

Depending on the use case, you might want to keep playing the sound from MediaPlayer even when the app is in the background, like when you can listen to music while sending a text message on your phone. Or you might not. But besides that, it is also important to plan what to do with MediaPlayer. Should the robot focus be lost or refused?

In the example presented here, we want the robot to both dance and play music. Once the robot stops dancing, the music should also stop.

We have now covered all the elements needed to understand what this is all about, so let's start!

A robot with 3 modes, how to get to a unique solution

The 3 modes presented above make it complicated to handle the loss of robot focus. We present a solution that works in any of the rest and sleep modes including the default app focus loss setting it to the background.

The naive activity

First let's consider the naive way of dealing with music and dance starting and stopping. In this first example the dance starts and stops, and the MediaPlayer is launched but never interrupted.

Note that the methods startDancing(), startPlayingMusic(), cancelDancing(), pauseMusic(), stopMusic() are helper functions. They are described in the Appendix, at the end of this document as the goal of this article is to explain when these actions should be executed and how.

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        setSpeechBarDisplayStrategy(SpeechBarDisplayStrategy.OVERLAY)
        QiSDK.register(this, this)
    }

    override fun onResume() {
        super.onResume()
        startPlayingMusic()
    }

    override fun onRobotFocusGained(qiContext: QiContext) {
        this.shouldBeDancing = true
        startDancing(qiContext)
    }

    override fun onRobotFocusLost() {
        this.shouldBeDancing = false
        dancingFuture.requestCancellation()
    }

    override fun onRobotFocusRefused(reason: String?) {
        this.shouldBeDancing = false
    }

    override fun onDestroy() {
        QiSDK.unregister(this, this)
        stopMusic()
        super.onDestroy()
    }

It is obvious that appart when the app is killed, the music will keep playing regardless of the state the robot is in, as summed up below:

Getting to state... Music status is...
App in background Playing
Robot in sleep mode Playing
Robot in rest mode Playing

This is not what we expect. Let's start the music when we gain the focus and pause the music when the application loses the focus or goes to the background with the following modifications:

    override fun onPause() {
        super.onPause()
        pauseMusic()
    }

With this new code, the app will stop playing music when sent to background, but it will not be the case for rest mode, and when the robot wakes up from sleep mode, the music will start before the robot is fully alive.

Getting to state... Music status...
App in background Interrupted
Robot in sleep mode Interrupted
Robot in rest mode Keeps playing
Recovering from state... Music status...
App in background Starts
Robot in sleep mode Starts too soon when
robot is stretching
Robot in rest mode Still playing

We can fix the fact that the music starts to soon, launching it only when the app gets the robot focus as follows:

    override fun onRobotFocusGained(qiContext: QiContext) {
        this.shouldBeDancing = true
        startDancing(qiContext)
        startPlayingMusic()
    }

But it is not enough. We need to go deeper to understand what happens when the robot switches from one mode to another.

The 3 Pepper states

As you can see, the application behaves differently depending on the mode the robot goes to or resumes from. Differences appear in the process of registering/unregistering, the timing with which the robot callbacks are executed and what is displayed on the screen.

Moreover, some differences between QiSDK versions and API levels make it difficult to produce a solution that works in any situation.

The two following tables summarize the methods called depending on the app status or the 3 modes the robot/app goes to.

Callbacks that are called when getting to...

... backgrounded app ... sleep mode (purple - cover head) ... rest mode (double press chest button)
onPause() <delay>*
onPause()
onRobotFocusLost()
onRobotFocusLost()

Callbacks that are called when resuming from...

... backgrounded app ... sleep mode (purple - cover head) ... rest mode (double press chest button)
onResume()
onRobotFocusedGained()
onResume()
<delay>*
onRobotFocusedGained()
None

* the delay represents the time the robot takes to physically reach the rest or standing position.

When entering sleep mode:

  • Pepper moves to the rest position before onPause() is called.
  • onRobotFocusLost() is called as usual, after onPause() is called.

When leaving sleep mode:

  • onResume() is called before Pepper moves to the standing position.
  • onRobotFocusGained() is called as usual, after onResume()

When entering rest mode:

  • onRobotFocusLost() is called without onPause() being called first.
  • Pepper moves to the rest position after.

When leaving rest mode:

  • No callbacks are called

This highlights noticeable behaviours:

In sleep mode:

  • a built-in black screen view is displayed on top of the application

In rest mode:

  • onRobotFocusLost() is called when the robot goes to rest, but there are no callbacks executed for the back to alive mode phase
  • the focused application remains visible.

Therefore differences appear between the two modes when the robot leaves them to go back to alive mode.

The sleep mode black screen does not allow users to interact with the tablet, while in rest mode it is still possible to use it, which can be confusing and misleading for users.

We can also see that when the robot leaves rest mode, the robot focus is not gained because the registering does not happen again, so no actions are possible at that point.

For these reasons, it is necessary to create an intermediate activity that will be launched when rest mode is triggered.

Note that this activity is not necessary to handle sleep mode, as it already has a built-in black screen that is displayed automatically.

The RobotUnavailableActivity activity

So, we need a second activity to handle rest mode: it will be launched when the robot loses the focus or cannot obtain it. This activity, called RobotUnavailableActivity, is necessary for two reasons.

The focus detection reason

When the robot goes from rest mode to alive mode, we need a way to detect when it gains the robot focus back. The API does not have such a function which would return the robot focus status. We need to deduce it from the Robot lifecycle callbacks calls.

For this we add an activity to our application that is launched when the focus of the main activity is lost or refused.

You need to understand the following specificities:

  • the loss of robot focus is due to the change of mode
  • the refusal of the robot focus is due to requesting the focus in a situation where it cannot be obtained
  • the robot focus can only be gained when the robot is in alive mode and ready to run an action. Actions cannot be run in rest or sleep modes.

RobotUnavailableActivity is started using an intent when the robot gets to safe posture. This has the consequence of forbidding the app to get the robot focus because it is launched during rest mode. RobotUnavailableActivity gets the Android focus and onRobotFocusedRefused() is called. Then a timer is started for a short period of time (we set this timeout to 5 seconds). When the time is over, the activity finishes which will give the app focus back to the main activity. The main activity gets briefly the app focus but the robot focus is refused as the robot is still in rest mode. It restarts RobotUnavailableActivity. This process will go on until you get out of rest mode by double clicking the chest button.

The display reason

When the robot is in rest mode, the tablet still displays the focused application. This is the second reason why we need RobotUnavailableActivity, so that we can display a different screen (for example, a message saying that the robot is not available, or a set of instructions).

The timer

Here is the code for the timer:

   private var countdownTimer: Timer? = null
   private var isRunning: Boolean = false;
   private fun startTimer(timeInMilliseconds: Long) {

        countdownTimer?.cancel()
        isRunning = true

        countdownTimer = timer("HoldOn", initialDelay = timeInMilliseconds, period = 1000 ) {
            isRunning = false
            finish()
        }
    }

The timer should be launched and cancelled as follows:


override fun onRobotFocusGained(qiContext: QiContext) {
        countdownTimer?.cancel()
        // the application is closed
        finish()
}

override fun onRobotFocusLost() {
        countdownTimer?.cancel()
}

override fun onRobotFocusRefused(reason: String?) {
        startTimer(5000)
}

Unregistering the RobotUnavailableActivity

The QiSDK.register and QiSDK.unregister in the RobotUnavailableActivity must be done in the onResume() and in the onPause() respectively. As the stop and restart of RobotUnavailableActivity happen in a very close succession, it can cause conflicts between the timer running in the robot worker thread and the one of the previous instance of RobotUnavailableActivity, which is not unregistered yet.

Launch from the main activity

RobotUnavailableActivity is launched from the currently focused activity with startActivity using an Intent. As you know, changing the current activity to another one makes lose the focus.

Moreover, to avoid unexpected application states, it is necessary to call QiSDK.unregister() before calling startActivity. This unregistering allows RobotUnavailableActivity to get registered without trouble.

The following launchRobotUnavailableActivity method has to be added to your main activity and needs to be called from onRobotFocusLost and onRobotFocusRefused.

   private fun launchRobotUnavailableActivity() {
        Log.i(TAG, "Unregister before RobotUnavailableActivity")

        // The launchRobotUnavailableActivity() is called when robot focus is lost or refused
        // But losing the focus or getting it refused does not mean the app gets unregistered.
        // As the RobotUnavailableActivity tries to get the focus immediately through startActivity below
        // it is necessary to have the calling activity unregistered beforehand
        QiSDK.unregister(this, this)

        Log.i(TAG, "Launching RobotUnavailableActivity")
        val intent = Intent(this, RobotUnavailableActivity::class.java)

        startActivity(intent)
    }

Now let's consider the remaining code of MainActivity.

The complete application

Now a few adaptations need to be presented on the MainActivity.

First, the registering to QiSDK needs to be postponed to onResume(). As you have seen in The 3 Pepper states section, without using RobotUnavailableActivity, the robot loses the focus when rest mode is triggered, and nothing happens when he goes back to alive mode. With RobotUnavailableActivity, the process will go through Android lifecycle callbacks and onResume() will be called. We can now register again as follows:

    override fun onResume() {
        QiSDK.register(this, this)
        super.onResume()
    }

The robot lifecycle callbacks are as follows:

    override fun onRobotFocusGained(qiContext: QiContext) {
        this.shouldBeDancing = true
        startDancing(qiContext)
        startPlayingMusic()
    }

    override fun onRobotFocusLost() {
        this.shouldBeDancing = false
        cancelDancing()

        // RobotUnavailableActivity will finish after a timeout (5 seconds is great) and
        // return to the current activity calling FocusGained or FocusRefused
        launchRobotUnavailableActivity()
    }

    override fun onRobotFocusRefused(reason: String?) {
        this.shouldBeDancing = false
        cancelDancing()

        // RobotUnavailableActivity will finish after 5 seconds
        // return to the current activity calling FocusGained or FocusRefused
        launchRobotUnavailableActivity()

    }

Unregistering needs to be done in the onPause() as well. Remember, the unregistering is also done in onRobotFocusLost() and onRobotFocusRefused() of the launchRobotUnavailableActivity() method.

    override fun onPause() {
        this.shouldBeDancing = false
        pauseMusic()
        cancelDancing()
        QiSDK.unregister(this, this)
        super.onPause()
    }

You will find the full code on SoftBank Robotics Lab github account.

Conclusion

Congratulations! You have reached the end of the article.

You should now understand how to master Pepper Focus and Android Lifecycle. Based on this advanced knowledge, we wish you pleasant coding experience.

You can now rest or get some sleep ;-)

Appendix

Music credits

The full project demo contains the following music:

Title: Synthwave Vibe
Author: Meydän
Source: meydan.bandcamp.com
Licence: creativecommons.org/licenses/by/3.0/deed.fr
Download (5MB): auboutdufil.com

Helper functions

Here are the helper functions to handle music and dance execution:

    private fun startDancing(qiContext: QiContext) {
        val animation = AnimationBuilder.with(qiContext)
            .withResources(R.raw.headbang_a001)
            .build()

        val animate = AnimateBuilder.with(qiContext).withAnimation(animation)
            .build()

        dancingFuture = continuouslyRun(animate)
    }

    private fun continuouslyRun(animate: Animate): Future&lt;Void> {
        Log.i(TAG, "constinuouslyRun called")
        return if (this.shouldBeDancing) {
            Log.i(TAG, "pepper should be dancing")
            animate.async().run().thenCompose {
                if ( it.hasError() ) {
                    Log.e(TAG, "ERROR on animateFuture: ${it.errorMessage}")
                }
                continuouslyRun(animate)
            }
        } else {
            Log.i(TAG, "pepper should not be dancing")
            Future.cancelled()
        }
    }

    private fun cancelDancing() {
        if ( this::dancingFuture.isInitialized ) {
            if ( !dancingFuture.isCancelled) {
                Log.i(TAG, "dance was not yet cancelled, requesting cancellation")
                dancingFuture.requestCancellation()
            }
            else {
                Log.i(TAG, "dance was cancelled already")
            }
        }
        else {
            Log.i(TAG, "dance was not initialized")
        }
    }

    private fun startPlayingMusic() {
        mediaPlayer = MediaPlayer.create(
            applicationContext,
            R.raw.music
        ).apply {
            isLooping = true
            start()
        }
    }

    private fun pauseMusic() {
        Log.i(TAG, "pauseMusic() called...")
        if (mediaPlayer?.isPlaying == true) {
            Log.i(TAG, "mediaplayer was not yet paused, pausing mediaplayer")
            mediaPlayer?.pause()
        } else {
            Log.i(TAG, "mediaplayer was not playing or not initialized")
        }
    }

    private fun stopMusic() {
        Log.i(TAG, "mediaplayer was not yet released (or not created), releasing mediaplayer")
        mediaPlayer?.release()
    }