Fallback Policy in RASA

April 27, 2022 

Managing chatbot failure gracefully

Need of fallback

Building a chatbot that could possibly handle all of the user messages is something that isn’t possible at least not with the current advancements of NLP, but we can deal with these uncontrollable failures gracefully in RASA by using the concept of Fallback policy.

Fallback Policy is a concept whereby using a certain threshold value we can control the possible wrong responses of the bot.

For the sake of example let’s consider a certain user while interacting with a chatbot says “hiiiiiiiiii” with a lot of i’s in it and the bot misclassified it with some intent other than greet and based on that misclassified intent bot will take some action which might confuse the user. 

In actual production level projects, there can be many instances where the bot might get confused because of intents that have a certain similarity to each other. Because of these misclassified responses sent to the user the end-user experience is not so good which could lead to fewer users engaging with the chatbot.

General Idea about Fallback Policy

Let's suppose we are building a farmbot for farmers, Given the fallback threshold as 75%. The user entered an utterance “Can you please tell me how to control weeds at the pre-emergence stage in wheat?”. The classifier predicts the intent for this utterance with 65% confidence, which is less than the defined threshold value (Predicted Confidence < Threshold value). In this scenario, the fallback classifier will predict an intent named nlu_fallback which will lead to a default rephrase response to the user.

Let’s see how we can implement an NLU Fallback Policy:

To handle NLU Fallback rasa uses a FallbackClassifier.

• By using FallbackClassifier in the configuration pipeline, rasa ensures that an intent named nlu_fallback is predicted when all other intents predictions fall below the configured threshold value, further a rule can be defined for this nlu_fallback intent and the action required by the bot.

Step to follow:

• Updating the configuration file
  • We need to add the following in the config.yml file to enable the FallbackClassifier

config.yml

pipeline: # other components in the pipeline - name: FallbackClassifier   threshold: 0.8

Defining the response in the domain.yml file

domain.yml

responses:   utter_please_rephrase:   - text: I’m sorry, I didn’t quite understand that. Could you please rephrase?

You can specify whatever response you want the bot to say above in the text

• Creating a rule for the fallback

rules.yml

rules: - rule: ask to rephrase   steps:   - intent: nlu_fallback   - action: utter_please_rephrase

Below shown is an example of how the policy would work in action

Let’s now see how a Custom Fallback Policy can be implemented for response selectors:

To implement a custom fallback for response selectors we need to write a new action in the actions.py file

There are two functions that we need to implement for the action to perform well

1. 1. name -> This function is responsible for setting up the name of the action.
2. 2. run -> In this function, we write the code logic of how the action should behave. There are multiple things that we can use in this function.

                                                 a. dispatcher -> This object is responsible for sending messages to the user.

                         b. tracker -> It keeps track of the last user message and the response of the bot to that utterance(message) and other details                                                                     like slots, entities, etc.

                           c. Domain -> It contains the information about all the responses defined in the domain file.

To use this custom action we need to define it in the domain.yml file and in the rules.yml file

domain.yml

actions: action_agri_faq

rules.yml

- rule: rule for category-1   steps:   - intent: faq-category-1   - action: action_agri_faq

To implement our custom fallback we would be using the tracker object to put up some basic logic to handle the conversation.

1. Firstly we need to get the intent name from the tracker store.

_intent = tracker.latest_message['intent'].get('name')

 2. Using this intent key we can find the confidence of response selectors' prediction.

retrieval_intent_confidence = tracker.latest_message['response_selector'][_intent]['response']['confidence'] * 100

3. Now we can put up threshold confidence here

if retrieval_intent_confidence < 90:     dispatcher.utter_message(text="Custom Fallback can be implemented here")     return [ ]

This idea can be extended further for implementing different fallbacks for different response selectors too.

For example, let us say we have used two different retrieval Intents

• faq-category-1
• faq-category-2

We can dispatch two different fallbacks if we have some kind of requirement for it. All we need to do is put up a check-in step 3 defined above.

if retrieval_intent_confidence < 90:   if "category-1" in _intent:     dispatcher.utter_message(text="Custom Fallback for category-1 can be implemented here")   elif "category-2" in _intent:     dispatcher.utter_message(text="Custom Fallback for category-2 can be implemented here")   return [ ]

Github: https://github.com/Bavalpreet/RASA-BLOG

References: https://rasa.com/docs/rasa/fallback-handoff

Contributors: Mr. Bavalpreet Singh and Lovepreet Singh