Managing chatbot failure gracefully

Need of fallback

Building a chatbot that could possibly handle all of the user messages is something that isn’t possible at least not with the current advancements of NLP, but we can deal with these uncontrollable failures gracefully in RASA by using the concept of Fallback policy.

Fallback Policy is a concept whereby using a certain threshold value we can control the possible wrong responses of the bot.

For the sake of example let’s consider a certain user while interacting with a chatbot says “hiiiiiiiiii” with a lot of i’s in it and the bot misclassified it with some intent other than greet and based on that misclassified intent bot will take some action which might confuse the user. 

In actual production level projects, there can be many instances where the bot might get confused because of intents that have a certain similarity to each other. Because of these misclassified responses sent to the user the end-user experience is not so good which could lead to fewer users engaging with the chatbot.

General Idea about Fallback Policy

Let’s suppose we are building a farmbot for farmers, Given the fallback threshold as 75%. The user entered an utterance “Can you please tell me how to control weeds at the pre-emergence stage in wheat?”. The classifier predicts the intent for this utterance with 65% confidence, which is less than the defined threshold value (Predicted Confidence < Threshold value). In this scenario, the fallback classifier will predict an intent named nlu_fallback which will lead to a default rephrase response to the user.

 

Fallback policy not triggered

 

Let’s see how we can implement an NLU Fallback Policy:

To handle NLU Fallback rasa uses a FallbackClassifier.

Step to follow:

 

config.yml
pipeline:

# other components in the pipeline

– name: FallbackClassifier

  threshold: 0.8

 

Defining the response in the domain.yml file

domain.yml
responses:

  utter_please_rephrase:

  – text: I’m sorry, I didn’t quite understand that. Could you please rephrase? 

 

You can specify whatever response you want the bot to say above in the text

 

rules.yml
rules:

– rule: ask to rephrase

  steps:

  – intent: nlu_fallback

  – action: utter_please_rephrase

 

Below shown is an example of how the policy would work in action

Your input in RASA

Let’s now see how a Custom Fallback Policy can be implemented for response selectors:

 

 

Predicted retrieval response

To implement a custom fallback for response selectors we need to write a new action in the actions.py file

custom action by implementing a class that inherits from the Action class of rasa

 

There are two functions that we need to implement for the action to perform well

  1. 1. name -> This function is responsible for setting up the name of the action.
  2. 2. run -> In this function, we write the code logic of how the action should behave. There are multiple things that we can use in this function.

                       

                         a. dispatcher -> This object is responsible for sending messages to the user.

                         b. tracker -> It keeps track of the last user message and the response of the bot to that utterance(message) and other details                                                                     like slots, entities, etc.

 

                           c. Domain -> It contains the information about all the responses defined in the domain file.

 

To use this custom action we need to define it in the domain.yml file and in the rules.yml file

domain.yml
actions:

  • action_agri_faq
rules.yml
– rule: rule for category-1

  steps:

  – intent: faq-category-1

  – action: action_agri_faq

 

To implement our custom fallback we would be using the tracker object to put up some basic logic to handle the conversation.


1. Firstly we need to get the intent name from the tracker store.

_intent = tracker.latest_message[‘intent’].get(‘name’)

 

 2. Using this intent key we can find the confidence of response selectors’ prediction.

retrieval_intent_confidence =
tracker.latest_message[‘response_selector’][_intent][‘response’][‘confidence’] * 100

 

3. Now we can put up threshold confidence here

if retrieval_intent_confidence < 90:

    dispatcher.utter_message(text=”Custom Fallback can be implemented here”)

    return [ ]

This idea can be extended further for implementing different fallbacks for different response selectors too.

 

 

implementing different fallbacks for different response selectors too.

 

For example, let us say we have used two different retrieval Intents

 

We can dispatch two different fallbacks if we have some kind of requirement for it. All we need to do is put up a check-in step 3 defined above.

 

if retrieval_intent_confidence < 90:

  if “category-1” in _intent:

    dispatcher.utter_message(text=”Custom Fallback for category-1 can be implemented here”)

  elif “category-2” in _intent:

    dispatcher.utter_message(text=”Custom Fallback for category-2 can be implemented here”)

  return [ ]

 

 

Github: https://github.com/Bavalpreet/RASA-BLOG


References: https://rasa.com/docs/rasa/fallback-handoff

Contributors: Mr. Bavalpreet Singh and Lovepreet Singh