Slot Mappings in forms don't work when intent name is a substring of another intent

[owgardiner]

Rasa version: 2.3.4

Rasa SDK version (if used & relevant): 2.3.1

Rasa X version (if used & relevant): None

Python version: 3.7.4

Operating system (windows, osx, ...): macOS Catalina 10.15.7

Issue:
In a form, when using from_intent the wrong slot mapping is used if one intent name is a substring of another intent.

in the example in the below image - the user intent is correctly classified by the nlu model as happy but the slot user_is_happy is set to False

This appears to be because this slot mapping is pulled out

Got mapping '{'type': 'from_intent', 'value': False, 'intent': 'unhappy'}'

note if the order of slot mappings in the below domain.yml are swapped this error disappears (it appears to be checking these in order?)

Error (including full traceback):

Bot loaded. Type a message and press enter (use '/stop' to exit): 
Your input ->  hello                                                                                                                                                                             
2021-03-03 18:31:59 DEBUG    rasa.core.lock_store  - Issuing ticket for conversation '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:31:59 DEBUG    rasa.core.lock_store  - Acquiring lock for conversation '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:31:59 DEBUG    rasa.core.lock_store  - Acquired lock for conversation '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:31:59 DEBUG    rasa.core.tracker_store  - Could not find tracker for conversation ID '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Starting a new session for conversation ID '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.shared.core.events.SessionStarted object at 0x7fa733327c90>, ActionExecuted(action: action_listen, policy: None, confidence: None)]'.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Current slot values: 
	user_is_happy: None
	requested_slot: None
	session_started_metadata: None
I2021-03-03 18:31:59 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2021-03-03 18:31:59 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Received user message 'hello' with intent '{'id': -26760195889278156, 'name': 'greet', 'confidence': 1.0}' and entities '[]'
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2021-03-03 18:31:59 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
2021-03-03 18:31:59 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'example_form'
2021-03-03 18:31:59 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user text: hello | previous action name: action_listen
2021-03-03 18:31:59 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-03-03 18:31:59 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
2021-03-03 18:31:59 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-03-03 18:31:59 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2021-03-03 18:31:59 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2021-03-03 18:31:59 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_MemoizationPolicy.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Predicted next action 'example_form' with confidence 1.00.
2021-03-03 18:31:59 DEBUG    rasa.core.actions.forms  - Activated the form 'example_form'.
2021-03-03 18:31:59 DEBUG    rasa.core.actions.forms  - No pre-filled required slots to validate.
2021-03-03 18:31:59 DEBUG    rasa.core.actions.forms  - Validating user input 'UserUttered(text: hello, intent: greet, use_text_for_featurization: False)'.
2021-03-03 18:31:59 DEBUG    rasa.core.actions.forms  - Validating extracted slots: {}
2021-03-03 18:31:59 DEBUG    rasa.core.actions.forms  - Request next slot 'user_is_happy'
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x7fa72e0e7f10>]'.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Action 'example_form' ended with events '[<rasa.shared.core.events.ActiveLoop object at 0x7fa72f73e810>, <rasa.shared.core.events.SlotSet object at 0x7fa730bee150>, BotUttered('How are you feeling today', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {"template_name": "utter_ask_user_is_happy"}, 1614796319.89027)]'.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Current slot values: 
	user_is_happy: None
	requested_slot: user_is_happy
	session_started_metadata: None
2021-03-03 18:31:59 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
[state 2] user intent: greet | previous action name: example_form | active loop: {'name': 'example_form'}
2021-03-03 18:31:59 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'action_listen'
2021-03-03 18:31:59 DEBUG    rasa.core.policies.rule_policy  - Predicted 'action_listen' after loop 'example_form'.
2021-03-03 18:31:59 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_RulePolicy.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2021-03-03 18:31:59 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2021-03-03 18:31:59 DEBUG    rasa.core.lock_store  - Deleted lock for conversation '77fa29d00b534d60a7c71023d0f29317'.
How are you feeling today
Your input ->  I'm happy                                                                                                                                                                         
2021-03-03 18:32:01 DEBUG    rasa.core.lock_store  - Issuing ticket for conversation '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:32:01 DEBUG    rasa.core.lock_store  - Acquiring lock for conversation '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:32:01 DEBUG    rasa.core.lock_store  - Acquired lock for conversation '77fa29d00b534d60a7c71023d0f29317'.
2021-03-03 18:32:01 DEBUG    rasa.core.tracker_store  - Recreating tracker for id '77fa29d00b534d60a7c71023d0f29317'
2021-03-03 18:32:01 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2021-03-03 18:32:01 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Received user message 'I'm happy' with intent '{'id': -5196762545415143517, 'name': 'happy', 'confidence': 0.999836802482605}' and entities '[]'
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 11 events.
2021-03-03 18:32:01 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
[state 2] user intent: greet | previous action name: example_form | active loop: {'name': 'example_form'}
[state 3] user intent: happy | previous action name: action_listen | active loop: {'name': 'example_form'}
2021-03-03 18:32:01 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2021-03-03 18:32:01 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
[state 2] user intent: greet | previous action name: example_form | active loop: {'name': 'example_form'}
[state 3] user text: I'm happy | previous action name: action_listen | active loop: {'name': 'example_form'}
2021-03-03 18:32:01 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-03-03 18:32:01 DEBUG    rasa.core.policies.rule_policy  - Predicted loop 'example_form'.
2021-03-03 18:32:01 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2021-03-03 18:32:01 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2021-03-03 18:32:01 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_RulePolicy.
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Predicted next action 'example_form' with confidence 1.00.
2021-03-03 18:32:01 DEBUG    rasa.core.actions.forms  - Validating user input 'UserUttered(text: I'm happy, intent: happy, use_text_for_featurization: False)'.
2021-03-03 18:32:01 DEBUG    rasa.core.actions.forms  - Trying to extract requested slot 'user_is_happy' ...
2021-03-03 18:32:01 DEBUG    rasa.core.actions.forms  - Got mapping '{'type': 'from_intent', 'value': False, 'intent': 'unhappy'}'
2021-03-03 18:32:01 DEBUG    rasa.core.actions.forms  - Successfully extracted 'False' for requested slot 'user_is_happy'
2021-03-03 18:32:01 DEBUG    rasa.core.actions.forms  - Validating extracted slots: {'user_is_happy': False}
2021-03-03 18:32:01 DEBUG    rasa.core.actions.forms  - Deactivating the form 'example_form'
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x7fa730c9e7d0>]'.
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Action 'example_form' ended with events '[<rasa.shared.core.events.SlotSet object at 0x7fa731fac950>, <rasa.shared.core.events.SlotSet object at 0x7fa731f41790>, <rasa.shared.core.events.ActiveLoop object at 0x7fa72ead2490>]'.
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Current slot values: 
	user_is_happy: False
	requested_slot: None
	session_started_metadata: None
2021-03-03 18:32:01 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
[state 2] user intent: greet | previous action name: example_form
2021-03-03 18:32:01 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2021-03-03 18:32:01 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
[state 2] user intent: greet | previous action name: example_form
2021-03-03 18:32:01 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-03-03 18:32:01 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_TEDPolicy.
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2021-03-03 18:32:01 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2021-03-03 18:32:01 DEBUG    rasa.core.lock_store  - Deleted lock for conversation '77fa29d00b534d60a7c71023d0f29317'.
Your input ->   

Command or request that led to error:

rasa shell --debug

user input: I'm happy

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en

pipeline:
# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
# # If you'd like to customize it, uncomment and adjust the pipeline.
# # See https://rasa.com/docs/rasa/tuning-your-model for more information.
#   - name: WhitespaceTokenizer
#   - name: RegexFeaturizer
#   - name: LexicalSyntacticFeaturizer
#   - name: CountVectorsFeaturizer
#   - name: CountVectorsFeaturizer
#     analyzer: char_wb
#     min_ngram: 1
#     max_ngram: 4
#   - name: DIETClassifier
#     epochs: 100
#     constrain_similarities: true
#   - name: EntitySynonymMapper
#   - name: ResponseSelector
#     epochs: 100
#     constrain_similarities: true
#   - name: FallbackClassifier
#     threshold: 0.3
#     ambiguity_threshold: 0.1

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
# # No configuration for policies was provided. The following default policies were used to train your model.
# # If you'd like to customize them, uncomment and adjust the policies.
# # See https://rasa.com/docs/rasa/policies for more information.
#   - name: MemoizationPolicy
#   - name: TEDPolicy
#     max_history: 5
#     epochs: 100
#     constrain_similarities: true
#   - name: RulePolicy

Content of domain file (domain.yml) (if relevant):

version: "2.0"

intents:
  - greet
  - happy
  - unhappy

forms:
  example_form:
    user_is_happy:
      - type: from_intent
        value: False
        intent: unhappy
      - type: from_intent
        value: True
        intent: happy

responses:
  utter_ask_user_is_happy:
  - text: "How are you feeling today"

slots:
  user_is_happy:
    auto_fill: false
    type: bool
    influence_conversation: false

Content of stories file (stories.yml) (if relevant):

version: "2.0"

stories:

- story: happy path
  steps:
  - intent: greet
  - action: example_form
  - active_loop: example_form

no rules

[sara-tagger]

Thanks for raising this issue, @alopez will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[kaustuk]

Hi @owgardiner, thanks for providing such a good description and details about the issue. It really helped in debugging 👍.

I have debugged the about issue. Actually here the problem is we are not validating intent name at all. We simply extracting the first value we find and setting that as slot value. Here is the code link just see link number 358 and 359.

rasa/rasa/core/actions/forms.py

Line 358 in 4cfc48a

elif mapping_type == str(SlotMapping.FROM_INTENT):

Proposed solution:

elif mapping_type == str(SlotMapping.FROM_INTENT):
      latest_intent_name = tracker.latest_message.intent_name
      if latest_intent_name == requested_slot_mapping.get("intent"):
           value = requested_slot_mapping.get("value")

@wochinge @alopez what do you think?

[wochinge]

@kaustuk We actually validate the intent here. I think the issue is that intent is not defined as list (intent: [unhappy]) but as single string and we would should either raise an error for this case or call _to_list

rasa/rasa/core/actions/forms.py

Line 616 in 7b26ccb

def _list_intents(

as we do it here

[kaustuk]

@wochinge ohh yeh.. I completely missed that my bad 😅.

So any suggestions on which approach we should do here?

I think we should go with _to_list approach. Raising error doesn't make sense to me. Since it's kind of valid case. What do you think?

[wochinge]

I agree! Let's call _to_list. Would you mind creating a PR?

[kaustuk]

@wochinge Yeah sure I would raise PR with this fix.

[wochinge]

Awesome, thank you! Just tag me for review then 🚀

[owgardiner]

Thanks, @kaustuk @wochinge for resolving this so swiftly!

Apologies that this is a little off-topic but I also posted something recently on the rasa-forum - here - regarding multi-intents. If either of you had a quick chance to look at my question there and point me in the correct direction I would really appreciate it!

[kaustuk]

Hi @wochinge I have raised PR, please review it.