`/restart` in `rasa shell` results in "zombie" slots

[samsucik]

Normally, you expect /restart to reset all slots (dump their values). However, I found a way to bring such "dead" slot values back to life 👻 which occurs in 3.0rc2 but not in 2.8.

Example set-up is similar to this: To a rasa init project add these domain parts:

intents:
  - inform

responses:
  utter_ask_slot_0:
  - text: "Let's start this form. Say something!"

  utter_ask_slot_A:
  - text: "Could you, possibly, please, give me a city."

  utter_ask_slot_B:
  - text: "Yo pal gimme a city!"

entities:
- city

slots:
  slot_0:
    type: any
    mappings:
    - type: from_text
      not_intent: greet
  slot_A:
    type: any
    mappings:
    - type: from_entity
      entity: city
      conditions:
      - active_loop: my_form
  slot_B:
    type: any
    mappings:
    - type: from_entity
      entity: city
      conditions:
      - active_loop: my_form
forms:
  my_form:
    required_slots:
        - slot_0
        - slot_A
        - slot_B

some NLU data for the new intent & entity:

- intent: inform
  examples: |
    - [Bratislava](city)
    - [Berlin](city)
    - [Prague](city)
    - [Amsterdam](city)
    - [Edinburgh](city)

the complete rules will look like this:

version: "2.0"

rules:

- rule: Say goodbye anytime the user says goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye

- rule: Say 'I am a bot' anytime the user challenges
  steps:
  - intent: bot_challenge
  - action: utter_iamabot

- rule: Activate form
  steps:
  - intent: greet
  - action: my_form
  - active_loop: my_form

- rule: Submit form
  condition:
  - active_loop: my_form
  steps:
  - action: my_form
  - active_loop: null
  - slot_was_set:
    - requested_slot: null
  - action: utter_happy

and you'll have to comment out all stories.

Then, rasa train the bot and rasa shell --debug talk to it. Below is the sequence of user messages and the interesting slots after the message has been processed, as reported by rasa shell:

message     slot_0      slot_A      slot_B
hi          None        None        None
Berlin      Berlin      None        None
Bratislava  Bratislava  Bratislava  None
Prague      Prague      Bratislava  Prague
/restart    None        None        None
hi          None        None        None
Bratislava  Bratislava  Bratislava  Prague

Notice how all slots are reported to be reset after /restart but old values come back to life after the last message.

[tayfun]

I think I found the core of the problem, but before that, I want to note that I see some duplicate events. For example, for the above playback, I see two events in tracker store which have the same exact slot set events for the single user utterance. Eg.

    {
      "event": "user",
      "timestamp": 1637330172.321048,
      "metadata": {
        "model_id": "37e752418f6f45a28cb94110449d5876"
      },
      "text": "Prague",
      "parse_data": {
        "intent": {
          "name": "inform",
          "confidence": 0.9999785423278809
        },
        "entities": [
          {
            "entity": "city",
            "start": 0,
            "end": 6,
            "confidence_entity": 0.9990973472595215,
            "value": "Prague",
            "extractor": "DIETClassifier"
          }
        ],
        "text": "Prague",
        "message_id": "8b98c13dbb9b4745b637e8f8fd67d393",
        "metadata": {},
        "text_tokens": [
          [
            0,
            6
          ]
        ],
        "intent_ranking": [
          {
            "name": "inform",
            "confidence": 0.9999785423278809
          },],
        "response_selector": {
          "all_retrieval_intents": [],
          "default": {
            "response": {
              "responses": null,
              "confidence": 0,
              "intent_response_key": null,
              "utter_action": "utter_None"
            },
            "ranking": []
          }
        }
      },
      "input_channel": "cmdline",
      "message_id": "8b98c13dbb9b4745b637e8f8fd67d393"
    },
    {
      "event": "slot",
      "timestamp": 1637330172.3216739,
      "metadata": {
        "model_id": "37e752418f6f45a28cb94110449d5876"
      },
      "name": "slot_0",
      "value": "Prague"
    },
    {
      "event": "user_featurization",
      "timestamp": 1637330172.3262131,
      "metadata": {
        "model_id": "37e752418f6f45a28cb94110449d5876"
      },
      "use_text_for_featurization": false
    },
    {
      "event": "action",
      "timestamp": 1637330172.3262231,
      "metadata": {
        "model_id": "37e752418f6f45a28cb94110449d5876"
      },
      "name": "my_form",
      "policy": "RulePolicy",
      "confidence": 1,
      "action_text": null,
      "hide_rule_turn": true
    },
    {
      "event": "slot",
      "timestamp": 1637330172.3263,
      "metadata": {
        "model_id": "37e752418f6f45a28cb94110449d5876"
      },
      "name": "slot_0",
      "value": "Prague"
    },
}

Now the core problem might be related to above so once a test and a fix is there, we should check above again. The problem right now is that we assume two UserUttered events are same if they have text,intent,entities and then in FormAction._get_events_since_last_user_uttered we check if it UserUttered is already in the dialogue tracker and its index using that equality. That is, if we have two hi messages like above, we'll start from the first one and use all the SlotSet events since then. See my comments in https://github.com/RasaHQ/rasa/pull/10295/files

@ancalita I would probably need your help in fixing this :)

[ancalita]

I see two events in tracker store which have the same exact slot set events for the single user utterance.

@tayfun Yes, there's a wrap-up issue to fix this in FormAction: #9945
Let me know if the issue description is unclear and if applying the fix described there fixes this issue.
We can also have a quick call this morning.