Merge branch 'master' into switch-to-mypy

.github/workflows/continous-integration.yml

@@ -101,10 +101,8 @@ jobs:
       # trying to run the `rasa` command once before the docs build.
       run: poetry run rasa --help
 
-    - name: Test Docs 📃
-      run: |
-        poetry run python -c "from scripts import release; release.generate_changelog('major.minor.patch')"
-        make docs
+    - name: Test Docs 🕸
+      run: make test-docs
 
   test:
     name: Run Tests

.gitignore

@@ -37,6 +37,7 @@ lastmile_ai/learn/plots/
 .coverage
 .coveralls.yml
 .idea/
+.venv/
 *.iml
 out/
 .vscode/

CHANGELOG.mdx

@@ -15,25 +15,48 @@ You can find more information on how to add a new change log entry at
 https://github.com/RasaHQ/rasa/tree/master/changelog/ . -->
 
 <!-- TOWNCRIER -->
+## [1.10.11] - 2020-08-21
+
+### Improvements
+
+- [#6044](https://github.com/rasahq/rasa/issues/6044>): Do not deepcopy slots when instantiating trackers. This leads to a significant
+  speedup when training on domains with a large number of slots.
+- [#6226](https://github.com/rasahq/rasa/issues/6226>): Added more debugging logs to the [Lock Stores](./lock-stores.mdx) to simplify debugging in case of
+  connection problems.
+
+  Added a new parameter `socket_timeout` to the `RedisLockStore`. If Redis doesn't
+  answer within `socket_timeout` seconds to requests from Rasa Open Source, an error
+  is raised. This avoids seemingly infinitely blocking connections and exposes connection
+  problems early.
+
+### Bugfixes
+
+- [#5182](https://github.com/rasahq/rasa/issues/5182): Fixed a bug where domain fields such as `store_entities_as_slots` were overridden
+  with defaults and therefore ignored.
+- [#6191](https://github.com/rasahq/rasa/issues/6191): If two entities are separated by a comma (or any other symbol), extract them as two separate entities.
+- [#6340](https://github.com/rasahq/rasa/issues/6340): If two entities are separated by a single space and uses BILOU tagging,
+  extract them as two separate entities based on their BILOU tags.
+
+
 ## [1.10.10] - 2020-08-04
 
-Bugfixes
---------
+### Bugfixes
+
 - [#6280](https://github.com/rasahq/rasa/issues/6280): Fixed `TypeError: expected string or bytes-like object`
   issue caused by integer, boolean, and null values in templates.
 
 
 ## [1.10.9] - 2020-07-29
 
-Improvements
-------------
+### Improvements
+
 - [#6255](https://github.com/rasahq/rasa/issues/6255): Rasa Open Source will no longer add `responses` to the `actions` section of the
   domain when persisting the domain as a file. This addresses related problems in Rasa X
   when Integrated Version Control introduced big diffs due to the added utterances
   in the `actions` section.
 
-Bugfixes
---------
+### Bugfixes
+
 - [#6160](https://github.com/rasahq/rasa/issues/6160): Consider entity roles/groups during interactive learning.
 
 
@@ -392,7 +415,7 @@ Bugfixes
   specified as a list in the `endpoints.yml` event broker config under a new key
   `queues`. Example config:
 
-  ```yaml
+  ```yaml-rasa
   event_broker:
     type: pika
     url: localhost
@@ -809,13 +832,14 @@ Bugfixes
 
   The following settings match the previous `NGramFeaturizer`:
 
-  ```yaml
+  ```yaml-rasa
+  pipeline:
   - name: 'CountVectorsFeaturizer'
-      analyzer: 'char_wb'
-      min_ngram: 3
-      max_ngram: 17
-      max_features: 10
-      min_df: 5
+    analyzer: 'char_wb'
+    min_ngram: 3
+    max_ngram: 17
+    max_features: 10
+    min_df: 5
   ```
 
 * [#4957](https://github.com/rasahq/rasa/issues/4957): To [use custom features in the `CRFEntityExtractor`](./components/entity-extractors.mdx#passing-custom-features-to-crfentityextractor)

Makefile

@@ -97,6 +97,20 @@ test: clean
 	# OMP_NUM_THREADS can improve overall performance using one thread by process (on tensorflow), avoiding overload
 	OMP_NUM_THREADS=1 poetry run pytest tests -n $(JOBS) --cov rasa
 
+generate-pending-changelog:
+	poetry run python -c "from scripts import release; release.generate_changelog('major.minor.patch')"
+
+cleanup-generated-changelog:
+	# this is a helper to cleanup your git status locally after running "make test-docs"
+	# it's not run on CI at the moment
+	git status --porcelain | sed -n '/^D */s///p' | xargs git reset HEAD
+	git reset HEAD CHANGELOG.mdx
+	git ls-files --deleted | xargs git checkout
+	git checkout CHANGELOG.mdx
+
+test-docs: generate-pending-changelog docs
+	poetry run pytest tests/docs/*
+
 docs:
 	cd docs/ && poetry run yarn pre-build && yarn build
 

changelog/6055.improvement.rst

@@ -0,0 +1 @@
+Support for rasa conversation id with special characters on the server side - necessary for some channels (e.g. Viber)

changelog/6453.removal.md

@@ -0,0 +1,34 @@
+Removed support for `queue` argument in `PikaEventBroker` (use `queues` instead).
+
+Domain file:
+- Removed support for `templates` key (use `responses` instead).
+- Removed support for string `responses` (use dictionaries instead).
+
+NLU `Component`:
+- Removed support for `provides` attribute, it's not needed anymore.
+- Removed support for `requires` attribute (use `required_components()` instead).
+
+Removed `_guess_format()` utils method from `rasa.nlu.training_data.loading` (use `guess_format` instead).
+
+Removed several config options for [TED Policy](./policies#ted-policy), [DIETClassifier](./components/intent-classifiers#dietclassifier) and [ResponseSelector](./components/selectors#responseselector):
+- `hidden_layers_sizes_pre_dial`
+- `hidden_layers_sizes_bot`
+- `droprate`
+- `droprate_a`
+- `droprate_b`
+- `hidden_layers_sizes_a`
+- `hidden_layers_sizes_b`
+- `num_transformer_layers`
+- `num_heads`
+- `dense_dim`
+- `embed_dim`
+- `num_neg`
+- `mu_pos`
+- `mu_neg`
+- `use_max_sim_neg`
+- `C2`
+- `C_emb`
+- `evaluate_every_num_epochs`
+- `evaluate_on_num_examples`
+
+Please check the documentation for more information.

data/test_config/config_empty_fr.yml

@@ -0,0 +1,3 @@
+language: fr
+pipeline:
+policies:

data/test_config/config_empty_fr_after_dumping.yml

@@ -0,0 +1,27 @@
+language: fr
+pipeline:
+# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
+# # If you'd like to customize it, uncomment and adjust the pipeline.
+# # See https://rasa.com/docs/rasa/nlu/choosing-a-pipeline/ for more information.
+#   - name: WhitespaceTokenizer
+#   - name: RegexFeaturizer
+#   - name: LexicalSyntacticFeaturizer
+#   - name: CountVectorsFeaturizer
+#   - name: CountVectorsFeaturizer
+#     analyzer: char_wb
+#     min_ngram: 1
+#     max_ngram: 4
+#   - name: DIETClassifier
+#     epochs: 100
+#   - name: EntitySynonymMapper
+#   - name: ResponseSelector
+#     epochs: 100
+policies:
+# # No configuration for policies was provided. The following default policies were used to train your model.
+# # If you'd like to customize them, uncomment and adjust the policies.
+# # See https://rasa.com/docs/rasa/core/policies/ for more information.
+#   - name: MemoizationPolicy
+#   - name: TEDPolicy
+#     max_history: 5
+#     epochs: 100
+#   - name: RulePolicy

data/test_domains/default_unfeaturized_entities.yml

@@ -14,8 +14,8 @@ entities:
 
 responses:
   utter_greet:
-    - hey there!
+    - text: hey there!
   utter_goodbye:
-    - goodbye :(
+    - text: goodbye :(
   utter_default:
-    - default message
+    - text: default message

data/test_domains/empty_response_format.yml

@@ -0,0 +1,25 @@
+intents:
+ - greet
+ - default
+ - goodbye
+
+slots:
+  cuisine:
+    type: text
+  location:
+    type: text
+
+entities:
+ - name
+
+responses:
+  utter_greet:
+  utter_goodbye:
+    - text: goodbye :(
+  utter_default:
+    - text: default message
+
+actions:
+  - utter_default
+  - utter_greet
+  - utter_goodbye

data/test_domains/wrong_custom_response_format.yml

@@ -0,0 +1,26 @@
+intents:
+ - greet
+ - default
+ - goodbye
+
+slots:
+  cuisine:
+    type: text
+  location:
+    type: text
+
+entities:
+ - name
+
+responses:
+  utter_greet:
+    - super: cool
+  utter_goodbye:
+    - text: goodbye :(
+  utter_default:
+    - text: default message
+
+actions:
+  - utter_default
+  - utter_greet
+  - utter_goodbye

data/test_domains/wrong_response_format.yml

@@ -0,0 +1,26 @@
+intents:
+ - greet
+ - default
+ - goodbye
+
+slots:
+  cuisine:
+    type: text
+  location:
+    type: text
+
+entities:
+ - name
+
+responses:
+  utter_greet:
+    - hey there!
+  utter_goodbye:
+    - goodbye :(
+  utter_default:
+    - stuff: default message
+
+actions:
+  - utter_default
+  - utter_greet
+  - utter_goodbye

docs/docs/chitchat-faqs.mdx

@@ -110,7 +110,7 @@ Finally, we'll copy over some user message training data from Sara to train our
 intents (more can be found [here](https://github.com/RasaHQ/rasa-demo/blob/master/data/nlu/nlu.md)):
 
 
-```yaml title="data/nlu.yml"
+```yaml-rasa title="data/nlu.yml"
 nlu:
 - intent: greet
   examples: |

docs/docs/components/custom-nlu-components.mdx

@@ -17,7 +17,7 @@ You can add a custom component to your pipeline by adding the module path.
 So if you have a module called `sentiment`
 containing a `SentimentAnalyzer` class:
 
-```yaml
+```yaml-rasa
 pipeline:
 - name: "sentiment.SentimentAnalyzer"
 ```

docs/docs/components/entity-extractors.mdx

@@ -61,7 +61,7 @@ Entity extractors extract entities, such as person names or locations, from the
 
 * **Configuration**
 
-  ```yaml
+  ```yaml-rasa
   pipeline:
   - name: "MitieEntityExtractor"
   ```
@@ -126,7 +126,7 @@ The `SpacyEntityExtractor` extractor does not provide a `confidence` level and w
   the [spaCy documentation](https://spacy.io/api/annotation#section-named-entities).
   Leaving the dimensions option unspecified will extract all available dimensions.
 
-  ```yaml
+  ```yaml-rasa
   pipeline:
   - name: "SpacyEntityExtractor"
     # dimensions to extract
@@ -229,7 +229,7 @@ The `SpacyEntityExtractor` extractor does not provide a `confidence` level and w
 
   * `BILOU_flag` determines whether to use BILOU tagging or not. Default `True`.
 
-  ```yaml
+  ```yaml-rasa
   pipeline:
   - name: "CRFEntityExtractor"
     # BILOU_flag determines whether to use BILOU tagging or not.
@@ -345,7 +345,7 @@ The `SpacyEntityExtractor` extractor does not provide a `confidence` level and w
   the [duckling documentation](https://duckling.wit.ai/).
   Leaving the dimensions option unspecified will extract all available dimensions.
 
-  ```yaml
+  ```yaml-rasa
   pipeline:
   - name: "DucklingHTTPExtractor"
     # url of the running duckling server
@@ -439,7 +439,7 @@ The `SpacyEntityExtractor` extractor does not provide a `confidence` level and w
 
 * **Configuration**
 
-  ```yaml
+  ```yaml-rasa
   pipeline:
   - name: "EntitySynonymMapper"
   ```