Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dense dimension sparse input #6563

Merged
merged 17 commits into from
Sep 8, 2020
Merged

Dense dimension sparse input #6563

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f15ea6c
remove dense dimension
tabergma Sep 3, 2020
4a98acd
add changelog
tabergma Sep 3, 2020
945e7cd
update response selector and docs
tabergma Sep 3, 2020
405b418
do not calculate min of dense dim
tabergma Sep 4, 2020
5091ba2
Merge branch 'master' into dense-dimension-sparse-input
tabergma Sep 4, 2020
8ca8344
set default dimension to 256
tabergma Sep 4, 2020
278490a
update default values
tabergma Sep 4, 2020
83c3ec8
update docs
tabergma Sep 4, 2020
f1c2560
Merge branch 'master' into dense-dimension-sparse-input
tabergma Sep 4, 2020
be52bf5
Merge branch 'master' into dense-dimension-sparse-input
tabergma Sep 4, 2020
6cc2154
update changelog
tabergma Sep 7, 2020
89078e1
Merge branch 'master' into dense-dimension-sparse-input
tabergma Sep 7, 2020
113ad1d
Merge branch 'master' into dense-dimension-sparse-input
tabergma Sep 7, 2020
093b427
Merge branch 'master' into dense-dimension-sparse-input
rasabot Sep 7, 2020
be9364a
Merge branch 'master' into dense-dimension-sparse-input
rasabot Sep 7, 2020
dd48d4c
Merge branch 'master' into dense-dimension-sparse-input
rasabot Sep 7, 2020
984703e
Merge branch 'master' into dense-dimension-sparse-input
rasabot Sep 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog/6555.improvement.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Do not set the output dimension of the `sparse-to-dense` layers to the same dimension as the dense features.

Update default value of `dense_dimension` and `concat_dimension` for `text` in `DIETClassifier` to 128.
4 changes: 2 additions & 2 deletions docs/docs/components/intent-classifiers.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,10 +364,10 @@ Intent classifiers assign one of the intents defined in the domain file to incom
+---------------------------------+------------------+--------------------------------------------------------------+
| embedding_dimension | 20 | Dimension size of embedding vectors. |
+---------------------------------+------------------+--------------------------------------------------------------+
| dense_dimension | text: 512 | Dense dimension for sparse features to use if no dense |
| dense_dimension | text: 128 | Dense dimension for sparse features to use if no dense |
| | label: 20 | features are present. |
+---------------------------------+------------------+--------------------------------------------------------------+
| concat_dimension | text: 512 | Concat dimension for sequence and sentence features. |
| concat_dimension | text: 128 | Concat dimension for sequence and sentence features. |
| | label: 20 | |
+---------------------------------+------------------+--------------------------------------------------------------+
| number_of_negative_examples | 20 | The number of incorrect labels. The algorithm will minimize |
Expand Down
10 changes: 4 additions & 6 deletions rasa/nlu/classifiers/diet_classifier.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@
EPOCHS,
RANDOM_SEED,
LEARNING_RATE,
DENSE_DIMENSION,
RANKING_LENGTH,
LOSS_TYPE,
SIMILARITY_TYPE,
Expand Down Expand Up @@ -87,6 +86,7 @@
TENSORBOARD_LOG_LEVEL,
CONCAT_DIMENSION,
FEATURIZERS,
DENSE_DIMENSION,
)


Expand Down Expand Up @@ -174,9 +174,9 @@ def required_components(cls) -> List[Type[Component]]:
# Dimension size of embedding vectors
EMBEDDING_DIMENSION: 20,
# Default dense dimension to use if no dense features are present.
DENSE_DIMENSION: {TEXT: 512, LABEL: 20},
DENSE_DIMENSION: {TEXT: 128, LABEL: 20},
# Default dimension to use for concatenating sequence and sentence features.
CONCAT_DIMENSION: {TEXT: 512, LABEL: 20},
CONCAT_DIMENSION: {TEXT: 128, LABEL: 20},
# The number of incorrect labels. The algorithm will minimize
# their similarity to the user input during training.
NUM_NEG: 20,
Expand Down Expand Up @@ -1340,14 +1340,12 @@ def _prepare_sparse_dense_layers(
) -> None:
sparse = False
dense = False

for is_sparse, feature_dimension in feature_signatures:
if is_sparse:
sparse = True
else:
dense = True
# if dense features are present
# use the feature dimension of the dense features
dense_dim = feature_dimension

if sparse:
self._tf_layers[f"sparse_to_dense.{name}"] = layers.DenseForSparse(
Expand Down
2 changes: 1 addition & 1 deletion rasa/nlu/selectors/response_selector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@
EPOCHS,
RANDOM_SEED,
LEARNING_RATE,
DENSE_DIMENSION,
RANKING_LENGTH,
LOSS_TYPE,
SIMILARITY_TYPE,
Expand Down Expand Up @@ -72,6 +71,7 @@
TENSORBOARD_LOG_LEVEL,
CONCAT_DIMENSION,
FEATURIZERS,
DENSE_DIMENSION,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in response selector, 128 is not enough?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we changed it to predict labels, it is basically another DIET

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I set it to something lower, the performance dropped (see the tables above).

)
from rasa.nlu.constants import (
RESPONSE,
Expand Down