Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add git commit sha of the dataset branch in comment #8506

Conversation

HotThoughts
Copy link
Contributor

@HotThoughts HotThoughts commented Apr 20, 2021

Proposed changes:

Status (please check what you already did):

  • test cpu with the branch test-branch. The commit shows in the comment is 395f75e58e092eedd31cb09fc622baa5de4dbce4, which matches the latest commit of the test-branch in our dataset repo.
  • test gpu

@github-actions
Copy link
Contributor

Commit: 6b563ff, The full report is available as an artifact.

Dataset branch: test-branch, commit: 395f75e58e092eedd31cb09fc622baa5de4dbce4

Dataset: Carbon Bot, Dataset repository branch: test-branch

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
Sparse + DIET(bow) + ResponseSelector(bow)
test: 36s, train: 10m28s, total: 11m3s
0.7282 (-0.01) 0.7529 (0.00) 0.4702 (0.01)

@github-actions
Copy link
Contributor

Commit: a9f22a8, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: test-branch, commit: 395f75e58e092eedd31cb09fc622baa5de4dbce4

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
Sparse + DIET(bow) + ResponseSelector(bow)
test: 42s, train: 13m51s, total: 14m32s
0.7476 (0.01) 0.7529 (0.00) 0.4901 (0.03)

@HotThoughts HotThoughts marked this pull request as ready for review April 20, 2021 10:05
@HotThoughts HotThoughts requested a review from a team as a code owner April 20, 2021 10:05
@HotThoughts HotThoughts requested review from mprazz and tczekajlo and removed request for a team and mprazz April 20, 2021 10:05
@github-actions
Copy link
Contributor

github-actions bot commented Apr 20, 2021

@HotThoughts HotThoughts marked this pull request as draft April 20, 2021 11:47
@github-actions
Copy link
Contributor

Hey @tczekajlo! 👋 To run model regression tests, comment with the /modeltest command and a configuration.

Tips 💡: The model regression test will be run on push events. You can re-run the tests by re-add status:model-regression-tests label or use a Re-run jobs button in Github Actions workflow.

Tips 💡: Every time when you want to change a configuration you should edit the comment with the previous configuration.

You can copy this in your comment and customize:

/modeltest

```yml
##########
## Available datasets
##########
# - "Carbon Bot"
# - "Hermit"
# - "Private 1"
# - "Private 2"
# - "Private 3"
# - "Sara"

##########
## Available configurations
##########
# - "BERT + DIET(bow) + ResponseSelector(bow)"
# - "BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Spacy + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

## Example configuration
#################### syntax #################
## include:
##   - dataset: ["<dataset_name>"]
##     config: ["<configuration_name>"]
#
## Example:
## include:
##  - dataset: ["Carbon Bot"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Shortcut:
## You can use the "all" shortcut to include all available configurations or datasets
#
## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration
## for all available datasets
## include:
##  - dataset: ["all"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets
## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and
## "BERT + DIET + ResponseSelector(T2T)" configurations:
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##  - dataset: ["Hermit"]
##    config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"]
#
## Example: Define a branch name to check-out for a dataset repository. Default branch is 'main'
## dataset_branch: "test-branch"
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]


include:
 - dataset: ["Carbon Bot"]
   config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]

```

@github-actions
Copy link
Contributor

/modeltest

dataset_branch: "test-branch"
include:
 - dataset: ["Carbon Bot"]
   config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]

@github-actions
Copy link
Contributor

The model regression tests have started. It might take a while, please be patient.
As soon as results are ready you'll see a new comment with the results.

Used configuration can be found in the comment.

@github-actions
Copy link
Contributor

Commit: acfec15, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: test-branch, commit: 395f75e58e092eedd31cb09fc622baa5de4dbce4

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
Sparse + DIET(bow) + ResponseSelector(bow)
test: 40s, train: 2m45s, total: 3m24s
0.7340 (-0.01) 0.7529 (0.00) 0.5099 (0.05)

@HotThoughts HotThoughts marked this pull request as ready for review April 26, 2021 08:22
@HotThoughts
Copy link
Contributor Author

@tczekajlo This PR is ready for review. The result reports look great on both cpu and gpu runner.

@HotThoughts HotThoughts self-assigned this Apr 26, 2021
@HotThoughts HotThoughts enabled auto-merge (squash) May 3, 2021 11:51
@tczekajlo tczekajlo disabled auto-merge May 4, 2021 12:25
Copy link

@tczekajlo tczekajlo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎖️

@HotThoughts HotThoughts merged commit 51185ee into main May 4, 2021
@HotThoughts HotThoughts deleted the show-git-commit-sha-of-dataset-branch-in-model-regression-tests branch May 4, 2021 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Show git commit id of dataset branch in model regression tests
3 participants