[Examples] refactor examples and add github action to automatically validate the scripts. #138
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TheaperDeng
changed the title
TheaperDeng
force-pushed
the
example-refactor
branch
from
e517a1c to
586c344
Compare
TheaperDeng
changed the title
jiaqima
approved these changes
Sep 14, 2024
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("--method", type=str, default="explicit") | ||
| parser.add_argument("--device", type=str, default="cuda") | ||
| args = parser.parse_args() | ||
|
|
||
| # load the dataset, we only need the train dataset |
There was a problem hiding this comment.
nit: this line is not grammatically correct
| parser.add_argument("--device", type=str, default="cuda") | ||
| args = parser.parse_args() | ||
|
|
||
| # load the dataset, we only need the train dataset |
| @@ -42,17 +49,18 @@ def get_mnist_indices_and_adjust_labels(dataset): | |||
| sampler=SubsetSampler(range(1000)), | |||
| ) | |||
|
|
|||
| model = train_mnist_lr(train_loader_full) | |||
| model.cuda() | |||
| model = train_cifar2_resnet9(train_loader, num_epochs=3, num_classes=10) | |||
There was a problem hiding this comment.
maybe rename this function as train_cifar_resnet9?
There was a problem hiding this comment.
good catch. I have revised the module to be named as dattri.benchmark.datasets.cifar that contains both cifar2 and cifar10 functions.
examples/readme.md
Outdated
| @@ -0,0 +1,25 @@ | |||
| # `dattri` examples | |||
| This folder contains bite-sized examples which can help users to build their own application by `dattri`. | |||
There was a problem hiding this comment.
This folder contains bite-sized examples that can help users build their own applications with dattri.
examples/readme.md
Outdated
| ## Noisy label detection | ||
| This section contains using different attributors to detect the noisy label in various datasets. | ||
|
|
||
| [Use influence function to detect noisy label in Mnist10 + Logistic regression.](./noisy_label_detection/influence_function_noisy_label.py) |
examples/readme.md
Outdated
|
|
||
| [Use influence function to detect noisy label in Mnist10 + Logistic regression.](./noisy_label_detection/influence_function_noisy_label.py) | ||
|
|
||
| [Use TracIN to detect noisy label in Mnist10 + MLP.](./noisy_label_detection/tracin_noisy_label.py) |
examples/readme.md
Outdated
|
|
||
| [Use TracIN to detect noisy label in Mnist10 + MLP.](./noisy_label_detection/tracin_noisy_label.py) | ||
|
|
||
| [Use TRAK to detect noisy label in CIFAR10 + ResNet-9.](./noisy_label_detection/trak_noisy_label.py) |
examples/readme.md
Outdated
|
|
||
| ## Use pretrained checkpoints and pre-calculated ground truth | ||
|
|
||
| This section contains examples to use the pretrained checkpoints and pre-calculated ground truth provided by `dattri` to evaluate the data attribution methods. |
examples/readme.md
Outdated
|
|
||
| ## Estimate the brittleness | ||
|
|
||
| This section contains examples to use attribution score to estimate the brittleness of a model. |
There was a problem hiding this comment.
to use -> using
attribution score -> attribution scores
TheaperDeng
force-pushed
the
example-refactor
branch
from
586c344 to
1561cbb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
1. Motivation and Context
Our example collection is kind of messy and hard to navigate. This PR aims to make the organized and add github action to test them to enable long-term usability.
This PR does not add any examples. New example scripts will be updated according to the necessity and request.
2. Summary of the change
/example->/examplesdattrirepo (add cifar10 support,AttributionTaskwill load the checkpoints in the same device as the model, auc will transform the score to cpu to avoid user redundant code).3. What tests have been added/updated for the change?