Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve gitutils.py by using the Github CLI when available #1088

Merged
merged 12 commits into from
Aug 17, 2023
Prev Previous commit
Next Next commit
Updating versions script with latest code
  • Loading branch information
mdemoret-nv committed Jul 14, 2023
commit 7758db11f6f08d2eb33e2e66dfa2edb35e8f8eb4
18 changes: 15 additions & 3 deletions ci/release/update-version.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,15 @@ NEXT_SHORT_TAG_PEP440=$(python -c "from setuptools.extern import packaging; prin

echo "Preparing release $CURRENT_FULL_VERSION (PEP ${CURRENT_SHORT_TAG_PEP440}) => $NEXT_FULL_VERSION (PEP ${NEXT_SHORT_TAG_PEP440})"

# Inplace sed replace; workaround for Linux and Mac
# Inplace sed replace; workaround for Linux and Mac. Accepts multiple files
function sed_runner() {
sed -i.bak ''"$1"'' $2 && rm -f ${2}.bak

pattern=$1
shift

for f in $@ ; do
sed -i.bak ''"$pattern"'' "$f" && rm -f "$f.bak"
done
}

# .gitmodules
Expand All @@ -70,7 +76,9 @@ sed_runner 's/'"VERSION ${CURRENT_FULL_VERSION}.*"'/'"VERSION ${NEXT_FULL_VERSIO
sed_runner "s/mrc=${CURRENT_SHORT_TAG}/mrc=${NEXT_SHORT_TAG}/g" docker/conda/environments/cuda11.8_dev.yml

# examples/digital_fingerprinting
sed_runner "s/v${CURRENT_FULL_VERSION}-runtime/v${NEXT_FULL_VERSION}-runtime/g" examples/digital_fingerprinting/production/docker-compose.yml
sed_runner "s/v${CURRENT_FULL_VERSION}-runtime/v${NEXT_FULL_VERSION}-runtime/g" \
examples/digital_fingerprinting/production/docker-compose.yml \
examples/digital_fingerprinting/production/Dockerfile
sed_runner "s/v${CURRENT_FULL_VERSION}-runtime/v${NEXT_FULL_VERSION}-runtime/g" examples/digital_fingerprinting/production/Dockerfile
sed_runner "s|blob/branch-${CURRENT_SHORT_TAG}|blob/branch-${NEXT_SHORT_TAG}|g" examples/digital_fingerprinting/starter/README.md

Expand All @@ -82,3 +90,7 @@ sed_runner "s|tree/branch-${CURRENT_SHORT_TAG}|tree/branch-${NEXT_SHORT_TAG}|g"
# docs/source/getting_started.md
# Only do the minor version here since the full version can mess up the examples
sed_runner "s/${CURRENT_SHORT_TAG}/${NEXT_SHORT_TAG}/g" docs/source/getting_started.md

# models/model-cards
sed_runner "s|blob/branch-${CURRENT_SHORT_TAG}|blob/branch-${NEXT_SHORT_TAG}|g" models/model-cards/*.md
sed_runner "s|tree/branch-${CURRENT_SHORT_TAG}|tree/branch-${NEXT_SHORT_TAG}|g" models/model-cards/*.md
6 changes: 3 additions & 3 deletions models/model-cards/abp-model-card.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ limitations under the License.

**Link:**

* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/abp-sample-nvsmi-training-data.json <br>
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/abp-sample-nvsmi-training-data.json <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

Expand All @@ -102,7 +102,7 @@ limitations under the License.

**Link:**

* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/validation-data/abp-validation-data.jsonlines <br>
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/validation-data/abp-validation-data.jsonlines <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

Expand Down Expand Up @@ -226,7 +226,7 @@ limitations under the License.

### Link the location of the training dataset's repository.

* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/abp-sample-nvsmi-training-data.json
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/abp-sample-nvsmi-training-data.json

### Is the model used in an application with physical safety impact?

Expand Down
32 changes: 16 additions & 16 deletions models/model-cards/dfp-model-card.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,39 +64,39 @@ The model architecture consists of an Autoencoder, where the reconstruction loss
* Linux<br>

## Model Version(s):
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/dfp-models/hammah-role-g-20211017-dill.pkl
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/dfp-models/hammah-user123-20211017-dill.pkl
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/dfp-models/hammah-role-g-20211017-dill.pkl
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/dfp-models/hammah-user123-20211017-dill.pkl

# Training & Evaluation:

## Training Dataset:

**Link:**
* https://github.com/nv-morpheus/Morpheus/tree/branch-23.07/models/datasets/training-data/cloudtrail
* https://github.com/nv-morpheus/Morpheus/tree/branch-23.11/models/datasets/training-data/cloudtrail

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

The training dataset consists of AWS CloudTrail logs. It contains logs from two entities, providing information about their activities within the AWS environment.
* [hammah-role-g-training-part1.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/cloudtrail/hammah-role-g-training-part1.json): 700 records <br>
* [hammah-role-g-training-part2.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/cloudtrail/hammah-role-g-training-part2.json): 1187 records <br>
* [hammah-user123-training-part2.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/cloudtrail/hammah-user123-training-part2.json): 1000 records <br>
* [hammah-user123-training-part3.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/cloudtrail/hammah-user123-training-part3.json): 1000 records <br>
* [hammah-user123-training-part4.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/cloudtrail/hammah-user123-training-part4.json): 387 records <br>
* [hammah-role-g-training-part1.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/cloudtrail/hammah-role-g-training-part1.json): 700 records <br>
* [hammah-role-g-training-part2.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/cloudtrail/hammah-role-g-training-part2.json): 1187 records <br>
* [hammah-user123-training-part2.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/cloudtrail/hammah-user123-training-part2.json): 1000 records <br>
* [hammah-user123-training-part3.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/cloudtrail/hammah-user123-training-part3.json): 1000 records <br>
* [hammah-user123-training-part4.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/cloudtrail/hammah-user123-training-part4.json): 387 records <br>

**Dataset License:**
* [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0)<br>

## Evaluation Dataset:
**Link:**
* https://github.com/nv-morpheus/Morpheus/tree/branch-23.07/models/datasets/validation-data/cloudtrail <br>
* https://github.com/nv-morpheus/Morpheus/tree/branch-23.11/models/datasets/validation-data/cloudtrail <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

The evaluation dataset consists of AWS CloudTrail logs. It contains logs from two entities, providing information about their activities within the AWS environment.
* [hammah-role-g-validation.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/validation-data/cloudtrail/hammah-role-g-validation.json): 314 records
* [hammah-user123-validation-part1.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/validation-data/cloudtrail/hammah-user123-validation-part1.json): 300 records
* [hammah-user123-validation-part2.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/validation-data/cloudtrail/hammah-user123-validation-part2.json): 300 records
* [hammah-user123-validation-part3.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/validation-data/cloudtrail/hammah-user123-validation-part3.json): 247 records
* [hammah-role-g-validation.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/validation-data/cloudtrail/hammah-role-g-validation.json): 314 records
* [hammah-user123-validation-part1.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/validation-data/cloudtrail/hammah-user123-validation-part1.json): 300 records
* [hammah-user123-validation-part2.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/validation-data/cloudtrail/hammah-user123-validation-part2.json): 300 records
* [hammah-user123-validation-part3.json](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/validation-data/cloudtrail/hammah-user123-validation-part3.json): 247 records

**Dataset License:**
* [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0)<br>
Expand Down Expand Up @@ -165,7 +165,7 @@ The evaluation dataset consists of AWS CloudTrail logs. It contains logs from tw

### List the steps explaining how this model works.
* The model works by training on baseline behaviors and subsequently detecting deviations from the established baseline, triggering alerts accordingly.
* [Training notebook](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/training-tuning-scripts/dfp-models/hammah-20211017.ipynb)
* [Training notebook](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/training-tuning-scripts/dfp-models/hammah-20211017.ipynb)

### Name the adversely impacted groups (protected classes) this has been tested to deliver comparable outcomes regardless of:
* Not Applicable
Expand All @@ -188,7 +188,7 @@ The evaluation dataset consists of AWS CloudTrail logs. It contains logs from tw
## Model Card ++ Saftey & Security Subcard

### Link the location of the training dataset's repository (if able to share).
* https://github.com/nv-morpheus/Morpheus/tree/branch-23.07/models/datasets/training-data/cloudtrail
* https://github.com/nv-morpheus/Morpheus/tree/branch-23.11/models/datasets/training-data/cloudtrail

### Is the model used in an application with physical safety impact?
* No
Expand Down Expand Up @@ -255,7 +255,7 @@ The evaluation dataset consists of AWS CloudTrail logs. It contains logs from tw
* No

### Are we able to identify and trace source of dataset?
* Yes ([fully synthetic dataset](https://github.com/nv-morpheus/Morpheus/tree/branch-23.07/models/datasets/training-data/cloudtrail))
* Yes ([fully synthetic dataset](https://github.com/nv-morpheus/Morpheus/tree/branch-23.11/models/datasets/training-data/cloudtrail))

### Does data labeling (annotation, metadata) comply with privacy laws?
* Not applicable (as the dataset is fully synthetic)
Expand Down
2 changes: 1 addition & 1 deletion models/model-cards/gnn-fsi-model-card.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ This model is an example of a fraud detection pipeline using a graph neural netw
### What training is recommended for developers working with this model? If none, please state "none."
* Familiarity with the Morpheus SDK is recommended for developers working with this model.
### Link the relevant end user license agreement
* [Apache 2.0](https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/LICENSE)
* [Apache 2.0](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/LICENSE)

## Model Card ++ Saftey & Security Subcard

Expand Down
2 changes: 1 addition & 1 deletion models/model-cards/phishing-model-card.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ limitations under the License.

**Link:**

* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/validation-data/phishing-email-validation-data.jsonlines <br>
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/validation-data/phishing-email-validation-data.jsonlines <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

Expand Down
6 changes: 3 additions & 3 deletions models/model-cards/root-cause-analysis-model-card.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ limitations under the License.

**Link:**

* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/root-cause-training-data.csv <br>
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/root-cause-training-data.csv <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

Expand All @@ -103,7 +103,7 @@ limitations under the License.

**Link:**

* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/validation-data/root-cause-validation-data-input.jsonlines <br>
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/validation-data/root-cause-validation-data-input.jsonlines <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

Expand Down Expand Up @@ -216,7 +216,7 @@ limitations under the License.
## Model Card ++ Saftey & Security Subcard

### Link the location of the training dataset's repository.
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/models/datasets/training-data/root-cause-training-data.csv
* https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/models/datasets/training-data/root-cause-training-data.csv

### Is the model used in an application with physical safety impact?
* No
Expand Down