Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes all sensible occurrences of the just commands to have them run using ov #4483

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/new_source_suggestion.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ body:
- The script must not use anything from `catalog/dags/retired/providers/provider_api_scripts/modules/etlMods.py`, since that module is deprecated.
- If the provider API has can be queried by 'upload date' or something similar, the script should take a `--date` parameter when run as a script, giving the date for which we should collect images. The form should be `YYYY-MM-DD` (so, the script can be run via `python my_favorite_provider.py --date 2018-01-01`).
- The script must provide a main function that takes the same parameters as from the CLI. In our example from above, we'd then have a main function `my_favorite_provider.main(date)`. The main should do the same thing calling from the CLI would do.
- The script *must* conform to [PEP8][pep8]. Please use `just lint` before committing.
- The script *must* conform to [PEP8][pep8]. Please use `./ov just lint` before committing.
- The script should use small, testable functions.
- The test suite for the script may break PEP8 rules regarding long lines where appropriate (e.g., long strings for testing).

Expand Down
6 changes: 3 additions & 3 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ Fixes #[issue number] by @[issue author]
- [ ] I added or updated tests for the changes I made (if applicable).
- [ ] I added or updated documentation (if applicable).
- [ ] I tried running the project locally and verified that there are no visible errors.
- [ ] I ran the DAG documentation generator (`just catalog/generate-docs` for catalog
PRs) or the media properties generator (`just catalog/generate-docs media-props`
for the catalog or `just api/generate-docs` for the API) where applicable.
- [ ] I ran the DAG documentation generator (`./ov just catalog/generate-docs` for catalog
PRs) or the media properties generator (`./ov just catalog/generate-docs media-props`
for the catalog or `./ov just api/generate-docs` for the API) where applicable.

[best_practices]:
https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project#_commit_guidelines
Expand Down
4 changes: 2 additions & 2 deletions .vale/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ the Openverse monorepo, refer to the README at

To run Vale with Openverse's configuration, use the just recipe:

```
$ just .vale/run
```bash
./ov just .vale/run
```

This recipe _always_ builds Vale. The Openverse Vale docker image is fast to
Expand Down
2 changes: 1 addition & 1 deletion catalog/templates/template_test.py_template
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ TODO: Add additional tests for any methods you added in your subclass.
Try to test edge cases (missing keys, different data types returned, Nones, etc).
You may also need to update the given test names to be more specific.

Run your tests locally with `just test -k {provider_underscore}`
Run your tests locally with `./ov just catalog/test -k {provider_underscore}`
"""

import json
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Run these tests locally with `just test -k cc_mixter`"""
"""Run these tests locally with `./ov just catalog/test -k cc_mixter`"""

import json
from pathlib import Path
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
# to pull a sample of records without breaking referential integrity.
# - Putting your final zipped test files in /tests/s3-data/inaturalist-open-data
# so that they will be synced over to minio.
# - Run `just down -v` and then `just recreate` to make sure that the test data gets to
# - Run `./ov just down -v` and then `./ov just recreate` to make sure that the test data gets to
# the test s3 instance. That process may take on the order of 15 minutes for the full
# dataset. You'll know that it's done when the s3-load container in docker exits.
# - Then, in airflow, trigger the dag possibly with configuration
Expand Down
4 changes: 2 additions & 2 deletions docker/minio/load_to_s3_entrypoint.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash
# This does *not* allow for testing permissions issues that may come up in real AWS.
# And, if you remove files from /tests/s3-data, you will need to use `just down -v`
# and `just up` or `just recreate` to see the minio bucket without those files.
# And, if you remove files from /tests/s3-data, you will need to use `./ov just down -v`
# and `./ov just up` or `./ov just recreate` to see the minio bucket without those files.
# Loop through subdirectories mounted to the volume and load them to s3/minio.
# This takes care of filesystem delays on some local dev environments that may make
# minio miss files included directly in the minio volume.
Expand Down
8 changes: 4 additions & 4 deletions documentation/api/guides/https.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Additionally, you will need to install
1. Create certificates for NGINX to use.

```bash
just docker/nginx/cert
./ov just docker/nginx/cert
```

This will create a certificate file `openversse.crt` and a key file
Expand All @@ -29,7 +29,7 @@ Additionally, you will need to install
services.

```bash
just api/up
./ov just api/up
```

The `api/up` recipe orchestrates the following services: `cache`, `db`,
Expand All @@ -41,8 +41,8 @@ Additionally, you will need to install
3. Make an API call over HTTPS.

```bash
just api/stats images https://localhost:50243
just _curl-get "images/stats/" https://localhost:50243
./ov just api/stats images https://localhost:50243
./ov just _curl-get "images/stats/" https://localhost:50243
curl "https://localhost:50243/v1/images/stats/"
[{"source_name":"flickr","display_name":"Flickr","source_url":"https://www.flickr.com","logo_url":null,"media_count":2500},{"source_name":"stocksnap","display_name":"StockSnap","source_url":"https://stocksnap.io","logo_url":null,"media_count":2500}]%
```
16 changes: 8 additions & 8 deletions documentation/api/guides/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ you need to run this.
services.

```bash
just api/up
./ov just api/up
```

The `api/up` recipe orchestrates the following services: `cache`, `db`,
Expand All @@ -56,24 +56,24 @@ you need to run this.
4. Load the sample data. This step can take a few minutes to complete.

```bash
just api/init
./ov just api/init
```

````{admonition} Troubleshooting
If this step fails, cleaning up and restarting usually fixes it.

```bash
just down -v
just api/init
./ov just down -v
./ov just api/init
```
````

5. With the data loaded, the API can now return JSON responses to your HTTP
requests.

```bash
just api/stats
just _curl-get "images/stats/" http://localhost:50280
./ov just api/stats
./ov just _curl-get "images/stats/" http://localhost:50280
curl "http://localhost:50280/v1/images/stats/"
[{"source_name":"flickr","display_name":"Flickr","source_url":"https://www.flickr.com","logo_url":null,"media_count":2500},{"source_name":"stocksnap","display_name":"StockSnap","source_url":"https://stocksnap.io","logo_url":null,"media_count":2500}]%
```
Expand All @@ -84,7 +84,7 @@ you need to run this.
transform it.

```bash
just api/stats | jq '.[0]'
./ov just api/stats | jq '.[0]'
{
"source_name": "flickr",
"display_name": "Flickr",
Expand All @@ -93,7 +93,7 @@ you need to run this.
"media_count": 2500
}

just api/stats 'audio' | jq '[.[] | .source_name]'
./ov just api/stats 'audio' | jq '[.[] | .source_name]'
[
"freesound",
"jamendo",
Expand Down
2 changes: 1 addition & 1 deletion documentation/api/guides/test.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Once you've made some changes to the codebase, it is important to run tests.
2. Run the tests.

```bash
just api/test
./ov just api/test
```

```{note}
Expand Down
6 changes: 3 additions & 3 deletions documentation/automations/guides/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ you need to run this.

## Installation

`just install` in the repository root directory will install all Python and
`./ov just install` in the repository root directory will install all Python and
Node.js dependencies for automations as well as other parts of the repository.
If you wish to install only dependencies for automations, run the following:

Expand All @@ -27,7 +27,7 @@ Run Python automation scripts using the `automations/python/run` just recipe.
Ex.:

```bash
just automations/python/run print_labels.py
./ov just automations/python/run print_labels.py
```

The recipe is an alias for running `pipenv run <script>` inside the
Expand All @@ -39,7 +39,7 @@ configuration as expected by many scripts.
Run Node.js automation scripts using the `automations/js/run` just recipe. Ex.:

```bash
just automations/js/run render-jinja.js
./ov just automations/js/run render-jinja.js
```

The recipe is an alias for running `pnpm exec <script>` inside the
Expand Down
22 changes: 11 additions & 11 deletions documentation/catalog/guides/adding_a_new_provider.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,18 +62,18 @@ that can be used to generate the files you'll need and get you started:
# MEDIA: Optionally, a space-delineated list of media types ingested by this provider
# (and supported by Openverse). If not provided, defaults to "image".

> just catalog/add-provider <PROVIDER_NAME> <ENDPOINT> <MEDIA>
> ./ov just catalog/add-provider <PROVIDER_NAME> <ENDPOINT> <MEDIA>

# Example usages:

# Creates a provider that supports just audio
> just catalog/add-provider TestProvider https://test.test/search audio
> ./ov just catalog/add-provider TestProvider https://test.test/search audio

# Creates a provider that supports images and audio
> just catalog/add-provider "Foobar Museum" https://foobar.museum.org/api/v1 image audio
> ./ov just catalog/add-provider "Foobar Museum" https://foobar.museum.org/api/v1 image audio

# Creates a provider that supports the default, just image
> just catalog/add-provider TestProvider https://test.test/search
> ./ov just catalog/add-provider TestProvider https://test.test/search
```

You should see output similar to this:
Expand Down Expand Up @@ -101,7 +101,7 @@ You can run the provider script directly from the command line via a just
recipe. This will open a bash shell inside the docker stack of the catalog.

```
just catalog/run
./ov just catalog/run
```

Now you can just run the script like so:
Expand Down Expand Up @@ -151,8 +151,8 @@ These are documented in the definition of the `ProviderWorkflow` dataclass.

<!--TODO: add docs for other options.-->

After adding your configuration, run `just up` and you should now have a fully
functioning provider DAG!
After adding your configuration, run `./ov just up` and you should now have a
fully functioning provider DAG!

<!--TODO: add and link to docs for how to run provider DAGs locally, preferably with images.-->

Expand All @@ -173,21 +173,21 @@ manually turn the DAG on in production.
selecting only the tests associated with the provider.

```bash
just catalog/test-session
./ov just catalog/test-session
pytest -k <provider_name>
```

Alternatively, the test selection can be run in Docker directly with:

```bash
just catalog/test -k <provider_name>
./ov just catalog/test -k <provider_name>
```

```{note}
Using `just catalog/test-session` opens Docker to access a shell which is set up to run
Using `./ov just catalog/test-session` opens Docker to access a shell which is set up to run
tests. This allows one to run tests repeatedly while potentially modifying the code,
without having to start the Docker container up each time the tests need to be run.
Running the tests on Docker directly (e.g. using `just catalog/test`) will spin up the
Running the tests on Docker directly (e.g. using `./ov just catalog/test`) will spin up the
container, run the selected tests if any are provided (or all by default) and then stop
and remove the container. That can be useful for ensuring that all tests pass if one
does not need to iterate and check the test failures repeatedly.
Expand Down
2 changes: 1 addition & 1 deletion documentation/catalog/guides/dag_testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ might need to get the API keys from the provider and add them to the
1. Refer to and follow the instructions at [Quickstart](./quickstart.md) to
setup and make sure the catalog service is up and running. If you have
successfully completed the general setup then this can be started by running
`just catalog/up`.
`./ov just catalog/up`.

2. Navigate to http://localhost:9090

Expand Down
30 changes: 16 additions & 14 deletions documentation/catalog/guides/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,11 @@ To set up the local python environment along with the pre-commit hook, run:
<!-- vale Vale.Repetition = NO -->

```shell
just venv
source .venv/bin/activate
./ov just catalog/venv
./ov bash
source catalog/.venv/bin/activate
just catalog/install
exit
```

<!-- vale Vale.Repetition = YES -->
Expand All @@ -63,15 +65,15 @@ The containers will be built when starting the stack up for the first time. If
you'd like to build them prior to that, run:

```shell
just build
./ov just build
```

### Environment

To set up environment variables run:

```shell
just env
./ov just env
```

This will generate a `.env` file which is used by the containers.
Expand All @@ -94,7 +96,7 @@ There is a [`docker-compose.yml`][dockercompose] provided in the
[`catalog`][cc_airflow] directory, so from that directory, run

```shell
just catalog/up
./ov just catalog/up
```

This results, among other things, in the following running containers:
Expand Down Expand Up @@ -126,10 +128,10 @@ The various services can be accessed using these links:
At this stage, you can run the tests via:

```shell
just catalog/test
./ov just catalog/test

# Alternatively, run all tests including longer-running ones
just catalog/test --extended
./ov just catalog/test --extended
```

Edits to the source files or tests can be made on your local machine, then tests
Expand All @@ -138,39 +140,39 @@ can be run in the container via the above command to see the effects.
If you'd like, it's possible to login to the webserver container via:

```shell
just catalog/shell
./ov just catalog/shell
```

If you just need to run an airflow command, you can use the `airflow` recipe.
Arguments passed to airflow must be quoted:

```shell
just run scheduler airflow config list
./ov just run scheduler airflow config list
```

To follow the logs of the running container:

```shell
just logs
./ov just logs
```

To begin an interactive [`pgcli` shell](https://www.pgcli.com/) on the database
container, run:

```shell
just catalog/pgcli
./ov just catalog/pgcli
```

If you'd like to bring down the containers, run

```shell
just down
./ov just down
```

To reset the test DB (wiping out all databases, schemata, and tables), run

```shell
just down -v
./ov just down -v
```

`docker volume prune` can also be useful if you've already stopped the running
Expand All @@ -180,7 +182,7 @@ stopped containers, not just catalog ones.
To fully recreate everything from the ground up, you can use:

```shell
just recreate
./ov just recreate
```

> **Note**: Any recipes or scripts which output files to the container's mounted
Expand Down
Loading
Loading