Skip to content

TIMX 491 - remove parquet feature flags #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ghukill merged 3 commits into from
May 9, 2025
Merged

TIMX 491 - remove parquet feature flags #363

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f63429c
Replace pipenv check with pip-audit
ghukill May 6, 2025
cdb5c84
Remove parquet dataset feature flags
ghukill May 7, 2025
280262f
Update tests for removal of bulk-index and bulk-delete
ghukill May 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ repos:
language: system
pass_filenames: true
types: ["python"]
- id: safety
name: safety
entry: pipenv check
- id: pip-audit
name: pip-audit
entry: pipenv run pip-audit
language: system
pass_filenames: false
5 changes: 2 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,10 @@ ruff: # Run 'ruff' linter and print a preview of errors
pipenv run ruff check .

safety: # Check for security vulnerabilities and verify Pipfile.lock is up-to-date
pipenv check
pipenv run pip-audit
pipenv verify

lint-apply: # Apply changes with 'black' and resolve 'fixable errors' with 'ruff'
black-apply ruff-apply
lint-apply: black-apply ruff-apply # Apply changes with 'black' and resolve 'fixable errors' with 'ruff'

black-apply: # Apply changes with 'black'
pipenv run black .
Expand Down
2 changes: 2 additions & 0 deletions Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ pre-commit = "*"
pytest = "*"
ruff = "*"
vcrpy = "*"
pip-audit = "*"
ipython = "*"

[requires]
python_version = "3.12"
Expand Down
1,611 changes: 996 additions & 615 deletions Pipfile.lock
View file
Open in desktop

Large diffs are not rendered by default.

54 changes: 26 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,33 +115,31 @@ SENTRY_DSN=### If set to a valid Sentry DSN, enables Sentry exception monitoring
All CLI commands can be run with `pipenv run`.

```
Usage: tim [OPTIONS] COMMAND [ARGS]...

TIM provides commands for interacting with OpenSearch indexes.
For more details on a specific command, run tim COMMAND -h.

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --url -u TEXT The OpenSearch instance endpoint minus the http scheme, e.g. │
│ 'search-timdex-env-1234567890.us-east-1.es.amazonaws.com'. If not provided, will attempt to get from the │
│ TIMDEX_OPENSEARCH_ENDPOINT environment variable. Defaults to 'localhost'. │
│ --verbose -v Pass to log at debug level instead of info │
│ --help -h Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Get cluster-level information ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ping Ping OpenSearch and display information about the cluster. │
│ indexes Display summary information about all indexes in the cluster. │
│ aliases List OpenSearch aliases and their associated indexes. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Index management commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ create Create a new index in the cluster. │
│ delete Delete an index. │
│ promote Promote index as the primary alias and add it to any additional provided aliases. │
│ demote Demote an index from all its associated aliases. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Bulk record processing commands ───────────────────────────────────────────────────────────────────────────────────────────────────╮
│ bulk-index Bulk index records into an index. │
│ bulk-delete Bulk delete records from an index. │
│ bulk-update Bulk update records from an index. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: tim [OPTIONS] COMMAND [ARGS]...

TIM provides commands for interacting with OpenSearch indexes.
For more details on a specific command, run tim COMMAND -h.

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --url -u TEXT The OpenSearch instance endpoint minus the http scheme, e.g. │
│ 'search-timdex-env-1234567890.us-east-1.es.amazonaws.com'. If not provided, will attempt to get from the │
│ TIMDEX_OPENSEARCH_ENDPOINT environment variable. Defaults to 'localhost'. │
│ --verbose -v Pass to log at debug level instead of info │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Get cluster-level information ────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ping Ping OpenSearch and display information about the cluster. │
│ indexes Display summary information about all indexes in the cluster. │
│ aliases List OpenSearch aliases and their associated indexes. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Index management commands ────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ create Create a new index in the cluster. │
│ delete Delete an index. │
│ promote Promote index as the primary alias and add it to any additional provided aliases. │
│ demote Demote an index from all its associated aliases. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Bulk record processing commands ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ bulk-update Bulk update records for an index. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

36 changes: 36 additions & 0 deletions tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import pytest
import vcr
from click.testing import CliRunner
from timdex_dataset_api import TIMDEXDataset

import tim.opensearch as tim_os

Expand Down Expand Up @@ -33,3 +34,38 @@ def test_opensearch_client():
@pytest.fixture
def runner():
return CliRunner()


@pytest.fixture
def timdex_dataset() -> TIMDEXDataset:
td = TIMDEXDataset("tests/fixtures/dataset")
td.load()
return td


@pytest.fixture
def five_valid_index_libguides_records(timdex_dataset):
return timdex_dataset.read_transformed_records_iter(
run_id="85cfe316-089c-4639-a5af-c861a7321493"
)


@pytest.fixture
def one_invalid_index_libguides_records(timdex_dataset):
return timdex_dataset.read_transformed_records_iter(
run_id="21e7f272-7b96-480c-9c25-36075355fc4c"
)


@pytest.fixture
def one_valid_delete_libguides_records(timdex_dataset):
return timdex_dataset.read_transformed_records_iter(
run_id="59d938b9-df61-481b-bec9-9d9eb8fbf21c"
)


@pytest.fixture
def one_valid_delete_libguides_records_not_found(timdex_dataset):
return timdex_dataset.read_transformed_records_iter(
run_id="3718935d-2bc0-4385-919e-c7a83238215e"
)
72 changes: 0 additions & 72 deletions tests/fixtures/cassettes/cli/bulk_delete_with_index_name_success.yaml
View file
Open in desktop

This file was deleted.

72 changes: 0 additions & 72 deletions tests/fixtures/cassettes/cli/bulk_delete_with_source_success.yaml
View file
Open in desktop

This file was deleted.

48 changes: 48 additions & 0 deletions tests/fixtures/cassettes/opensearch/TO_DELETE/bulk_delete_record_not_found.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
interactions:
- request:
body: '{"delete":{"_id":"no-record-here","_index":"alma-2022-09-01t00-00-00"}}

'
headers:
Content-Length:
- '72'
content-type:
- application/json
user-agent:
- opensearch-py/2.0.1 (Python 3.10.8)
method: POST
uri: http://localhost:9200/_bulk
response:
body:
string: '{"took":5,"errors":false,"items":[{"delete":{"_index":"alma-2022-09-01t00-00-00","_type":"_doc","_id":"no-record-here","_version":1,"result":"not_found","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":27,"_primary_term":1,"status":404}}]}'
headers:
content-length:
- '248'
content-type:
- application/json; charset=UTF-8
status:
code: 200
message: OK
- request:
body: null
headers:
Content-Length:
- '0'
content-type:
- application/json
user-agent:
- opensearch-py/2.0.1 (Python 3.10.8)
method: POST
uri: http://localhost:9200/alma-2022-09-01t00-00-00/_refresh
response:
body:
string: '{"_shards":{"total":2,"successful":1,"failed":0}}'
headers:
content-length:
- '49'
content-type:
- application/json; charset=UTF-8
status:
code: 200
message: OK
version: 1
0 ...ch/bulk_delete_record_weird_response.yaml → ...TE/bulk_delete_record_weird_response.yaml
View file
Open in desktop
File renamed without changes.
Loading