Skip to content

Add BATCH_SIZE env var #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
higs4281 merged 3 commits into from
Sep 9, 2025
Merged

Add BATCH_SIZE env var #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b6db078
Add an optional env var for controlling indexing batch size
higs4281 Sep 9, 2025
a897f8b
fix double import
higs4281 Sep 9, 2025
db4bedf
Merge branch 'main' into add-batch-size-env-var
higs4281 Sep 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env_example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ export VENV_NAME="ccdb-data-pipeline"
# set this value to limit the number of records to load to reduce load time
# export MAX_RECORDS=10000

# set an optional batch size for bulk indexing runs; defaults to 2000
# export BATCH_SIZE=5000

# export INPUT_S3_BUCKET="foo"
# export INPUT_S3_KEY="bar"
# export INPUT_S3_KEY_METADATA="barrr"
Expand Down
5 changes: 4 additions & 1 deletion complaints/ccdb/index_ccdb.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import json
import os
import sys
from functools import partial

Expand All @@ -12,6 +13,8 @@
get_es_connection)
from common.log import setup_logging

BATCH_SIZE = os.getenv("BATCH_SIZE", 2000)

# -----------------------------------------------------------------------------
# Enhancing Functions
# -----------------------------------------------------------------------------
Expand Down Expand Up @@ -137,7 +140,7 @@ def yield_chunked_docs(get_data_function, data, chunk_size):

def index_json_data(
es, logger, doc_type_name, settings_json, mapping_json, data, index_name,
backup_index_name, alias, chunk_size=2000, qas_timestamp=0
backup_index_name, alias, chunk_size=BATCH_SIZE, qas_timestamp=0
):
settings = load_json(logger, settings_json)
mapping = load_json(logger, mapping_json)
Expand Down