USE 355 - index current embeddings for a source

[ghukill]

Purpose and background context

This PR allows us to use TIM to bulk update pre-existing documents for a given source with all current embeddings for that source.

As noted in both the ticket and commits, the first pass focused only on indexing a single ETL run, which remains the common path in the ETL StepFunction. But until we're relying on the ETL StepFunction entirely, there is value in having the ability to update a source in Opensearch with current embeddings for that source. More specifically, to do so without fully reindexing the source as reindex-source would do.

How can a reviewer manually see the effects of these changes?

1- Set Dev1 AWS credentials

2- Run a bulk updating for a small source like libguides:

pipenv run tim --verbose \
bulk-update-embeddings \
--source libguides \
s3://timdex-extract-dev-222053980223/dataset

When complete, observe the following results:

{"updated": 0, "skipped": 279, "errors": 0, "total": 279}

Because embeddings already existed, and were identical to the ones used for updating, they are all skipped.

You could instead perform a full re-index of a source, e.g. gismit:

pipenv run tim --verbose reindex-source --source gismit s3://timdex-extract-dev-222053980223/dataset

Results:

{"index": {"created": 2043, "updated": 0, "errors": 16, "total": 2059}, "update": {"updated": 2043, "errors": 16, "total": 2059, "skipped": 0}}

This is interesting for a couple of reasons:

• we see that during indexing (creation) of documents we have 16 errors
• during updating of documents we have 16 errors because the associated Opensearch document didn't exist
• but the other 2043 are updated successfully

Includes new or updated dependencies?

YES

Changes expectations for external applications?

YES

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-355

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[coveralls]

Pull Request Test Coverage Report for Build 21405528265

Details

• 10 of 12 (83.33%) changed or added relevant lines in 2 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.2%) to 95.573%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
tim/opensearch.py	2	4	50.0%

Files with Coverage Reduction	New Missed Lines	%
tim/opensearch.py	1	93.81%

Totals
Change from base Build 20438471979:	-0.2%
Covered Lines:	475
Relevant Lines:	497

---

💛 - Coveralls

[jonavellecuerdo]

@ghukill Can you clarify what the cause of those 16 errors were? 🤔 Are they records that were deleted since the last time we created embeddings for the gismit source?

[ghukill]

@ghukill Can you clarify what the cause of those 16 errors were? 🤔 Are they records that were deleted since the last time we created embeddings for the gismit source?

Those errors happen each time we do a "full" run for gismit. The errors are the strict geospatial fields getting bad values. It's something we need to resolve, as they aren't getting indexed into Opensearch (I don't think).

But in the context of this PR, I think it revealed that for some reason we may occassionally have documents missing in Opensearch and we should not fail an entire updating pass -- e.g. for embeddings -- just because a subset don't have anything to update. I fully acknowledge this may be at odds with discussions or decisions when that updating logic was written, when it felt correct to immediately throw an error for a missing document, but this felt like a good example where that behavior is not ideal.

In short: it feels like "updating" work, e.g. embeddings, should update docs that exist, but not fail entirely if some documents don't exist in Opensearch.