feat: Add upload parquet and manifest files #25

SantanaTiago · 2025-04-23T10:58:52Z

This PR adds upload of parquet and manifest file.
Improve use of temporary files by creating temporary folder at top.

Resolves #20 #22

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

…pload it later to s3 Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

…hash to filename Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

mergify · 2025-04-23T10:59:26Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

ceberam

@SantanaTiago some observations, since maybe the feature was not explained in detail:

the parquet file should be a means to pack the conversion of several files into a single compressed file, instead of creating 1 parquet file for each converted document.
to make the parquet files manageable and still make best use of the storage, we could set a maximum size of 500Mb. We keep piling conversion results in a parquet file and as soon as we reach 500Mb, we stop and we start piling in a new parquet file.
the manifest file should then help users locate a converted PDF file in the set of parquet files. In addition, the row field could contain the row in the specific parquet file where the conversion results are located.

docling_jobkit/connectors/s3_helper.py

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

…parquet file Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

ceberam

@SantanaTiago Thanks for this PR.
I posted some questions we may want to clarify.
Also please resolve the conflicts with uv.lock.

ceberam · 2025-05-22T08:44:47Z

docling_jobkit/connectors/s3_helper.py


 from docling_jobkit.model.s3_inputs import S3Coordinates

 logging.basicConfig(level=logging.INFO)

+# Set the maximum file size of parquet to 100MB
+MAX_PARQUET_FILE_SIZE = 100 * 1024 * 1024


Let's set it to 500MB since it is still manageable and we thus make it a bit more efficient.

ceberam · 2025-05-22T08:48:17Z

docling_jobkit/connectors/s3_helper.py

+classifier_labels = [
+    "bar_chart",
+    "bar_code",
+    "chemistry_markush_structure",
+    "chemistry_molecular_structure",
+    "flow_chart",
+    "icon",
+    "line_chart",
+    "logo",
+    "map",
+    "other",
+    "pie_chart",
+    "qr_code",
+    "remote_sensing",
+    "screenshot",
+    "signature",
+    "stamp",
+]


Just a comment: we need to coordinate with @nikos-livathinos and @Matteo-Omenetti to define these labels in a data schema for better integration

ceberam · 2025-05-22T08:49:15Z

docling_jobkit/connectors/s3_helper.py

@@ -123,25 +156,35 @@ def generate_presign_url(
        return None


-def get_source_files(s3_source_client, s3_source_resource, s3_coords):
+def get_source_files(s3_source_client, s3_source_resource, s3_coords: S3Coordinates):


Can we also provide type hints to the other method arguments?

ceberam · 2025-05-22T08:59:22Z

docling_jobkit/connectors/s3_helper.py

+                    manifest[f"{parquet_file_name}"] = {
+                        "filenames": current_df["filename"].tolist(),
+                        "doc_hashes": current_df["doc_hash"].tolist(),
+                        "row_number": 3,
+                        "timestamp": timestamp,


Could you please share a mock example of how that manifest file would look like?
I think that if we do not need nested structures, a CSV file could be more efficient.

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

SantanaTiago added 12 commits April 21, 2025 12:01

add method export to parquet

924ca50

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

typo

00f744c

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add missing dependencies for parquet export

474df33

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add upload of manifest file. add download file to temporary file to u…

e85400c

…pload it later to s3 Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

typo

05ca2e4

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

changed usage of temporaryfile to temporary directory at top

97e101e

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

improved manifest file

07bf861

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

removed pages info from manifest files

52c8de2

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add check file size limit before download. changed save s3 path from …

9466f17

…hash to filename Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

fix s3 path name

f3631d0

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

typo

4da5ef4

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

improve key_prefix path in get_source_files

0958312

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

SantanaTiago requested review from ceberam, dolfim-ibm and vku-ibm April 23, 2025 10:59

removed unused key from parquet

80d7b7c

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

ceberam requested changes Apr 25, 2025

View reviewed changes

docling_jobkit/connectors/s3_helper.py Outdated Show resolved Hide resolved

ceberam mentioned this pull request Apr 29, 2025

fix: export pdf, check existing conversions #26

Merged

merge with main

f681710

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

SantanaTiago marked this pull request as draft May 12, 2025 08:34

SantanaTiago added 9 commits May 12, 2025 09:36

changed parquet to a single file in conversion.

c97b7d1

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

merge with main

22e7f76

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

redone uv lock

a9bc859

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

merge with main

e9d4aa0

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

set parquet engine

cef5357

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add check file is not empty before appending data

57d40a9

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

changed engine in the create of parquet file

8471edb

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

disable parquet export for testing

c378cb0

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

set export parquet default to True. add exception handler in writing …

236f55c

…parquet file Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

SantanaTiago added 8 commits May 13, 2025 17:49

commented pdf file write in parquet file

59704c5

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

changed parquet write file from a single dataframe

da0555e

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

typo

2da3ae6

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add debug logs

b080ce4

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add return to document_to_dataframe

3f64da4

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add write parquet file with max size file. add filename to parquet info

f2c785d

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

improved naming of parquet and manifest files. clear dev logs

71327d2

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

removed usage of Optional and Union

c6e37da

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

SantanaTiago marked this pull request as ready for review May 15, 2025 08:05

ceberam reviewed May 22, 2025

View reviewed changes

SantanaTiago added 3 commits May 22, 2025 15:55

merge with main

39a0edc

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

run uv lock

659422b

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

add missing types. set max parquet size to 500MB

b7afa33

Signed-off-by: Tiago Santana <54704492+SantanaTiago@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add upload parquet and manifest files #25

feat: Add upload parquet and manifest files #25

Uh oh!

SantanaTiago commented Apr 23, 2025

Uh oh!

mergify bot commented Apr 23, 2025

Uh oh!

ceberam left a comment

Uh oh!

Uh oh!

ceberam left a comment

Uh oh!

ceberam May 22, 2025

Uh oh!

ceberam May 22, 2025

Uh oh!

ceberam May 22, 2025

Uh oh!

ceberam May 22, 2025

Uh oh!

Uh oh!

feat: Add upload parquet and manifest files #25

Are you sure you want to change the base?

feat: Add upload parquet and manifest files #25

Uh oh!

Conversation

SantanaTiago commented Apr 23, 2025

Uh oh!

mergify bot commented Apr 23, 2025

Merge Protections

🟢 Enforce conventional commit

Uh oh!

ceberam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ceberam left a comment

Choose a reason for hiding this comment

Uh oh!

ceberam May 22, 2025

Choose a reason for hiding this comment

Uh oh!

ceberam May 22, 2025

Choose a reason for hiding this comment

Uh oh!

ceberam May 22, 2025

Choose a reason for hiding this comment

Uh oh!

ceberam May 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!