add incremental processing example

[shcheklein]

Adds an example of delta (incremental) processing.

TODO

• fix tests / make it run as a test

[cloudflare-workers-and-pages]

Deploying datachain-documentation with    Cloudflare Pages

Latest commit:	e114989
Status:	✅  Deploy successful!
Preview URL:	https://d3b2d34e.datachain-documentation.pages.dev
Branch Preview URL:	https://add-delta-example.datachain-documentation.pages.dev

View logs

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.97%. Comparing base (082c4e3) to head (e114989).
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1101   +/-   ##
=======================================
  Coverage   87.97%   87.97%           
=======================================
  Files         148      148           
  Lines       12747    12747           
  Branches     1783     1783           
=======================================
  Hits        11214    11214           
  Misses       1094     1094           
  Partials      439      439           

Flag	Coverage Δ
datachain	87.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ilongin]

Note that without explicit delta_compare it will look for all fields in schema except file.path (since it's in delta_on already) to say if file is changed or not. This means that two rows need to be identical (all fields the same) in order to not count them as "modified / changed". If it count's them as changed there is no performance gain in delta update. You usually want to set `delta_compare=["file.version", "file.etag"].

There is the case though with non-versioned sources where file.version and file.etag are randomly set every time on re-index which causes the same thing to happen regardless as it will catch everything as modified. In this cases, and in every other case where user doesn't even want or can track changed rows, workaround is to put delta_update to be the same as delta_on but we need a better way.
Options are:

• make default delta_compare=None to disable tracking changed rows instead to look into all fields. If we go with this path then DataChain.compare() and DataChain.diff() needs to be changed as well to be consistent.
• Add additional flag for this, e.g delta_ignore_changed.

I'm leaning more on first option, although then user then needs to explicitly set all fields in some cases (we loose "shortcut" of default being all fields). I don't have strong opinion though.

[shcheklein]

thanks, I think we are fine in this particular example (?)

There is the case though with non-versioned sources where file.version and file.etag are randomly set every time on re-index

where did you experience this?

[shcheklein]

@ilongin please let me know ^^

[ilongin]

I thought I saw it in one of our gs buckets but now I checked and it seems like it's ok. version and etag should be set to empty string if they don't exist.

Regarding your example, yea you don't need to put anything as non of the columns will be changed since you only append new files. If you would re-create files every time when calling generate_next_file then it would be a problem as for local files etag we put mtime which would mean that delta would find all files being modified every time.