Switch csv and tsv method 'sv' from ReadAll() to stream each record with Read() #355

splashing-atom · 2022-12-16T17:01:11Z

This should improve memory usage when parsing large files and checking if they match CSV/TSV.

fixes #354

…ith Read(), to improve memory usage. related to issue gabriel-vasile#354

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3.1.0 to 3.1.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md) - [Commits](codecov/codecov-action@v3.1.0...v3.1.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3.3.0 to 3.5.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v3.3.0...v3.5.0) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/checkout](https://github.com/actions/checkout) from 3.0.2 to 3.3.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3.0.2...v3.3.0) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.1.22 to 2.2.4. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2.1.22...v2.2.4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…e#360) Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.2.0 to 3.4.0. - [Release notes](https://github.com/golangci/golangci-lint-action/releases) - [Commits](golangci/golangci-lint-action@v3.2.0...v3.4.0) --- updated-dependencies: - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.2.4 to 2.2.5. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2.2.4...v2.2.5) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.7.0 to 0.8.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.7.0...v0.8.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: ioutil deprecated and add ignore file * fix: delete ignore file Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.8.0 to 0.10.0. - [Commits](golang/net@v0.8.0...v0.10.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Revert ioutil deprecation commit While go 1.16 is 2 years old, I feel like I should not go out of my way to break the library for users on go versions lower that 1.16. This commit reverts part of the changes from gabriel-vasile#392. * fix some doc comments for go v1.19 rules * Fix wrong total number of supported mimes The mistake was introduced in a merge commit.

* Remove old travis build status link from readme * Remove unneeded go-version tag in CI It is not supported inside golangci-lint-action and triggers a warning

One group is for gomod and one is for github actions PRs

Bumps the gomod group with 1 update: [golang.org/x/net](https://github.com/golang/net). - [Commits](golang/net@v0.10.0...v0.12.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor dependency-group: gomod ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps the github-actions group with 5 updates: | Package | Update | | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | 3.3.0 to 3.5.3 | | [github/codeql-action](https://github.com/github/codeql-action) | 2.2.5 to 2.20.4 | | [actions/setup-go](https://github.com/actions/setup-go) | 3.5.0 to 4.0.1 | | [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) | 3.4.0 to 3.6.0 | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | 3.1.1 to 3.1.4 | Updates `actions/checkout` from 3.3.0 to 3.5.3 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3.3.0...v3.5.3) Updates `github/codeql-action` from 2.2.5 to 2.20.4 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2.2.5...v2.20.4) Updates `actions/setup-go` from 3.5.0 to 4.0.1 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v3.5.0...v4.0.1) Updates `golangci/golangci-lint-action` from 3.4.0 to 3.6.0 - [Release notes](https://github.com/golangci/golangci-lint-action/releases) - [Commits](golangci/golangci-lint-action@v3.4.0...v3.6.0) Updates `codecov/codecov-action` from 3.1.1 to 3.1.4 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v3.1.1...v3.1.4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com>

These files were introduced when changing csv detection from allocating all results at once to streaming each line. This change improves performance but the functionality remains exactly the same. Since the functionality is unchanged, existing test cases should suffice.

gabriel-vasile

Thank you @splashing-atom and sorry for the late response.
Considering your change improves performance without affecting the functionality, I will remove the additional test cases.

ankon · 2023-10-18T13:28:57Z

internal/magic/text_csv.go

+		lines++
+	}
+
+	return r.FieldsPerRecord > 1 && lines > 1


Given that this only checks for "lines > 1": The loop above could just break out, and there is no point at all to exhaust the reader (which could still be millions of lines?)

The ReadAll() method would return error if any line in the file was not valid CSV. For this reason the loop calls Read() on each line and returns false overall if any line is not valid CSV.

Breaking out of the loop when lines > 1 means input like this would wrongfully pass as csv:

val1, val2, val3 val4, val5, val6 some text that is not related to the csv above

Thanks for checking and replying!

…#149) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/gabriel-vasile/mimetype](https://togithub.com/gabriel-vasile/mimetype) | indirect | patch | `v1.4.2` -> `v1.4.3` | --- ### Release Notes <details> <summary>gabriel-vasile/mimetype (github.com/gabriel-vasile/mimetype)</summary> ### [`v1.4.3`](https://togithub.com/gabriel-vasile/mimetype/releases/tag/v1.4.3) [Compare Source](https://togithub.com/gabriel-vasile/mimetype/compare/v1.4.2...v1.4.3) #### What's Changed - Switch csv and tsv method 'sv' from ReadAll() to stream each record with Read() by [@splashing-atom](https://togithub.com/splashing-atom) in [https://github.com/gabriel-vasile/mimetype/pull/355](https://togithub.com/gabriel-vasile/mimetype/pull/355) - Bump golang.org/x/net from 0.8.0 to 0.17.0 by [@dependabot](https://togithub.com/dependabot) in [https://github.com/gabriel-vasile/mimetype/pull/441](https://togithub.com/gabriel-vasile/mimetype/pull/441) - enable reusing records in csv/tsv detection by [@gabriel-vasile](https://togithub.com/gabriel-vasile) in [https://github.com/gabriel-vasile/mimetype/pull/443](https://togithub.com/gabriel-vasile/mimetype/pull/443) #### New Contributors - [@splashing-atom](https://togithub.com/splashing-atom) made their first contribution in [https://github.com/gabriel-vasile/mimetype/pull/355](https://togithub.com/gabriel-vasile/mimetype/pull/355) **Full Changelog**: gabriel-vasile/mimetype@v1.4.2...v1.4.3 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

switch csv and tsv method 'sv' from ReadAll() to stream each record w…

fe45260

…ith Read(), to improve memory usage. related to issue gabriel-vasile#354

splashing-atom force-pushed the streaming-csv-tsv-detection branch from 62635cd to fe45260 Compare December 16, 2022 17:15

viprerk and others added 18 commits December 17, 2022 11:15

add testcases for one/two line csv/tsv

6677711

Upgrade golang.org/x/net dependency (gabriel-vasile#365)

e3ac364

replace deprecated ioutil calls (gabriel-vasile#392)

dbcd5ed

* chore: ioutil deprecated and add ignore file * fix: delete ignore file Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>

Remove old travis build status link from readme (gabriel-vasile#407)

8bdb082

* Remove old travis build status link from readme * Remove unneeded go-version tag in CI It is not supported inside golangci-lint-action and triggers a warning

Group all dependabot PRs together (gabriel-vasile#409)

189fcfd

One group is for gomod and one is for github actions PRs

Merge branch 'master' into streaming-csv-tsv-detection

2d392b2

gabriel-vasile approved these changes Oct 10, 2023

View reviewed changes

gabriel-vasile merged commit 9df6903 into gabriel-vasile:master Oct 10, 2023
7 checks passed

ankon reviewed Oct 18, 2023

View reviewed changes

wagoodman mentioned this pull request Jul 10, 2024

Very High Memory Usage Using Syft anchore/stereoscope#233

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch csv and tsv method 'sv' from ReadAll() to stream each record with Read() #355

Switch csv and tsv method 'sv' from ReadAll() to stream each record with Read() #355

splashing-atom commented Dec 16, 2022 •

edited

Loading

gabriel-vasile left a comment

ankon Oct 18, 2023

splashing-atom Oct 18, 2023

gabriel-vasile Oct 18, 2023

ankon Oct 18, 2023

Switch csv and tsv method 'sv' from ReadAll() to stream each record with Read() #355

Switch csv and tsv method 'sv' from ReadAll() to stream each record with Read() #355

Conversation

splashing-atom commented Dec 16, 2022 • edited Loading

gabriel-vasile left a comment

Choose a reason for hiding this comment

ankon Oct 18, 2023

Choose a reason for hiding this comment

splashing-atom Oct 18, 2023

Choose a reason for hiding this comment

gabriel-vasile Oct 18, 2023

Choose a reason for hiding this comment

ankon Oct 18, 2023

Choose a reason for hiding this comment

splashing-atom commented Dec 16, 2022 •

edited

Loading