Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relax parsing rules for CSV translation files #82491

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nolandc
Copy link
Contributor

@nolandc nolandc commented Sep 28, 2023

Fixes #82002

This PR relaxes the parsing rules in the CSV translation importer, so that CSVs that are not properly formatted don't fail the import. This allows users to change the delimiter options (or set the mode to "keep").

More specifically, this would no longer throw an error for:

  • CSVs that don't have more than 1 line (this could happen if a non-comma delimiter is set)
  • CSVs that have keys without the expected number of translations (this could happen if a non-comma delimiter is set but a comma is used elsewhere in the file)

Testing this is a bit tedious at the moment due a crash in import_dock.cpp when reimporting files that don't return a resource from ResourceLoader::load. Fix for that is up here: #82490

@nolandc nolandc requested a review from a team as a code owner September 28, 2023 17:20
@Calinou Calinou added this to the 4.2 milestone Sep 28, 2023
@Scoppio
Copy link

Scoppio commented Sep 29, 2023

There are no chances of adding unit tests for this? Because this can totally be a thing to be unit tested without too much trouble.

@nolandc
Copy link
Contributor Author

nolandc commented Sep 29, 2023

Good idea, I'll add some tests!

@nolandc nolandc force-pushed the fix_csv_import_for_non_comma_separated_files branch from adfadde to 85a6d6d Compare September 29, 2023 23:11
@nolandc nolandc requested review from a team as code owners September 29, 2023 23:11
@nolandc nolandc force-pushed the fix_csv_import_for_non_comma_separated_files branch from 85a6d6d to 3358c6c Compare September 29, 2023 23:23
@nolandc
Copy link
Contributor Author

nolandc commented Sep 29, 2023

Added tests for:

  • importing a file that has a malformed header row (imports with no data)
  • importing a file that has a malformed translation row (skips importing the line in question)

I also moved the existing csv import test to this new file. I don't think it belongs in the test_translation.h file.

There didn't seem to be much precedent for these tests, as none of the other importers have tests. It seemed like it might be a good idea to have a folder for things that test pure editor code, as opposed to being shoved into existing files with a bunch of TOOLS_ENABLED conditionals. Then the test files can just be included with a single conditional. In any case, let me know if there's a better way to organize this and I'll update as needed.

@YuriSizov YuriSizov modified the milestones: 4.2, 4.3 Nov 10, 2023
Copy link
Member

@Calinou Calinou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasoning behind this makes sense to me.

However, recent commits have added uses of ERR_CONTINUE_MSG instead of using WARN_PRINT. When you have time, could you rebase this PR on master and change your added code to use ERR_CONTINUE_MSG instead? Note that 4.3's release is imminent, so you don't have to rush for this as this PR is now targeting 4.4 (although it could be cherry-picked to 4.3.x later).

@Calinou Calinou modified the milestones: 4.3, 4.4 Jul 26, 2024
@Calinou Calinou added the cherrypick:4.3 Considered for cherry-picking into a future 4.3.x release label Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug cherrypick:4.3 Considered for cherry-picking into a future 4.3.x release topic:import
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CSV importer works only with comma delimiter
4 participants