In Android not able to read Arabic translation JSON file using JSONStream

[midhunm-c]

Arabic translation JSON file is downloaded to document directory and it is reading and parsing using RNFetchBlob.JSONStream.

Issue:
In android when we tried to read Arabic translation file using JSONStream the callbacks done or fail in not getting called. And not throwing any exceptions.
In iOS JSONStream is calling done block once the same Arabic json file read complete.

Please note: I have checked this with joltup/rn-fetch-blob
but I also tried to install the latest version of this library, and the result is same.

[RonRadtke]

Could you please provide a sample json that is problematic?
Don't need you full translation file, just a few lines that will cause the problem.

[midhunm-c]

@RonRadtke
this is a portion of the json file which makes problem.

{
    "app/screens/onboarding/onboarding-button-texts/next": "Next",
    "app/screens/contact/send-bulk-email": "إرسال بريد إلكتروني دعائي"
  }

This is added to a json file in android document directory and tried to read using JSONStream. done or fail block not getting called.

If I removed this Arabic text from the json the JSONStream is calling the done block.

[midhunm-c]

@RonRadtke
While reading the json file with Arabic text from application document directory as JSONStream this following exception was throwing

I have fixed the issue in this PR, #53
Please review this PR.

[RonRadtke]

Thank you for your PR.
My problem is, it is not actually fixing the problem. It's a simple bad aid working for now. But in general there seems to be something off with either the data itself or what happens to the data before it being encoded.
So either there are characters in which are not covered by utf8 or somewhere goes something wrong.
So I'll have a closer look at the problem as soon as I have time for.

Also I don't like to have different handling for the different encodings. If we go for it, it would be better to ignore the encoding errors for all different types of encoding to have a similar behavior all over the code.

[devinm-hrbl]

Hello! I completely understand your comments, in fact I would say the same thing. That being said, is there any chance we can have a beta version with this quick fix until you can fix the actual problem? I'd really rather not fork this repo (yet again) and re-host just to have a temporary fix, and I don't have the time to invest in exploring other avenues for a fix.

[RonRadtke]

I pushed your fix on a feature branch. Due to the merge conflict I just copied it quickly.
It's the branch features/arabic_json_fix

So you should be able to install it by running npm install RonRadtke/react-native-blob-util#feature/arabic_json_fix

[devinm-hrbl]

Wonderful! Thank you so much, this is going to save us a lot of trouble!

[RonRadtke]

I'm relatively sure I know the cause of the problem now.
Basically there are only two options:

• Your file contains non-utf8 chars
• Something in the Lib is broken

I assume your file is fine, otherwise you would most likely have problems in iOS too.
Currently we are splitting the input in chunks of 4096 Bytes, but this doesn't respect the fact that UTF8 uses 1,2 or 3 bytes to encode a single character. So most likely we are cutting a valid character in half and thus you're seeing the error.
I hope I'll have the time to reimplement that part in the next week.

I would like to ask you to test the changes with your file to verify it's working fine - I hope you could help there @devinm-hrbl / @midhunm-c ?
I would come back to you then, as soon as it's implemented.

[midhunm-c]

Thank you very much @RonRadtke !!
Yeah sure, we can definitely test once you implemented the fix.

[RonRadtke]

@midhunm-c would it be a problem for you if I bump the minSDK to 19? I think I have a working solution for > APi 19.
That would on the other hand mean, you have to stay on a dev branch till I make the next major release, which I plan for September. (Hopefully I get the android scoped storage working till then)
Don't need to bump the API for this :) So I think you will get a branch to test tomorrow.

[RonRadtke]

@midhunm-c branch is ready for you.
You can install it via npm install RonRadtke/react-native-blob-util#feature/utf8_fix

[midhunm-c]

Thank you @RonRadtke !!
I have checked from the branch feature/utf8_fix and the language is loading without any exception. 👍

[RonRadtke]

Glad to hear :)