[YouTube] Support csv and zip subscription import (Google Takeout) #709

talanc · 2021-08-08T11:30:10Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

Extractor fix for issue TeamNewPipe/NewPipe#6757

This is the extractor PR for supporting the subscriptions.csv file from Google Takeout.
Also supports selecting a ZIP file (which will in turn find the subscriptions.csv file)
I introduced a new method on SubscriptionExtractor to determine what the stream is (e.g. json, zip, or csv):
List<SubscriptionItem> fromInputStream(@Nonnull final InputStream contentInputStream, String contentType)

Ready for review and happy to make any changes to get this accepted.

Stypox

Thank you for this needed usability improvement!

Stypox · 2021-08-09T16:13:02Z

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java


 import javax.annotation.Nonnull;

 import static org.schabi.newpipe.extractor.subscription.SubscriptionExtractor.ContentSource.INPUT_STREAM;

 /**
- * Extract subscriptions from a Google takout export (the user has to get the JSON out of the zip)
+ * Extract subscriptions from a Google takeout export (the user has to get the JSON out of the zip)


Suggested change

* Extract subscriptions from a Google takeout export (the user has to get the JSON out of the zip)

* Extract subscriptions from a Google takeout export

Stypox · 2021-08-09T16:14:40Z

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java

+    public static final String[] ENTRY_PATHS = {
+            "Takeout/YouTube and YouTube Music/subscriptions/subscriptions.csv",
+            "Takeout/YouTube y YouTube Music/suscripciones/suscripciones.csv" // There is a <NBSP> between YouTube and Music
+    };


You can't rely on a list of paths. It will be different for each language. I'd navigate the whole zip searching for a valid file instead.

A possible way to find this file is to look for files in the glob Takeout/YouTube*/*/*.csv and find one where the file starts with a match for this regex (this regex is a JS one, unsure of the Java equiv):

/^.*\n([0-9A-Za-z_-]{20,}),[^,]*?\/\1,.+\n/

^ intuition, this regex skips the first line, checks the second line to see if it starts with something that looks like a channel id, then looks for / followed by that same id again followed by a comma ,

Good idea, it should work out fine

Done--brief summary of changes:

CSV: after 5 lines, checks to make sure it's found subscription items (ensures the 2nd entry is a channel url)
if there's no items, then it exits out of the function, returning an empty list.

ZIP: iterates over every CSV file, gets the subscription items from that CSV file, if there's an error or no items in the list, it goes to the next CSV file in the zip.

talanc · 2021-08-10T11:50:11Z

Thanks for your feedback -- ready for review again.

Stypox

Thank you, this approach looks good to me! Have you tested it? Also, please put final in front of every variable you never change after declaration.

talanc · 2021-08-14T04:50:51Z

@Stypox
Testing:
Yes.
It has unit tests for the CSV and ZIP import.
I've manually tested (via an emulator) with two different subscription files.
And I'm sure there will be another round of testing when it gets merged and the UI layer is tested: TeamNewPipe/NewPipe#6882

Final:
I've added final to variables where appropriate.

Stypox

Almost good ;-)

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java

Stypox

Looks good code-wise :-)

litetex

Code looks good 😄
Test could be in theory parametrized, however because this is pain in JUnit 4 and creates a bunch of overhead, it's okay for me to go with a foreach-loop.

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java

talanc · 2021-08-17T05:11:19Z

I think this is good to be merged now.

The failed tests are unrelated to my changes (a Peertube test is failing).

Stypox · 2021-08-24T11:21:42Z

@TobiGr please merge this, I don't have permission as apparently snyk failed. And I don't have access to Snyk.

XiangRongLin · 2021-08-24T12:00:15Z

@Stypox Synk fails because of outdated jsoup version having a vulnerability. The version was already updated and merged. So if this is rebased, the check should succeed

csv: Improved error messages Exits early if it hasnt found any items in the first few lines zip: Now checks all CSV files instead of hard-coded paths final qualifiers for immutable locals and parameters Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>

Stypox · 2021-08-24T13:45:33Z

Rebased, thank you!

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java

Simon311 · 2021-09-15T22:24:14Z

Importing the CSV file doesn't work in the latest NewPipe available from F-Droid.
But perhaps more importantly, my Google export only returned 6 channels out of hundreds. I know there's nothing you guys can do about it, but has anyone had this same issue? Is there a way to report this to Google, since this is a clear GDPR violation?

opusforlife2 · 2021-09-17T15:11:47Z

Importing the CSV file doesn't work in the latest NewPipe available from F-Droid.

@Simon311 The version with this fix isn't released yet. You can install and test the Release Candidate by checking the pinned issue.

TobiGr added the youtube service, https://www.youtube.com/ label Aug 8, 2021

This was referenced Aug 9, 2021

Add support for CSV+ZIP subscriptions (Google Takeout) TeamNewPipe/NewPipe#6882

Merged

YouTube subscriptions cannot be imported (seem to be exported as .csv now) TeamNewPipe/NewPipe#6757

Closed

Stypox requested changes Aug 9, 2021

View reviewed changes

opusforlife2 requested a review from Stypox August 10, 2021 17:03

Stypox requested changes Aug 13, 2021

View reviewed changes

Stypox requested changes Aug 14, 2021

View reviewed changes

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java Show resolved Hide resolved

Stypox approved these changes Aug 14, 2021

View reviewed changes

litetex reviewed Aug 14, 2021

View reviewed changes

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java Outdated Show resolved Hide resolved

Stypox approved these changes Aug 24, 2021

View reviewed changes

Stypox force-pushed the dev branch 2 times, most recently from d5e74f3 to ca6d64c Compare August 24, 2021 13:42

Stypox force-pushed the dev branch from ca6d64c to 94a29fd Compare August 24, 2021 13:43

Stypox changed the title ~~YoutubeSubscriptionExtractor: Support for subscriptions.csv (and zip)~~ [YouTube] Support csv and zip subscription import (Google Takeout) Aug 24, 2021

Stypox merged commit db6c729 into TeamNewPipe:dev Aug 24, 2021

Sanjay15693 reviewed Aug 28, 2021

View reviewed changes

...a/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java Show resolved Hide resolved

litetex mentioned this pull request Aug 28, 2021

[SonarQube] Follow up to Sonar analyzing old branches TeamNewPipe/NewPipe#5926

Closed

opusforlife2 added the bug Issue is related to a bug label Aug 29, 2021

Stypox mentioned this pull request Aug 30, 2021

[YouTube] Small improvements to subscription import #721

Merged

2 tasks

This was referenced Sep 5, 2021

Release 0.21.10 TeamNewPipe/NewPipe#7063

Merged

Release 0.21.10 TeamNewPipe/NewPipe#7066

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YouTube] Support csv and zip subscription import (Google Takeout) #709

[YouTube] Support csv and zip subscription import (Google Takeout) #709

talanc commented Aug 8, 2021 •

edited by AudricV

Loading

Stypox left a comment

Stypox Aug 9, 2021

Stypox Aug 9, 2021

tbjgolden Aug 9, 2021 •

edited

Loading

tbjgolden Aug 9, 2021

Stypox Aug 9, 2021

talanc Aug 10, 2021

talanc commented Aug 10, 2021

Stypox left a comment

talanc commented Aug 14, 2021

Stypox left a comment

Stypox left a comment

litetex left a comment

talanc commented Aug 17, 2021

Stypox commented Aug 24, 2021

XiangRongLin commented Aug 24, 2021

Stypox commented Aug 24, 2021

Simon311 commented Sep 15, 2021

opusforlife2 commented Sep 17, 2021

	* Extract subscriptions from a Google takeout export (the user has to get the JSON out of the zip)
	* Extract subscriptions from a Google takeout export

[YouTube] Support csv and zip subscription import (Google Takeout) #709

[YouTube] Support csv and zip subscription import (Google Takeout) #709

Conversation

talanc commented Aug 8, 2021 • edited by AudricV Loading

Stypox left a comment

Choose a reason for hiding this comment

Stypox Aug 9, 2021

Choose a reason for hiding this comment

Stypox Aug 9, 2021

Choose a reason for hiding this comment

tbjgolden Aug 9, 2021 • edited Loading

Choose a reason for hiding this comment

tbjgolden Aug 9, 2021

Choose a reason for hiding this comment

Stypox Aug 9, 2021

Choose a reason for hiding this comment

talanc Aug 10, 2021

Choose a reason for hiding this comment

talanc commented Aug 10, 2021

Stypox left a comment

Choose a reason for hiding this comment

talanc commented Aug 14, 2021

Stypox left a comment

Choose a reason for hiding this comment

Stypox left a comment

Choose a reason for hiding this comment

litetex left a comment

Choose a reason for hiding this comment

talanc commented Aug 17, 2021

Stypox commented Aug 24, 2021

XiangRongLin commented Aug 24, 2021

Stypox commented Aug 24, 2021

Simon311 commented Sep 15, 2021

opusforlife2 commented Sep 17, 2021

talanc commented Aug 8, 2021 •

edited by AudricV

Loading

tbjgolden Aug 9, 2021 •

edited

Loading