Issue with SponserBlock / What is blocked.

[Lucas319315]

I check my sources menu and when I go into it shows sponser block enabled but the "what blocked?" is empty.
I go and re-add the categories again but when I eventually go back to it that section is blank again.

[meeb]

Thanks for the issue, I'll look into it.

[Lucas319315]

Thanks for the issue, I'll look into it.

Sounds good! Good luck trying to resolve this.

[Snijder]

This is an issue for me too, sponsorblock also does not seem to do anything.
I have this line in the logs:
ERROR: Object of type CommaSepChoiceField is not JSON serializable
which might be related.

[Snijder]

I managed to hack it back into working order by manually editing the youtube.py and copy pasting from a code snippet in a reddit post, I don't know enough about python to submit a PR unfortunately.

'postprocessors': [
    {'key': 'SponsorBlock'},
    {'key': 'ModifyChapters', 'remove_sponsor_segments': ['sponsor', 'intro', 'outro', 'selfpromo', 'preview', 'filler', 'interaction']}
 ]

Removing https://github.com/meeb/tubesync/blob/main/tubesync/sync/youtube.py#L117 also got rid of the error I mentioned in my previous post. (I appended ModifyChapters below it.)

(Not completely sure if this would also solve the settings disappearing)

[meeb]

Your patch is pretty much exactly what the default sponsor block management code should do, selecting the tick boxes should just generate the list like ['sponsor', 'intro', 'outro', 'selfpromo', 'preview', 'filler', 'interaction'] depending on your selection automatically. A PR containing this would override the web UI.

There's a bug somewhere in the handling and creation of the the list that only affects some deployments so it's tricky to catch.

[mylogon341]

fyi im getting this issue also. I can give details of my setup /the yt channel if you think it'd help with your debugging/ resolution of this.

thanks.

[Lucas319315]

I managed to hack it back into working order by manually editing the youtube.py and copy pasting from a code snippet in a reddit post, I don't know enough about python to submit a PR unfortunately.

'postprocessors': [
    {'key': 'SponsorBlock'},
    {'key': 'ModifyChapters', 'remove_sponsor_segments': ['sponsor', 'intro', 'outro', 'selfpromo', 'preview', 'filler', 'interaction']}
 ]

Removing https://github.com/meeb/tubesync/blob/main/tubesync/sync/youtube.py#L117 also got rid of the error I mentioned in my previous post. (I appended ModifyChapters below it.)

(Not completely sure if this would also solve the settings disappearing)

I am using this in unraid do you know if there is an option to make that change to the system? I have been trying but not successful.
I found the yourtube.py but cannot find the place to change it.

[meeb]

@Lucas319315 that edit would be made in the settings.py file, not youtube.py. Add postprocessors as a key to YOUTUBE_DEFAULTS here:

https://github.com/meeb/tubesync/blob/main/tubesync/tubesync/settings.py#L157

[Lucas319315]

@Lucas319315 that edit would be made in the settings.py file, not youtube.py. Add postprocessors as a key to YOUTUBE_DEFAULTS here:

https://github.com/meeb/tubesync/blob/main/tubesync/tubesync/settings.py#L157

Thanks for the assist!

[mylogon341]

Is it possible we're able to have an incremental release with this fix in please so we're not all having to manually edit files? That would be great.

[meeb]

There isn't a fix in the repo for this yet @mylogon341 - I've not been able to replicate the issue to debug it. If anyone has a bare SQLite database from a tubesync install which exhibits this behaviour feel free to add it to the issue.

[mylogon341]

Is it possible we're able to have an incremental release with this fix in please so we're not all having to manually edit files? That would be great.

Sorry - ignore. I got myself mixed up on bugs I'm following and just added a comment in my 'rona recovery. In my head this was for a ticket for sponsor block not working properly. Maybe this is related and maybe not 🤷‍♂️ I'll maybe come back to this and do some more testing when my brain-fog clears.

[duloo0]

I'm pretty confused about how to edit that settings.py file. I'd really like to get SponsorBlock to work.

[Lucas319315]

I'm pretty confused about how to edit that settings.py file. I'd really like to get SponsorBlock to work.

What are you using to host docker?
I was using unraid and ended up using console and then was able to run commands on it.

Here is what i had to do to get it edited.
apt update
apt upgrade
apt-get install nano
cd tubesync
nano settings.py

[meeb]

@duloo0 you should just be able to tick the sponsorblock categories from the edit source form, unless you are impacted with this bug.

[duloo0]

I'm impacted by the same bug unfortunately.

[meeb]

How big is your SQLite database? If you can upload it so I can replicate the issue I can fix why the option isn't working.

[thavelick]

Hi, I'm having this issue, or at least a similar one where sponsorblock isn't working. I have Sponsorblock enabled with:

What blocked?: | Sponsor: Intermission/Intro Animation: Endcards/Credits: Unpaid/Self Promotion: Preview/Recap: Filler Tangent: Interaction Reminder: Non-Music Section:

And I get:

2023-12-11 23:23:46,514 [tubesync/INFO] Scheduling task to download thumbnail for: But What IS A Lens Flare? from: https://i.ytimg.com/vi/z_WRaTsGbVE/maxresdefault.jpg
172.17.0.1 - thavelick [11/Dec/2023:16:23:46 -0700] "POST /media-redownload/6394ef85-9cd2-4885-afcc-c41df310c7d1 HTTP/1.0" 302 0 "https://tubesync.tristanhavelick.com/media-redownload/6394ef85-9cd2-4885-afcc-c41df310c7d1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
172.17.0.1 - thavelick [11/Dec/2023:16:23:46 -0700] "GET /media/6394ef85-9cd2-4885-afcc-c41df310c7d1?message=redownloading HTTP/1.0" 200 20270 "https://tubesync.tristanhavelick.com/media-redownload/6394ef85-9cd2-4885-afcc-c41df310c7d1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
172.17.0.1 - thavelick [11/Dec/2023:16:23:46 -0700] "GET /static/images/nothumb.png HTTP/1.0" 304 0 "https://tubesync.tristanhavelick.com/media/6394ef85-9cd2-4885-afcc-c41df310c7d1?message=redownloading" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
2023-12-11 23:23:50,827 [tubesync/INFO] Resizing 1280x720 thumbnail to 430x240: https://i.ytimg.com/vi/z_WRaTsGbVE/maxresdefault.jpg
2023-12-11 23:23:50,869 [tubesync/INFO] Saved thumbnail for: z_WRaTsGbVE from: https://i.ytimg.com/vi/z_WRaTsGbVE/maxresdefault.jpg
2023-12-11 23:23:55,806 [tubesync/INFO] Downloading media: z_WRaTsGbVE (UUID: 6394ef85-9cd2-4885-afcc-c41df310c7d1) to: "/downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.mkv"
2023-12-11 23:23:57,816 [tubesync/INFO] [youtube-dl] downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f248.webm - 55.2% of 35.19MiB at 30.72MiB/s, 00:00 remaining
2023-12-11 23:23:58,352 [tubesync/INFO] [youtube-dl] downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f248.webm - 100.0% of 35.19MiB at 34.36MiB/s, 00:00 remaining
2023-12-11 23:23:58,438 [tubesync/INFO] [youtube-dl] finished downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f248.webm - 35.19MiB in 00:00:01
2023-12-11 23:23:58,913 [tubesync/INFO] [youtube-dl] finished downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f251.webm - 4.90MiB in 00:00:00
ERROR: Object of type CommaSepChoiceField is not JSON serializable
2023-12-11 23:23:59,136 [tubesync/INFO] Successfully downloaded media: z_WRaTsGbVE (UUID: 6394ef85-9cd2-4885-afcc-c41df310c7d1) to: "/downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.mkv"
2023-12-11 23:23:59,172 [tubesync/INFO] Copying media thumbnail from: /config/media/thumbs/63/6394ef85-9cd2-4885-afcc-c41df310c7d1.jpg to: /downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.jpg
2023-12-11 23:23:59,172 [tubesync/INFO] Writing media NFO file to: to: /downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.nfo

in my logs when I try to redownload a video. The key thing seems to be the ERROR: Object of type CommaSepChoiceField is not JSON serializable error.

Also happens with just the "sponsor" option checked along with "enable sponsorblock". As far as I can tell, digging around , we just need to change:

 sbopt = {
        'key': 'SponsorBlock',
        'categories': [sponsor_categories]
    }

to

 sbopt = {
        'key': 'SponsorBlock',
        'categories': sponsor_categories.get_my_choices()
    }

But that's just my impression from looking at the code and is entirely untested. I'll try to play with it and possible post a PR sometime over the next week or so.

My DB is like a gig, so it probably isn't feasible to just upload. If I can't confirm or figure out a fix, I'll spin up an instance with a pared down version of the problem and post the db

[thavelick]

@meeb Okay, I was able to reproduce this on main (currently 6853c1f) with a small database which I've attached.

tubesync-sponsorblock-broken-db.zip

To reproduce the ERROR: Object of type CommaSepChoiceField is not JSON serializable problem:

1. spin up a container built against the above commit with the attached db
2. Open the website
3. Click Media
4. Click the single video (a ClimateTown video about climate change)
5. Click "Redownload Media"
6. Make sure you're tailing the logs per the README
7. Click the button to confirm
8. You should see something like this in the logs:

tubesync-1  | 2023-12-12 01:50:30,013 [tubesync/INFO] [youtube-dl] finished downloading: 2023-10-30_climatetown_chicago-doesnt-own-its-own-streets-climate-town_fDx6no-7HZE_1080p-vp09-opus.f248.webm - 342.07MiB in 00:01:02
tubesync-1  | 2023-12-12 01:50:37,476 [tubesync/INFO] [youtube-dl] finished downloading: 2023-10-30_climatetown_chicago-doesnt-own-its-own-streets-climate-town_fDx6no-7HZE_1080p-vp09-opus.f251.webm - 29.02MiB in 00:00:07
tubesync-1  | ERROR: Object of type CommaSepChoiceField is not JSON serializable
tubesync-1  | 2023-12-12 01:50:38,657 [tubesync/INFO] Successfully downloaded media: fDx6no-7HZE (UUID: 0b7affc5-dd94-4645-8006-e6adebd06e64) to: "/downloads/video/climatetown/2023-10-30_climatetown_chicago-doesnt-own-its-own-streets-climate-town_fDx6no-7HZE_1080p-vp09-opus.mkv"

9. Watch the video, you'll notice that the sponsor is not blocked

I'll continue looking at this, but I'm less confident that my proposed solution will work.

[meeb]

Thanks @thavelick - I've pushed a very minor tweak that may help with this. It'll be in the next release and :latest shortly.

[thavelick]

@meeb I built a container with the above commit and now I'm getting:

tubesync-1  | WARNING: Unable to communicate with SponsorBlock API: HTTP Error 400: Bad Request. Retrying (1/3)...
tubesync-1  | WARNING: Unable to communicate with SponsorBlock API: HTTP Error 400: Bad Request. Retrying (2/3)...
tubesync-1  | WARNING: Unable to communicate with SponsorBlock API: HTTP Error 400: Bad Request. Retrying (3/3)..

Seems like we're getting closer! But sponsor segments are still in there

[meeb]

Ah, well that looks like TubeSync's end is working, I just need to find why the upstream API is blocking the requests then (or find an alternative endpoint).

Edit: or perhaps the category names are wrong and that's what the API is complaining about? Mostly a note for future me to look into.

[meeb]

Yep it was me being an idiot and wrapping a list in another list as a typo. This should be fixed once the above commit builds into :latest.

[irishj]

Can you please commit the builds into latest so I can pull the latest image, ads in my videos are killing me. Thank you !

[meeb]

@irishj this should be in :latest, try pulling an updated image. If it's still not working check your sources sponsorblock categories.

[irishj]

Thanks @meeb - I'll pull the latest and check it out. Thanks again for this great software.

[uduncanu]

I'm having the same issue here, but I don't think those commits are helping. The problem seems to be that it clears the sponsorblock settings whenever it fetches the metadata.

Versions tested: 0.13.3 and master
Steps to reproduce:

1. Start up tubesync in a docker container (completely fresh, it can create a new db file): docker run -p 8080:4848 ghcr.io/meeb/tubesync:latest
2. Go to the web interface (on port 8080 using the above command) and go to add a source
3. Enter source information, preferably one that doesn't exist as that'll keep trying to get the metadata. I've been using https://www.youtube.com/c/testyoutubechannelidthathopefullywontexist532454
4. Make sure you set sponsorblock settings, and then hit save.
5. Wait a few seconds for it to try fetching metadata
6. Go back to the source and the sponsorblock settings are gone.
7. If you re-add them and save, wait a few mins for the next metadata fetch to try and it'll remove them again.

I tried digging into it a bit more and it looked like it was the bit in the index_source_task function in tasks.py that saves the source after retrieving it and setting has_failed to false. That then looked like it was a typing issue, as Django seems to expect CommaSepChoiceField's from_db_value to return a list but it actually returns an object. Changing it to a list fixes the issue with settings changing every time it fetches metadata, but it breaks the user interface and probably loads of other things, so I gave up at that point!

[meeb]

Thanks for investigating! I'll take a look at your reproduction steps.

[irishj]

I'm seeing a similar issue with existing sources which have already been added.

I can set the sponsor block options via the GUI or via a MySQL query (I'm using MySQL as the back-end DB) and the chosen block options are saved, but when the scheduled tasks are run, the sponsor block options are removed for all sources.

The sponsor block enabled option is still enabled, but the sponsor block options for each source have all been removed.

[meeb]

Yeah there's clearly a bug in the handling of the sponsorblock categories. I'll look into it when I get time.

[AlexStormwood]

Would love to see this fixed, it's the last thing in my archiving server to sort out.

I'm not brave enough to edit the containers I'm running locally, but I'm happy to test out any pushed container updates that get made to address this bug.

[meeb]

I've had a couple of looks at this and it's only not fixed because I've never seen it happen myself, no-one has been able to supply a trace of an error or a copy of an SQLite database that exhibits this behaviour so it's obviously quite hard to locate the bug. The reproduction steps haven't reproduced it for me in testing.

If you can supply your SQLite database somewhere, if you're OK with sharing that, I could probably locate the issue.

[AlexStormwood]

tubesync-sponsorblock-repro.zip

I just made this compose file, ran it, added a non-existent channel as per uduncanu's steps, ticked the box for "(ALL OPTIONS)", saved the source with an index schedule of 7 days, and waited. The page we get redirected to shows that sponsorblock settings were saved, but refreshing it (roughly when indexing has started) shows that the settings disappear.

I'm using Docker Desktop on Windows 11, hence the funky Docker Compose version used in the file too.

It doesn't seem to matter on my end, but my other containers are using a PostgreSQL database instead of SQLite. Sponsorblock issue happens either way.

[meeb]

Excellent, thanks. I'll see if I get some time later this week to spin up a test instance with your database.

[irishj]

Hello Again. Just wondering if you had an opportunity to review this issue / find a solution to sponsor block not working ?

[Lucas319315]

This is working for me.

[AlexStormwood]

Nothing has updated in the repo with regards to this issue, why would you close this?

If your manual edits work, @Lucas319315, then that's great, but they need to be added to this repo.

[Lucas319315]

Yeah the manual edits worked for me. I am new to github and don't know the best process on leaving this opened or closed.

[irishj]

@Lucas319315 Can you please share a copy of your "settings.py" file where you have made the changes so I can try it ?

I made the edits myself manually in the "settings.py" file and enabled the sponsor block categories for one channel, ran the "reset tasks" script and after the channel was indexed, the sponsor block categories were reset / disabled again. Maybe I made the changes incorrectly. Thanks.

YOUTUBE_DEFAULTS = {
    'no_color': True,       # Do not use colours in output
    'age_limit': 99,        # 'Age in years' to spoof
    'ignoreerrors': True,   # Skip on errors (such as unavailable videos in playlists)
    'cachedir': False,      # Disable on-disk caching
    'addmetadata': True,    # Embed metadata during postprocessing where available
    'postprocessors': [
    {'key': 'SponsorBlock'},
    {'key': 'ModifyChapters', 'remove_sponsor_segments': ['sponsor', 'intro', 'outro', 'selfpromo', 'preview', 'filler', 'interaction']}
 ]
}

[Lucas319315]

@irishj
I ended up not being able to get this working with the settings.py file and I went into the sync folder and edited the youtube.py file to be the following.
'postprocessors': [
{'key': 'SponsorBlock'},
{'key': 'ModifyChapters', 'remove_sponsor_segments': ['sponsor', 'intro', 'selfpromo', 'interaction']}
],

I do know meeb mentioned the following below but that only worked for about a day and then it would not work anymore where the edit to youtube.py I did has not broken since but it does need be be reapplyed after any updates to that docker and it does apply those settings across all sources.
"@Lucas319315 that edit would be made in the settings.py file, not youtube.py. Add postprocessors as a key to YOUTUBE_DEFAULTS here:
https://github.com/meeb/tubesync/blob/main/tubesync/tubesync/settings.py#L157"

The change is on line 147

YoutubeFileEdited.txt

[irishj]

Thanks for sharing your file. I tried it within my own installation and within a new installation and after a channel is synced, the sponsor block options are being disabled, not sure why that might be. Will probably need to wait for @meeb to sort out the issue so I can finally get rid of all those annoying ads.