YouTube Music -> Last.FM scrobbling same songs multiple times #156
Comments
|
I had this issue too, but it was difficult for me to reliably replicate, seemingly happening at random, so without any reliable data I didn't bother opening an issue. I've just been watching the entire repo. |
Haha you know what? You're the reason I found this repo! Googled for a YouTube Music scrobbler and found your blog. 😁 Hopefully the dev have some insight to what might be going on. So I'm guessing then (reading your blog) that the dev branch didn't solve this issue either? |
|
At the time of writing the dev branch didn't make a difference. |
|
Thanks for the writeup. There haven't been any changes to the YT Music code in MS because I don't personally use YT Music so if something breaks I depend on any users who use it to report an issue. I can't currently reproduce your issue but I'm fairly confident it's due to how YTM "organizes" your listens in its recent history -- it does not surface any actual "listened at" datetime info. Instead:
EX:
MS tries to be clever about how it determines if YTM has returned a new track. When it first starts up (polling) it stores all tracks returned from YTM history in a flat list. Then it polls that history API regularly and if it finds a track at the beginning of the list (assumed most recent) that does not match the last-seen first track it determines that is a new track and scrobbles it. Likely what is happening with the duplicate scrobbles is that YTM is either not returning tracks in reverse chronological order or it is not returning the playlists in reverse chronological order. Or maybe something else entirely...its pretty opaque. MS also can't do duplicate checking, which it does for all other music sources, because there are no datetimes so it just has to assume the list is in the correct order. 🤷 Having said all that, I've added some code to help surface information about those playlists to a feature branch in MS. Please use the You'll find on the dashboard the YTM card has a new link, See Recent from Source API. Navigating to this link will open a page that lists the tracks returned from YTM without modification. This is the exact order YTM returns them to MS. You'll also see the "playlist bucket" YTM puts these tracks in at the end of each track.
If everything is working normally this list should be essentially the same order as Tracks Discovered page for your YTM source. If you see duplicate scrobbles occurring please:
This will help me isolate if its a list order issue or if I need to dig further into this. thanks |
|
@FoxxMD thank you for the explanation and insights into why scrobbling from YouTube Music is such a pain, to be honest if it wasn't for multi-scrobbler I would stick with Spotify. Your work is much appreciated! To give a brief update since it's been almost a day since your reply, these are the steps I've taken to test this out:
So far everything has been scrobbling perfectly correct! The only change from when I ran the release version of 0.7.1 is:
I'll leave this issue open and get back to you once I've tested things out some more, hopefully @SomeAspy have time to try it out as well. Thanks again for all your work! |
|
Off topic but I read your blog post @SomeAspy (thanks for the kind words!) and wanted to let you know you do not need CF Access.
Last.fm uses a slightly modified authorization code flow but the relevant part is that its redirect behavior is the same as the normal oauth2.0 spec -- the redirect URI is is where the authorization service redirects the users browser after its side of the authorization flow is complete. This means the URI used only needs to be accessible from the users browser, not that is needs to be publicly accessible:
LFM's servers never initiate a connection with your MS instance so your instance does not need to be public in order to complete the auth flow. This is usually true for any oauth2.0 compliant authorization flow and auth code flows in general. |
|
I love to see people reading it! I was surprised to see the author found this from my silly blog. I have updated the post to add what you just said. I also can't help but comment on your swiftness of responding to any and all issues on this repository, so kudos! |
|
@roos-robert @SomeAspy have either of ya'll had a chance to test this? Any results? |
|
I'm actually gonna begin tomorrow, the circumstances where I scrobble a lot from mobile have started today. |
|
Something I've almost immediately noticed within the first few hours of using is my YouTube music cookie seems to get invalidated more than it previously was. |
|
@SomeAspy can you try enabling the option mentioned here? This should be included in the latest commits on the |
|
Ah yes I missed that. I haven't actually tested this exactly yet but it seems to drop on restart. I can test it when I get home. |
|
Another smaller detail I've noticed: if you play a song several times over only one play will be detected. I assume this is because of the way you make sure not to scrobble multiple times and am not sure if this is avoidable. |
|
As long as there is a different track played between the two it should be detecting them just fine (assuming MS was running and monitoring the entire time). I'll try to use YTM today and see if I can get api dumps to use for testing. |
|
There was no track between, and I just noticed that the YouTube history itself only shows once. In this case it was house of cards for about 7 replays. |
|
Yeah, unfortunately MS can't do any thing about that since YTM doesn't report any kind of player status or listening statistics. All it reports is the history, in order (hopefully), without any timestamps. It's the least data-rich platform MS supports. |
|
Yet again I feel like I shot myself in the foot using YouTube Music. Since this is the best we can get regarding mobile scrobbling on iOS, I've been trying to think of the best way to keep using my desktop scrobbler without interfering with multi-scrobbler, which will be running in the background. In short, I want to use MS for mobile, and the desktop client for desktop. I have a feeling these will naturally interfere with each other. I haven't really tried it yet though. In better news, scrobbling via mobile seems to be going great, to the best it can given the limitations of YTM. I will list the problems I have with it just for record, but I cannot think of a way to actually fix these.
Cross examining YTM history with last.fm scrobbles seems to be more reliable (???) but there are still some oddities and duplications.
Duplicates are often scrobbled in bunches, for what it is worth, with exact same times (which would not be possible since each song is at least 3 minutes) Strangely, these duplications and order errors don't seem to appear in the MS debug |
|
Thank you for the detailed writeup and track matching, this is very helpful!
Yes, unfortunately YTM records a song to the history playlist if it even loads in the player. It has no threshold for "this has been played for X seconds" before adding it.
This is a limitation of how MS tries to detect plays in "real-time" in the way I described in the earlier comment. It should be polling fast enough that when the history updates there is only one new track -- IE polling every 60 seconds, tracks are played for longer than 60 seconds so only one shows up between each poll -- BUT the code does not enforce this assumption: if multiple tracks show up between two polls it'll scrobble all the newly seen tracks with the same timestamp. It's erring on the side of caution in the event polling temporarily fails or YTM doesn't update history for an extended period.
This is good to see, I think. It at least tells us that the data scraped by yt-music-api library is the same as what you actually see in YTM, at least in a spontaneous context. During this window do you know if MS restarted polling or had any errors? Even temporary ones like it failed to contact api but restarted successfully before polling was stopped -- or had a reauthentication event for YTM? Any logs at all might be insightful. Based on what you've presented so far though it seems like either
I'll see about improving the history checking code and storing previous state so i can detect if/when this consistency occurs so it can drop the "stale" list or try to recover somehow. |
|
I should have included logs, you are right. |
* Use superdiff to diff PlayObject lists and detect changes as well as append/prepend scenarios * Replace YTM recently played logic with list diffing, only accept prepend-validated lists * On non-prepend scenarios replace existing recently played and log human readable diff
|
@SomeAspy @roos-robert I've pushed changes to the Please try these changes and let me know if you get any logs about non-prepend events. |
* Use superdiff to diff PlayObject lists and detect changes as well as append/prepend scenarios * Replace YTM recently played logic with list diffing, only accept prepend-validated lists * On non-prepend scenarios replace existing recently played and log human readable diff
|
I should link this in here as well. #158 (comment) Perhaps it might make more sense to merge these issues into one issue, as the difference between the 2 problems is starting to blur. |
…lays #156 Rather than generating our own just use the known "good" history we just compared against to see if this improves consistency.
|
Both issues are related to YTM but I don't think they are casual in nature, at least not yet. I'd prefer to keep them separate for now. RE your comment in 158: That's pretty eye-popping! There may be an issue with MS still but there are definitely inconsistencies in there that can't be accounted for by the way MS handles the history. I've added a few more changes. Please try testing with |
|
Alright, I just started it up (with the debug env variable) and your new PR. I guess I'll let you know how it goes sometime soon (most likely tomorrow) |
|
hey @roos-robert a bit off topic, but I thought I'd mention I commissioned a Youtube Music app tweak for iOS, It seems to work great! https://github.com/marioparaschiv/lastfm-yt-music |
…lays #156 Rather than generating our own just use the known "good" history we just compared against to see if this improves consistency.
|
The change to validate the sort order against the previous list and only scrobble new plays if they are prepended to returned history works well in not creating any wrong scrobbles, however it causes songs that should be scrobbled to not be scrobbled quite often for me. For example: I ran into an issue when I played a song briefly but then skipped it. MS scrobbled the song but when I skipped it, YouTube removed it from the history (replacing it with the next song I played) causing MS to reset its history and not scrobble the song I played directly after. (The skipped song was scrobbled but the one I actually listened to wasn't) Another reset I don't quite understand why it happened was this: Another issue I ran into is that it won't scrobble songs I have already played somewhat recently. (not sure if it's related) |
|
Thanks for reporting your issue. In order for MS to accurately determine if a song has been scrobbled it needs a source of truth. Whether that is information about the current state of the player (subsonic, spotify) like playing/"position within the track" or an actual list of played songs (lastfm, listenbrainz) the information MS receives needs to be consistent and predictable. Unfortunately there is no documentation for how YTM history behavior works and the observed behavior for information for that "source of truth" (list of played tracks) is problematic:
The lack of consistency is what was causing the duplicate scrobbles for this issue in the first place. In order to partially compensate for this MS is now extremely "strict" about watching the order of tracks in YTM's history. Changes to the list that are valid for recording a scrobble should only fall under these two conditions:
If either of those two conditions is violated it "resets" its source of truth to the "new" list and waits for changes that match the above conditions. The issues you are running in to, MS not scrobbling, are due to it resetting the source of truth after those conditions are violated. You can see that from the If you want better scrobbling your best bet would be to use a scrobbling app (on your phone, web browser, computer) that scrobble to last.fm and then relay that with MS to LZ or Maloja or w/e. |
|
All of the changes to prevent duplicate scrobbles and the logging these things are now included in v0.8.3 along with FAQ entries and doc mentions about the above issues with missing scrobbles. The original issue is "resolved" and there's not much more that can be done about the inconsistent upstream data from YTM so I'm closing this for now. If google decides to release an official API for YTM in the future or another way of access this data is discovered I'll be happy to revisit this. |
Describe the bug
Hi and thank you for your work on this great scrobbler!
I'm running into an issue using YouTube Music and scrobbling to Last.FM. It seems to be fetching the played history over and over, scrobbling the same songs to Last.FM, for example:
I played this song once, and it keeps being scrobbled over and over (same with other songs).
As an example:
Songs found here (just turned the listening history on) should be maybe 20% of that amount.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
To only scrobble a song once, not fetch the play history over and over and scrobble songs multiple times.
Logs
A piece of the logs, all of the songs saying they are being scrobbled to Last.FM here was listened to an hour ago.
Versions (please complete the following information):
Provide version information for any related sources/clients.
Additional context
Running latest release (cloned master of this repo)
The text was updated successfully, but these errors were encountered: