adding similarTracks and findPath new api for AudioMuse-AI sonic simiarity

[NeptuneHub]

This pr is to add similarTracks and findPath new API for support AudioMuse-AI Sonic Similarity as discussed here #172

Please let me know if I need to add or fix something.

Thanks.

[netlify]

✅ Deploy Preview for opensubsonic ready!

Name	Link
🔨 Latest commit	91c1473
🔍 Latest deploy log	https://app.netlify.com/projects/opensubsonic/deploys/69d2d74f0446c80008aeba93
😎 Deploy Preview	https://deploy-preview-219--opensubsonic.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Tolriq]

This is not an AudioMuse API :)

You need to remove all your specific stuff that no other solutions would be able to implement.

Either we only return only the itemId or we return a child item, but random single fields like author.

findPath should return the distance from the first song too.

Distance should be properly normalized 0..1f

Since this is all about sonic the endpoints should have Sonic in the names.

What is the actual usage of pathFixSize ? If there's only 3 songs and we request 10 it adds random unrelated songs ? I do not see the points as we have the max and client can remove what it wants after.

radiusSimilarity sounds very specific to your implementation should be a toggle on the server side as implementation details.

| album | No | | | Filter results by album name. | What is the purpose of that looks like copy paste error.

[epoupon]

In any case, we indeed only want to return song IDs, all the details are too specific to the data used for clustering.

I’m still not entirely sure it’s a good idea to expose anything related to distances in the API. From what I observe, the distance values can change quite significantly between a library composed only of, for example, metal music and the same library with many other varied genres added. The same path within the metal genre can end up having larger distances in the first case.
So what a client would put as a distance?

It seems to me that we’re already in a pretty good place if we have:

• a path between two songs. Too bad if the two songs are extremely close and the path is very short, we don’t necessarily want to force ourselves to wander all over the collection in that case?
• the N songs closest to a given song, with a maximum distance that the server internally considers representative of its current clustering. That said, if we want to display songs on a map with relationships, the distance to the requested song might be interesting to return for each song, but just for display purpose.

[NeptuneHub]

I renamed them findSonicPath and getSonicSimilarTracks.

Then to keep them deadly simple the Input parameter are the id of the song and the number of song in response.

Output parameter are id of the song and, just for getSonicSimilarTracks, the distance.

I really think that this is the bare minimum then to create update the queue and avoid API tailored on AudioMuse-AI. Hope that finally we can reach a consensus on this.

[deluan]

I'm not sure if we should return only song IDs. IMO we should return child elements, for two reasons:

• Keep consistency. All other endpoints return child when retrieving a list of songs from the server (getPlaylist, getPlayQueue, getSimilarSongs,...)
• If we return only song IDs, an online-first client would have to do multiple queries to get each song details (getSong), if needed.

[Tolriq]

The distance was in the result, so clients can better identify the data it get and can use it.

For example Symfonium have extra features for the mixes like excluded from mix value to for songs, or exclude recently skipped, this means that I always request more data than needed to be able to filter out those. But if I ask 20 similar to metal and it returns classical near the end, I can detect via the distribution of the distance, and remove those too far away.

So distance normalized to 0.0 1.0 is important for the client to take actions.

[NeptuneHub]

So let's follow the suggestion of both.

findSonicPath is just an array of child. getSonicSimilarTracks add in the array of child also the distance.

When a plugin (I think to AudioMuse-AI, but any plugin) will be implemented on top of this api, it will override just the ID and I suppose will be the music server backend to add all the other information. The frontend in this way will need to call only one api and will have all the information needed in it.

[epoupon]

The distance was in the result, so clients can better identify the data it get and can use it.

For example Symfonium have extra features for the mixes like excluded from mix value to for songs, or exclude recently skipped, this means that I always request more data than needed to be able to filter out those. But if I ask 20 similar to metal and it returns classical near the end, I can detect via the distribution of the distance, and remove those too far away.

So distance normalized to 0.0 1.0 is important for the client to take actions.

Yes, but normalized to what?
0.0 for the first song, 1.0 for the returned one which is the farthest?
If it is normalized regarding the whole library, I am afraid values will change from one library to another. If there are only metal songs in the library, you would eliminate legit values

[NeptuneHub]

Look there is NOTHING to invent here. This is mathematics.

When we talk of sonic analysis we talk of vector.

When we talk of vector the gold standard for distance is cosine similarity that assume value between -1 and 1.
This can be normalized in 0 - 1 interval. This means that starting from a song A, the totally opposite song that you can find have distance 1.

If you return 0 means that you are returning the exact same songs (AudioMuse-AI usually delete this result because a re dupplicate), 1 instead shouldn’t happen or you’re returning random song (or you have to sparse collection to don’t have enough similar one).

I just suggest use the result as they are, no need of distance.
Tolriq, anyway, if the distance is higher than (I don’t know) prefear to troncate the result.

Anyway I think that having the distance value here could have sense. I’ll leave as it is now.

Hope that we can agree on this.

[Tolriq]

This is not about arbitrary value, this is about relative distance to detect out of scope songs and improve path pruning.

1. sonic similarity

I start a metal song and ask 20 similar songs in case I need to drop many of them, the first songs are with distance like 0.01 to 0.2 then the 11th song is 0.5 the relative gap shows that we are not in the same similar song range and so they should probably be excluded because the server returned less relevant data just to reach the 20 asked.

This is also math and logic ;) Similarity with forced counts requires a score/distance to filter, else I ask for 1000 songs similar on a 1000 song library and it will always return the full library making the system useless ;)

2. find path

The distance in the find path system also serves a purpose, again because Symfonium have extra features to remove songs from mixes. When requesting a path from song A to song B I will requests a longer path that what the user may want (the path size is configurable).

So if I request 10 songs for a 5 song path, I need to prune 5 song if there's no internal exclusion done.

The best way to achieve that is to again take in account the distance to prune the parts that are over represented. If on the 10 songs the firsts 7 songs are very close to the initial song and only the last 3 are close to the end song, I will prefer to prune in the first part of the list than using a linear pruning at fixed spacing.

---

The API needs to fit the users and client needs, and since those features are already in the wild for quite some times with Plex, there's already needs known. (More needs may come, but the existing should be covered)

[NeptuneHub]

ok added the distance also to findSonicPath if this could be useful. AudioMuse-Ai don't return it but we said that this API must be general so ok. In this case the distance is intended from the starting song.

[Tolriq]

If you don't support that then we need to define a value for unknown like -1. If you return 0.0 for all songs clients might detect as exact same and drop them, and 1.0 as too far and drop them too.

[epoupon]

Not sure we want to enforce cosine similarity distances in the API.
Maybe we should not even mention "distance" but only something like "score" (1.0 closest, 0.0 farthest)
I imagine these endpoints could be very generic. An implementaiton could return similarity results directly based on tags, or on acoustic similarities, or even a mix of these two methods.

[NeptuneHub]

Ok also added the possibility of -1 for distance when not supported.

[NeptuneHub]

is something else missing ? Can we proceed with approving this PR to be merged ?

[Tolriq]

You really need to chill down :) Always too pushy everywhere :)

You forget to answer to @epoupon and I think he's right we should make it even more agnostic.

[NeptuneHub]

I think I already wrote some message above that because we made this API specific for Sonic Similarity, cosine similarity is the gold standard. So that I like to keep as it is.

[epoupon]

I'm fine using the term sonic similarity, but we shouldn't enforce what is supposed to represent the distance on the server side, because that's too specific to the chosen implementation

[rendyhd]

Following along with interest, I think this is really close to being ready!

Expanding on @epoupon's suggestion: I believe the concern is that the spec should define what the similarity value means, not how it's computed. Different servers that implement sonic similarity might use different audio analysis methods under the hood (we shouldn't force them to use the golden standard), and they should all be able to conform as long as they return a normalized value.

To summarize, the concrete suggestion would be:

• Rename distance to similarity (or score) with 1.0 = most similar, 0.0 = least similar
• Keep -1 for servers that don't support returning a value
• Spec just requires a normalized 0.0–1.0 value, without prescribing the method

This would still work for @Tolriq's pruning use cases since the relative gaps between values are preserved, just ordered high-to-low instead of low-to-high.

[NeptuneHub]

Ok, hoping this could help in reach an agreement and proceed with the merge:

• I renamed distance to similarity, still in the range 1.0 = most similar; 0.0 = least similar; -1 not provided

Then AudioMuse-AI will use Cosine Similarity normalized between 0 and 1, other implementation can use different one.

[Tolriq]

As @epoupon said do not forget to add the extension part (https://opensubsonic.netlify.app/docs/extensions/template/)

[NeptuneHub]

As @epoupon said do not forget to add the extension part (https://opensubsonic.netlify.app/docs/extensions/template/)

Added.