Independent audio-analyzer

[tukkek]

Would a stand-alone "audio analyzer" tool (separate executable rather than part of the core library) be good? Some hypotheticals I can think of:

1. Better design (own API; de-coupling [1]; separation of concerns);...
2. Can be tested and improved on its own (could be particularly helpful for de-bugging over the project's life-span);
3. Could stream or export analyses (with time-stamps) for third-party applications to use and maybe even develop its own ecosystem;
4. If using a modular or plug-in approach, could be easily replaced with another drop-in [2], which could help in early-development or to test new ideas later on.

Of-course, the visualizer would still use the analyzer internally but as two programs working together rather than a monolithic core.

Are these reasonable or am I over-thinking it? If developing an architecture that allows for it can be attained easily, it could be considered even if the benefits aren't major or immediately obvious, at least as an initial experiment for the next-generation architecture.

[1]: Loose coupling https://en.wikipedia.org/wiki/Loose_coupling
[2]: Modular programming https://simple.wikipedia.org/wiki/Modular_programming

[tukkek]

CodAv makes a good point that it makes more sense to use a pre-existing DSP library rather than re-inventing the wheel. Since a library is already modular by nature, it would also achieve the goals described initially.

Discord link: https://discord.com/channels/737206408482914387/737206408948220057/1282400825028776058

[Blaquewithaq]

I found this lib looking at DSP libraries:

https://github.com/cycfi/q
https://cycfi.github.io/q/q/v1.5-dev/quick_start.html

I'm not familiar with DSP so Im unaware if their is a more popular recommended lib or not.

[kblaschke]

I guess before choosing a specific library, we might first think about what kind of processing would help us with the goal of creating visuals. Like, low-/high-pass filters and graphical EQs are surely nice things to fine-tune the audible quality of music for the listener, but these wouldn't have much of a visual impact.

So, what are we talking about? I can see a few use cases here for processed audio data:

• Convert a waveform into a frequency spectrum. This is what Milkdrop/projectM does, and many audio players show when playing music (these "jumping bars"). Very useful to visualize audio signals.
• Beat detection. MD/pM do this by summming up some frequency bands of the previous point, but this is just a very basic way of doing it. I guess there are better ways of extracting certain frequencies or patterns, e.g. using specifically tailored wavelet envelopes.
• Filtering the waveform for visual representation, e.g. only keep high or low frequencies and remove the others. This can be done by applying a narrow-band high-/low-pass filter as mentioned above.

[producation]

If the audio-analyzer is independent it would be nice to go further with additional features like:

• Support of multiple audio channels (not just stereo) like having the Kick Drum on Channel 1, Bass Player on Channel 2, Keyboarder on Channel 3 & 4 and Vocals on Channel 5 if you do visuals i.e. for a band or with stems. So you can have one layer just visualize when the kick is hitting or the keyboarder is playing. Maybe this could also be handy to support 3D audio like Dolby Atmos, DTS:X, IAMF (Immersive Audio Model and Formats).
• Besides just filtering high/mid/low you could add (multiple) FFT per channel, preferably with a graphical eq, for modulation of specific effects and filtering which frequency range it should react to. This data could also be of use by sending it to another software via OSC [1] like in Synesthesia [2].
• Beat Detection could be syncable with Ableton Link [3] which is pretty common nowadays and also supported by several DJ and VJ software.

[1]: OSC - https://en.wikipedia.org/wiki/Open_Sound_Control
[2]: OSC of Synestesia - https://app.synesthesia.live/docs/manual/osc.html
[3]: Ableton Link - https://github.com/Ableton/link

[revmischa]

I bet ChatGPT etc would be great at writing this stuff