Skip to content

Commit 786088b

Browse files
Improve performance when rendering sample waveforms (#7366)
This PR attempts a number of improvements to the sample rendering (in the song editor, automation editor, AudioFileProcessor, SlicerT, etc) in LMMS: Thumbnails: Samples are aggregated into a set of thumbnails of multiple resolutions. The smallest larger thumbnail is then chosen for rendering during sample drawing. Each set of thumbnails is stored with its sample file metadata in an unordered_map, so that duplicate samples will use the same set of thumbnails. Partial rendering: additionally, this PR only renders the portions of the sample clips visible to the viewer to save rendering time. * Experimental sample thumbnail * Rename some classes and type aliases. Make some type declarations explicit * Use a combination of audioFile name and shared_ptrs to track samples; refactor some loops * That weird line at the end of the sample is now gone * Small changes to the code; Add comments * Add missing word to comment; Fix typo * Track `SharedSampleThumbnailList`s instead * Major refactor; implement thumbnailing for SlicerT, AFP and Automation editor * Code clean up, renames and documenting * Add the namespace lmms comments * More code updates and documentation * Fix error in comment * Comment out `qDebug`s * Fix formatting * Use alternative initialization of `SampleThumbnailVisualizeParameters` * Remove commented code * Fix style and simplify code * Use auto * Draw lines using floating point * Merge the classes into one nested class + Replace while loop with std::find_if * Fix comparison of different signedness * Include memory header * Fix a logic error when selecting samples; Rename a const * Fix more issues with signedness * Fix sample drawing error in `visualizeOriginal` * Only render regions in view of the sample * Allow partial repaints of sample clips * Remove unused variable * Limit most of the painting to the visible region * Revert back to using rect() in some places * Partial rendering for AutomationEditor * Don't redraw painted regions; allowHighResolution; remove `visualizeOriginal`; Remove s_sampleThumbnailCacheMap * Add s_sampleThumbnailCacheMap back for testing convenience * Minor change to the way `thumbnailSizeDivisor` is calculated * Extend update region by an amount * forgot about this * Adapt to master; Redesign VisualizeParameters; Don't rely entirely on needsUpdate() * Don't try to preserve painted regions * Allow for a bit more thumbnails; Fix incorrect rendering when vertically scrolling * Fix missing include statement * Remove the unused variable * Code optimization; Remove RMS member from Bit; Rename viewRect to drawRect * More code optimizations * Fix formatting * Use begin instead of cbegin * Improve generation of thumbnails * Improve expressiveness of the draw code * Add support for reversing * Fix drawing code * Fix draw code (part 2) * Apply more fixes and simplifications * Undo some out of scope changes * Remove SampleWaveform * Improve documentation * Use size_t for some counters * Change width parameter to be size_t * Remove temporary aggregated peak variable * Bump up AggregationPerZoomStep to 10 * Zoom out only requested range of thumbnail instead of clipping it separately * Rename targetSampleWidth to targetThumbnailWidth * Handle reversing for AFP; Iterate in reverse instead of reversing the entire thumbnail * Change names to be more accurate * Improve implementation of sample thumbnail cache map * Move AggregationPerZoomStep back down to 2, do not cap smallest thumbnail width to display width To improve performance with especially long samples (~20 minutes) * Simplify sample thumbnail cache handling in constructor * Call drawLines instead of drawLine in a loop QPainter::drawLine calls drawLines with a line count of 1. Therefore, calling drawLine in a loop means we are simply calling drawLines a lot more times than necessary. * Bump up AggregationPerZoomStep to 10 again Thought using 2 would help performance (when rendering). Maybe it does, but it's still quite slow when rendering a bunch of thumbnails at once. * Fix off-by-one error when constructing Thumbnail from buffer * Fix crash when viewport is not in bounds * Apply performance improvements Performance in the zoomOut function was bad because of the dynamic memory allocation. Huge chunks of memory were being allocated quite often, casuing a ton of cache misses and all around slower performance. To fix this, all the necessary downsampling is now done within the for loop and handled one pixel after another, instead of all at once. To further avoid allocations in the draw call, the change to use drawLines has been reverted. We now pass VisualizeParameters by value (it is only 64 bytes, which will fit quite nicely in a cache line, which is more benefical than reaching for that data by reference to some other part of the code). After applying these changes, we are now practically only bounded by the painting done by Qt (according to profiling done by Valgrind). Proper use of OpenGL could resolve this issue, which should finally make the performance quite optimal in variety of situations. * Apply minor changes Update copyright and unused functions. Move in newly created thumbnail into the cache instead of copying it. * Use C++20's designated initializers * Create param right before visualizing * Fix regressions with reversing * Fix incorrect rendering in AFP and SlicerT * Move MaxSampleThumbnailCacheSize and AggregationPerZoomStep into implementation file * Remove static keyword * Remove getter and setter for peak data --------- Co-authored-by: Sotonye Atemie <sakertooth@gmail.com>
1 parent fe0e8ba commit 786088b

13 files changed

+386
-177
lines changed

include/AutomationEditor.h

+3
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
#include "SampleClip.h"
3939
#include "TimePos.h"
4040
#include "lmms_basics.h"
41+
#include "SampleThumbnail.h"
4142

4243
class QPainter;
4344
class QPixmap;
@@ -290,6 +291,8 @@ protected slots:
290291
QColor m_ghostNoteColor;
291292
QColor m_detuningNoteColor;
292293
QColor m_ghostSampleColor;
294+
295+
SampleThumbnail m_sampleThumbnail;
293296

294297
friend class AutomationEditorWindow;
295298

include/SampleClipView.h

+3
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@
2727

2828
#include "ClipView.h"
2929

30+
#include "SampleThumbnail.h"
31+
3032
namespace lmms
3133
{
3234

@@ -63,6 +65,7 @@ public slots:
6365

6466
private:
6567
SampleClip * m_clip;
68+
SampleThumbnail m_sampleThumbnail;
6669
QPixmap m_paintPixmap;
6770
bool splitClip( const TimePos pos ) override;
6871
} ;

include/SampleThumbnail.h

+143
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
/*
2+
* SampleThumbnail.h
3+
*
4+
* Copyright (c) 2024 Khoi Dau <casboi86@gmail.com>
5+
* Copyright (c) 2024 Sotonye Atemie <sakertooth@gmail.com>
6+
*
7+
* This file is part of LMMS - https://lmms.io
8+
*
9+
* This program is free software; you can redistribute it and/or
10+
* modify it under the terms of the GNU General Public
11+
* License as published by the Free Software Foundation; either
12+
* version 2 of the License, or (at your option) any later version.
13+
*
14+
* This program is distributed in the hope that it will be useful,
15+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
16+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
17+
* General Public License for more details.
18+
*
19+
* You should have received a copy of the GNU General Public
20+
* License along with this program (see COPYING); if not, write to the
21+
* Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
22+
* Boston, MA 02110-1301 USA.
23+
*
24+
*/
25+
26+
#ifndef LMMS_SAMPLE_THUMBNAIL_H
27+
#define LMMS_SAMPLE_THUMBNAIL_H
28+
29+
#include <QDateTime>
30+
#include <QPainter>
31+
#include <QRect>
32+
#include <memory>
33+
34+
#include "Sample.h"
35+
#include "lmms_export.h"
36+
37+
namespace lmms {
38+
39+
/**
40+
Allows for visualizing sample data.
41+
42+
On construction, thumbnails will be generated
43+
at logarathmic intervals of downsampling. Those cached thumbnails will then be further downsampled on the fly and
44+
transformed in various ways to create the desired waveform.
45+
46+
Given that we are dealing with far less data to generate
47+
the visualization however (i.e., we are not reading from original sample data when drawing), this provides a
48+
significant performance boost that wouldn't be possible otherwise.
49+
*/
50+
class LMMS_EXPORT SampleThumbnail
51+
{
52+
public:
53+
struct VisualizeParameters
54+
{
55+
QRect sampleRect; //!< A rectangle that covers the entire range of samples.
56+
57+
QRect drawRect; //!< Specifies the location in `sampleRect` where the waveform will be drawn. Equals
58+
//!< `sampleRect` when null.
59+
60+
QRect viewportRect; //!< Clips `drawRect`. Equals `drawRect` when null.
61+
62+
float amplification = 1.0f; //!< The amount of amplification to apply to the waveform.
63+
64+
float sampleStart = 0.0f; //!< Where the sample begins for drawing.
65+
66+
float sampleEnd = 1.0f; //!< Where the sample ends for drawing.
67+
68+
bool reversed = false; //!< Determines if the waveform is drawn in reverse or not.
69+
};
70+
71+
SampleThumbnail() = default;
72+
SampleThumbnail(const Sample& sample);
73+
void visualize(VisualizeParameters parameters, QPainter& painter) const;
74+
75+
private:
76+
class Thumbnail
77+
{
78+
public:
79+
struct Peak
80+
{
81+
Peak() = default;
82+
83+
Peak(float min, float max)
84+
: min(min)
85+
, max(max)
86+
{
87+
}
88+
89+
Peak(const SampleFrame& frame)
90+
: min(std::min(frame.left(), frame.right()))
91+
, max(std::max(frame.left(), frame.right()))
92+
{
93+
}
94+
95+
Peak operator+(const Peak& other) const { return Peak(std::min(min, other.min), std::max(max, other.max)); }
96+
Peak operator+(const SampleFrame& frame) const { return *this + Peak{frame}; }
97+
98+
float min = std::numeric_limits<float>::max();
99+
float max = std::numeric_limits<float>::min();
100+
};
101+
102+
Thumbnail() = default;
103+
Thumbnail(std::vector<Peak> peaks, double samplesPerPeak);
104+
Thumbnail(const float* buffer, size_t size, size_t width);
105+
106+
Thumbnail zoomOut(float factor) const;
107+
108+
Peak& operator[](size_t index) { return m_peaks[index]; }
109+
const Peak& operator[](size_t index) const { return m_peaks[index]; }
110+
111+
int width() const { return m_peaks.size(); }
112+
double samplesPerPeak() const { return m_samplesPerPeak; }
113+
114+
private:
115+
std::vector<Peak> m_peaks;
116+
double m_samplesPerPeak = 0.0;
117+
};
118+
119+
struct SampleThumbnailEntry
120+
{
121+
QString filePath;
122+
QDateTime lastModified;
123+
124+
friend bool operator==(const SampleThumbnailEntry& first, const SampleThumbnailEntry& second)
125+
{
126+
return first.filePath == second.filePath && first.lastModified == second.lastModified;
127+
}
128+
};
129+
130+
struct Hash
131+
{
132+
std::size_t operator()(const SampleThumbnailEntry& entry) const noexcept { return qHash(entry.filePath); }
133+
};
134+
135+
using ThumbnailCache = std::vector<Thumbnail>;
136+
std::shared_ptr<ThumbnailCache> m_thumbnailCache = std::make_shared<ThumbnailCache>();
137+
138+
inline static std::unordered_map<SampleThumbnailEntry, std::shared_ptr<ThumbnailCache>, Hash> s_sampleThumbnailCacheMap;
139+
};
140+
141+
} // namespace lmms
142+
143+
#endif // LMMS_SAMPLE_THUMBNAIL_H

include/SampleWaveform.h

-49
This file was deleted.

plugins/AudioFileProcessor/AudioFileProcessorWaveView.cpp

+15-9
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,16 @@
2424

2525
#include "AudioFileProcessorWaveView.h"
2626

27+
#include "Sample.h"
2728
#include "ConfigManager.h"
29+
#include "SampleThumbnail.h"
2830
#include "FontHelper.h"
29-
#include "SampleWaveform.h"
3031

3132
#include <QPainter>
3233
#include <QMouseEvent>
3334

3435
#include <algorithm>
3536

36-
3737
namespace lmms
3838
{
3939

@@ -81,7 +81,8 @@ AudioFileProcessorWaveView::AudioFileProcessorWaveView(QWidget* parent, int w, i
8181
m_isDragging(false),
8282
m_reversed(false),
8383
m_framesPlayed(0),
84-
m_animation(ConfigManager::inst()->value("ui", "animateafp").toInt())
84+
m_animation(ConfigManager::inst()->value("ui", "animateafp").toInt()),
85+
m_sampleThumbnail(*buf)
8586
{
8687
setFixedSize(w, h);
8788
setMouseTracking(true);
@@ -338,13 +339,18 @@ void AudioFileProcessorWaveView::updateGraph()
338339
m_graph.fill(Qt::transparent);
339340
QPainter p(&m_graph);
340341
p.setPen(QColor(255, 255, 255));
341-
342-
const auto dataOffset = m_reversed ? m_sample->sampleSize() - m_to : m_from;
343342

344-
const auto rect = QRect{0, 0, m_graph.width(), m_graph.height()};
345-
const auto waveform = SampleWaveform::Parameters{
346-
m_sample->data() + dataOffset, static_cast<size_t>(range()), m_sample->amplification(), m_sample->reversed()};
347-
SampleWaveform::visualize(waveform, p, rect);
343+
m_sampleThumbnail = SampleThumbnail{*m_sample};
344+
345+
const auto param = SampleThumbnail::VisualizeParameters{
346+
.sampleRect = m_graph.rect(),
347+
.amplification = m_sample->amplification(),
348+
.sampleStart = static_cast<float>(m_from) / m_sample->sampleSize(),
349+
.sampleEnd = static_cast<float>(m_to) / m_sample->sampleSize(),
350+
.reversed = m_sample->reversed(),
351+
};
352+
353+
m_sampleThumbnail.visualize(param, p);
348354
}
349355

350356
void AudioFileProcessorWaveView::zoom(const bool out)

plugins/AudioFileProcessor/AudioFileProcessorWaveView.h

+2
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727

2828

2929
#include "Knob.h"
30+
#include "SampleThumbnail.h"
3031

3132

3233
namespace lmms
@@ -144,6 +145,7 @@ public slots:
144145
bool m_reversed;
145146
f_cnt_t m_framesPlayed;
146147
bool m_animation;
148+
SampleThumbnail m_sampleThumbnail;
147149

148150
friend class AudioFileProcessorView;
149151

plugins/SlicerT/SlicerTWaveform.cpp

+27-10
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
#include <QBitmap>
2828
#include <qpainterpath.h>
2929

30-
#include "SampleWaveform.h"
30+
#include "SampleThumbnail.h"
3131
#include "SlicerT.h"
3232
#include "SlicerTView.h"
3333
#include "embed.h"
@@ -115,10 +115,19 @@ void SlicerTWaveform::drawSeekerWaveform()
115115
brush.setPen(s_waveformColor);
116116

117117
const auto& sample = m_slicerTParent->m_originalSample;
118-
const auto waveform
119-
= SampleWaveform::Parameters{sample.data(), sample.sampleSize(), sample.amplification(), sample.reversed()};
120-
const auto rect = QRect(0, 0, m_seekerWaveform.width(), m_seekerWaveform.height());
121-
SampleWaveform::visualize(waveform, brush, rect);
118+
119+
m_sampleThumbnail = SampleThumbnail{sample};
120+
121+
const auto param = SampleThumbnail::VisualizeParameters{
122+
.sampleRect = m_seekerWaveform.rect(),
123+
.amplification = sample.amplification(),
124+
.sampleStart = static_cast<float>(sample.startFrame()) / sample.sampleSize(),
125+
.sampleEnd = static_cast<float>(sample.endFrame()) / sample.sampleSize(),
126+
.reversed = sample.reversed()
127+
};
128+
129+
m_sampleThumbnail.visualize(param, brush);
130+
122131

123132
// increase brightness in inner color
124133
QBitmap innerMask = m_seekerWaveform.createMaskFromColor(s_waveformMaskColor, Qt::MaskMode::MaskOutColor);
@@ -171,13 +180,21 @@ void SlicerTWaveform::drawEditorWaveform()
171180
size_t endFrame = m_seekerEnd * m_slicerTParent->m_originalSample.sampleSize();
172181

173182
brush.setPen(s_waveformColor);
174-
float zoomOffset = (m_editorHeight - m_zoomLevel * m_editorHeight) / 2;
183+
long zoomOffset = (m_editorHeight - m_zoomLevel * m_editorHeight) / 2;
175184

176185
const auto& sample = m_slicerTParent->m_originalSample;
177-
const auto waveform = SampleWaveform::Parameters{
178-
sample.data() + startFrame, endFrame - startFrame, sample.amplification(), sample.reversed()};
179-
const auto rect = QRect(0, zoomOffset, m_editorWidth, m_zoomLevel * m_editorHeight);
180-
SampleWaveform::visualize(waveform, brush, rect);
186+
187+
m_sampleThumbnail = SampleThumbnail{sample};
188+
189+
const auto param = SampleThumbnail::VisualizeParameters{
190+
.sampleRect = QRect(0, zoomOffset, m_editorWidth, static_cast<long>(m_zoomLevel * m_editorHeight)),
191+
.amplification = sample.amplification(),
192+
.sampleStart = static_cast<float>(startFrame) / sample.sampleSize(),
193+
.sampleEnd = static_cast<float>(endFrame) / sample.sampleSize(),
194+
.reversed = sample.reversed(),
195+
};
196+
197+
m_sampleThumbnail.visualize(param, brush);
181198

182199
// increase brightness in inner color
183200
QBitmap innerMask = m_editorWaveform.createMaskFromColor(s_waveformMaskColor, Qt::MaskMode::MaskOutColor);

plugins/SlicerT/SlicerTWaveform.h

+4
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@
3232
#include <QMouseEvent>
3333
#include <QPainter>
3434

35+
#include "SampleThumbnail.h"
36+
3537
namespace lmms {
3638

3739
class SlicerT;
@@ -108,6 +110,8 @@ public slots:
108110
QPixmap m_editorWaveform;
109111
QPixmap m_sliceEditor;
110112
QPixmap m_emptySampleIcon;
113+
114+
SampleThumbnail m_sampleThumbnail;
111115

112116
SlicerT* m_slicerTParent;
113117

src/gui/CMakeLists.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ SET(LMMS_SRCS
3535
gui/RowTableView.cpp
3636
gui/SampleLoader.cpp
3737
gui/SampleTrackWindow.cpp
38-
gui/SampleWaveform.cpp
38+
gui/SampleThumbnail.cpp
3939
gui/SendButtonIndicator.cpp
4040
gui/SideBar.cpp
4141
gui/SideBarWidget.cpp

0 commit comments

Comments
 (0)