Skip to content

Add processedStrides and processedSplits metrics #264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 24, 2023
Merged

Add processedStrides and processedSplits metrics #264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5091dd9
add metrics
rui-mo May 19, 2023
9da865c
test
rui-mo May 23, 2023
6027087
Revert "test"
rui-mo May 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions velox/connectors/hive/HiveConnector.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -547,6 +547,7 @@ void HiveDataSource::addSplit(std::shared_ptr<ConnectorSplit> split) {
runtimeStats_.skippedSplitBytes += split_->length;
return;
}
++runtimeStats_.processedSplits;

auto fileType = reader_->rowType();

Expand Down
10 changes: 9 additions & 1 deletion velox/dwio/common/Statistics.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -517,18 +517,26 @@ struct RuntimeStatistics {
// Number of splits skipped based on statistics.
int64_t skippedSplits{0};

// Number of splits processed based on statistics.
int64_t processedSplits{0};

// Total bytes in splits skipped based on statistics.
int64_t skippedSplitBytes{0};

// Number of strides (row groups) skipped based on statistics.
int64_t skippedStrides{0};

// Number of strides (row groups) processed based on statistics.
int64_t processedStrides{0};

std::unordered_map<std::string, RuntimeCounter> toMap() {
return {
{"skippedSplits", RuntimeCounter(skippedSplits)},
{"processedSplits", RuntimeCounter(processedSplits)},
{"skippedSplitBytes",
RuntimeCounter(skippedSplitBytes, RuntimeCounter::Unit::kBytes)},
{"skippedStrides", RuntimeCounter(skippedStrides)}};
{"skippedStrides", RuntimeCounter(skippedStrides)},
{"processedStrides", RuntimeCounter(processedStrides)}};
}
};

Expand Down
1 change: 1 addition & 0 deletions velox/dwio/parquet/reader/ParquetReader.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -612,6 +612,7 @@ bool ParquetRowReader::advanceToNextRowGroup() {
void ParquetRowReader::updateRuntimeStats(
dwio::common::RuntimeStatistics& stats) const {
stats.skippedStrides += skippedRowGroups_;
stats.processedStrides += rowGroupIds_.size();
}

void ParquetRowReader::resetFilterCaches() {
Expand Down
3 changes: 0 additions & 3 deletions velox/dwio/parquet/reader/TimestampColumnReader.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,6 @@ class TimestampColumnReader : public IntegerColumnReader {
common::ScanSpec& scanSpec)
: IntegerColumnReader(nodeType, nodeType, params, scanSpec) {}

static constexpr int64_t JULIAN_TO_UNIX_EPOCH_DAYS = 2440588LL;
static constexpr int64_t SECONDS_PER_DAY = 86400LL;

void read(
vector_size_t offset,
RowSet rows,
Expand Down
4 changes: 4 additions & 0 deletions velox/exec/tests/PrintPlanWithStatsTest.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,8 @@ TEST_F(PrintPlanWithStatsTest, innerJoinWithTableScan) {
{" prefetchBytes [ ]* sum: .+, count: 1, min: .+, max: .+"},
{" preloadedSplits[ ]+sum: .+, count: .+, min: .+, max: .+",
true},
{" processedSplits [ ]* sum: 1, count: 1, min: 1, max: 1"},
{" processedStrides [ ]* sum: 0, count: 1, min: 0, max: 0"},
{" queryThreadIoLatency[ ]* sum: .+, count: .+ min: .+, max: .+"},
{" ramReadBytes [ ]* sum: .+, count: 1, min: .+, max: .+"},
{" readyPreloadedSplits[ ]+sum: .+, count: .+, min: .+, max: .+",
Expand Down Expand Up @@ -257,6 +259,8 @@ TEST_F(PrintPlanWithStatsTest, partialAggregateWithTableScan) {
{" numRamRead [ ]* sum: 6, count: 1, min: 6, max: 6"},
{" numStorageRead [ ]* sum: .+, count: 1, min: .+, max: .+"},
{" prefetchBytes [ ]* sum: .+, count: 1, min: .+, max: .+"},
{" processedSplits [ ]* sum: 1, count: 1, min: 1, max: 1"},
{" processedStrides [ ]* sum: 0, count: 1, min: 0, max: 0"},
{" preloadedSplits[ ]+sum: .+, count: .+, min: .+, max: .+",
true},
{" queryThreadIoLatency[ ]* sum: .+, count: .+ min: .+, max: .+"},
Expand Down