Skip to content

Removes special null-but-not-null case in multiple comparison columns#11035

Closed
egalpin wants to merge 2 commits intoapache:masterfrom
egalpin:egalpin/remove-special-null-case-multi-comp-col
Closed

Removes special null-but-not-null case in multiple comparison columns#11035
egalpin wants to merge 2 commits intoapache:masterfrom
egalpin:egalpin/remove-special-null-case-multi-comp-col

Conversation

@egalpin
Copy link
Member

@egalpin egalpin commented Jul 5, 2023

Relates to #11027 (and #10704).

The summary of changes in this PR is: don't have null in ComparisonColumns list of Comparables, ever.

I believe this PR represents a way to avoid getting into a situation where #11027 would be needed to recover.

Given a column for which isNull(<docId>) returns true, previously an explicit null would be stored in the list of Comparables. After this PR, the defaultValue for that column would be stored in the list of Comparables instead of null. This allows for "regular" use of removeNullValueField for all (post-combined) non-null comparison columns.

cc @KKcorps @Jackie-Jiang thoughts?

"Upsert comparison column: %s must be comparable", columnName);
comparisonValues[i] = (Comparable) comparisonValue;
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change means that we always set a non-null value for each index of comparisonValues, whereas previous all indices != comparableIndex would be null.

if (comparisonResult != 0) {
return comparisonResult;
}
int comparisonResult = comparisonValue.compareTo(otherComparisonValue);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no more null-checking required, because _values will always hold all non-null entries (where some may be <defaultNullValue>

} else {
comparisonResult = comparisonValue.compareTo(otherComparisonValue);
}
int comparisonResult = comparisonValue.compareTo(otherComparisonValue);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no more null-checking required, because _values will always hold all non-null entries (where some may be <defaultNullValue>

@codecov-commenter
Copy link

codecov-commenter commented Jul 5, 2023

Codecov Report

Merging #11035 (ff178f4) into master (2fabbe4) will decrease coverage by 0.12%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #11035      +/-   ##
==========================================
- Coverage    0.11%    0.00%   -0.12%     
==========================================
  Files        2199     2184      -15     
  Lines      118779   118378     -401     
  Branches    18000    17940      -60     
==========================================
- Hits          137        0     -137     
+ Misses     118622   118378     -244     
+ Partials       20        0      -20     
Flag Coverage Δ
integration1temurin11 0.00% <0.00%> (ø)
integration1temurin17 0.00% <0.00%> (ø)
integration1temurin20 0.00% <0.00%> (ø)
integration2temurin11 0.00% <0.00%> (?)
integration2temurin17 0.00% <0.00%> (?)
integration2temurin20 0.00% <0.00%> (?)
unittests1temurin11 0.00% <0.00%> (?)
unittests1temurin17 0.00% <0.00%> (ø)
unittests1temurin20 ?
unittests2temurin11 ?
unittests2temurin17 ?
unittests2temurin20 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...local/indexsegment/mutable/MutableSegmentImpl.java 0.00% <0.00%> (ø)
.../pinot/segment/local/upsert/ComparisonColumns.java 0.00% <0.00%> (ø)
...not/segment/local/upsert/PartialUpsertHandler.java 0.00% <0.00%> (ø)
...apache/pinot/segment/local/upsert/UpsertUtils.java 0.00% <0.00%> (ø)

... and 22 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@egalpin
Copy link
Member Author

egalpin commented Jul 5, 2023

Confirmed with local integration (in addition to unit tests) that this change results in all the same desired outcomes as before, but lacks the undesirable outcome noted in #11027

@KKcorps
Copy link
Contributor

KKcorps commented Jul 6, 2023

Sounds good! Let me test it out with restart flows as well.

@egalpin egalpin force-pushed the egalpin/remove-special-null-case-multi-comp-col branch from 1e05dd6 to ff178f4 Compare July 6, 2023 16:52
@Jackie-Jiang
Copy link
Contributor

Picked the solution in #11044

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants