Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Static Analysis #1350

Merged
merged 7 commits into from
Aug 21, 2020
Merged

Update Static Analysis #1350

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ec364e7
Update Static Analysis WIP
RussellSpitzer Aug 17, 2020
3bc9cd6
Fix Style Issues that were Previously False Negatives
RussellSpitzer Aug 20, 2020
b78acbc
Fix Optional orElse
RussellSpitzer Aug 20, 2020
1aeaf75
Switch to Effectively Local Scheme in Reader
RussellSpitzer Aug 20, 2020
0f23296
Fix Spacing
RussellSpitzer Aug 21, 2020
e452662
Fix all Single Line Constructors
RussellSpitzer Aug 21, 2020
2ec9ead
Fix Comments without Whitespace
RussellSpitzer Aug 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix Optional orElse 
OrElse vs OrElseGet
  • Loading branch information
@RussellSpitzer
RussellSpitzer committed Aug 20, 2020
commit b78acbcfec9e0fa5f41fd3eec2bced304c36e42e
4 changes: 2 additions & 2 deletions mr/src/main/java/org/apache/iceberg/mr/mapred/MapredIcebergInputFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,14 +148,14 @@ public void close() throws IOException {

private static JobContext newJobContext(JobConf job) {
JobID jobID = Optional.ofNullable(JobID.forName(job.get(JobContext.ID)))
.orElse(new JobID());
.orElseGet(JobID::new);

return new JobContextImpl(job, jobID);
}

private static TaskAttemptContext newTaskAttemptContext(JobConf job, Reporter reporter) {
TaskAttemptID taskAttemptID = Optional.ofNullable(TaskAttemptID.forName(job.get(JobContext.TASK_ATTEMPT_ID)))
.orElse(new TaskAttemptID());
.orElseGet(TaskAttemptID::new);

return new TaskAttemptContextImpl(job, taskAttemptID, toStatusReporter(reporter));
}
Expand Down
2 changes: 1 addition & 1 deletion orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ private static TypeDescription buildOrcProjection(Integer fieldId, Type type, bo
// e.g. renaming column c -> d and adding new column d
String name = Optional.ofNullable(mapping.get(nestedField.fieldId()))
.map(OrcField::name)
.orElse(nestedField.name() + "_r" + nestedField.fieldId());
.orElseGet(() -> nestedField.name() + "_r" + nestedField.fieldId());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice that it catches these.

TypeDescription childType = buildOrcProjection(nestedField.fieldId(), nestedField.type(),
isRequired && nestedField.isRequired(), mapping);
orcType.addField(name, childType);
Expand Down
7 changes: 4 additions & 3 deletions spark2/src/main/java/org/apache/iceberg/spark/source/Reader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,9 @@ class Reader implements DataSourceReader, SupportsScanColumnarBatch, SupportsPus
} catch (IOException ioe) {
LOG.warn("Failed to get Hadoop Filesystem", ioe);
}
Boolean localityFallback = LOCALITY_WHITELIST_FS.contains(scheme);
this.localityPreferred = options.get("locality").map(Boolean::parseBoolean)
.orElse(LOCALITY_WHITELIST_FS.contains(scheme));
.orElse(localityFallback);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because schema is non-final we can't just switch this to a lambda

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think actually this would actually be fine, but to make the rules happy we would have to reassign Scheme anyway so I figured we may as well just put the result in a local var

Copy link
Contributor

@rdblue rdblue Aug 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could refactor scheme detection into a method to make scheme effectively final here.

} else {
this.localityPreferred = false;
}
Expand All @@ -154,10 +155,10 @@ class Reader implements DataSourceReader, SupportsScanColumnarBatch, SupportsPus
this.encryptionManager = encryptionManager;
this.caseSensitive = caseSensitive;

this.batchReadsEnabled = options.get("vectorization-enabled").map(Boolean::parseBoolean).orElse(
this.batchReadsEnabled = options.get("vectorization-enabled").map(Boolean::parseBoolean).orElseGet(() ->
PropertyUtil.propertyAsBoolean(table.properties(),
TableProperties.PARQUET_VECTORIZATION_ENABLED, TableProperties.PARQUET_VECTORIZATION_ENABLED_DEFAULT));
this.batchSize = options.get("batch-size").map(Integer::parseInt).orElse(
this.batchSize = options.get("batch-size").map(Integer::parseInt).orElseGet(() ->
PropertyUtil.propertyAsInt(table.properties(),
TableProperties.PARQUET_BATCH_SIZE, TableProperties.PARQUET_BATCH_SIZE_DEFAULT));
}
Expand Down
2 changes: 1 addition & 1 deletion spark2/src/main/java/org/apache/iceberg/spark/source/Writer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ class Writer implements DataSourceWriter {
private FileFormat getFileFormat(Map<String, String> tableProperties, DataSourceOptions options) {
Optional<String> formatOption = options.get("write-format");
String formatString = formatOption
.orElse(tableProperties.getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
.orElseGet(() -> tableProperties.getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
return FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
}

Expand Down
2 changes: 1 addition & 1 deletion spark3/src/main/java/org/apache/iceberg/spark/source/SparkBatchWrite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ class SparkBatchWrite implements BatchWrite {
protected FileFormat getFileFormat(Map<String, String> tableProperties, Map<String, String> options) {
Optional<String> formatOption = Optional.ofNullable(options.get("write-format"));
String formatString = formatOption
.orElse(tableProperties.getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
.orElseGet(() -> tableProperties.getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));
return FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
}

Expand Down