[DWIO] refactor the reader of dwrf/orc

[zuochunwei]

Why I do this? please refer to discussion

We use the proto::Stream_Kind_xxx both for dwrf and orc, and this usage is dangerous, so I refactor this part by the following modifications:

1. add another constructor with a param proto::orc::Stream for DwrfStreamIdentifier
2. add another member method of forKind(const proto::orc::Stream_Kind kind) for EncodingKey
3. using union to represent of proto::orc::StripeFooter* footerOrc_ for orc
4. add StripeStreams::getEncodingOrc() for orc, the response to StripeStreams::getEncoding() for dwrf
5. refactor loadStreams to avoid the mess
6. add init for sub-classes of SelectiveColumnReader, check the format in init(), call initDwrf for dwrf and call initOrc for orc

[zuochunwei]

@rui-mo @JkSelf

[rui-mo]

Do we plan to open a PR to Velox upstream for this change?

[Yuhta]

How about we use a little template magic to reduce code duplication?

void DwrfData::init(StripeStreams& stripe) {
  auto format = stripe.format();
  if (format == DwrfFormat::kDwrf) {
    init<DwrfFormat::kDwrf>(stripe);
  } else {
    VELOX_CHECK(format == DwrfFormat::kOrc);
    init<DwrfFormat::kOrc>(stripe);
  }
}

template <DwrfFormat kFormat>
void DwrfData::init(StripeStreams& stripe) {
  EncodingKey encodingKey{nodeType_->id, flatMapContext_.sequence};

  auto kind = kFormat == DwrfFormat::kDwrf ? proto::Stream_Kind_PRESENT : proto::orc::Stream_Kind_PRESENT;
  std::unique_ptr<dwio::common::SeekableInputStream> stream = stripe.getStream(
      encodingKey.forKind(kind), false);
  if (stream) {
    notNullDecoder_ = createBooleanRleDecoder(std::move(stream), encodingKey);
  }

  // We always initialize indexStream_ because indices are needed as
  // soon as there is a single filter that can trigger row group skips
  // anywhere in the reader tree. This is not known at construct time
  // because the first filter can come from a hash join or other run
  // time pushdown.
  kind = kFormat == DwrfFormat::kDwrf ? proto::Stream_Kind_ROW_INDEX : proto::orc::Stream_Kind_ROW_INDEX;
  indexStream_ = stripe.getStream(
      encodingKey.forKind(proto::orc::Stream_Kind_ROW_INDEX), false);
}

[zuochunwei]

sounds good

[zuochunwei]

the operator of condition ? a : b requires that type a is the same as type b, while proto::Stream_Kind_PRESENT is a different type from proto::orc::Stream_Kind_PRESENT, so it can't be compiled successfully.

let's keep it this way, a method for dwrf and another method for orc.

It's clean although it seems poor skills.

[Yuhta]

We should avoid these duplications as it would increase maintenance burden. We can do it like this instead:

template <DwrfFormat kFormat>
void DwrfData::init(StripeStreams& stripe) {
  EncodingKey encodingKey{nodeType_->id, flatMapContext_.sequence};
  auto id = kFormat == DwrfFormat::kDwrf ? encodingKey.forKind(proto::Stream_Kind_PRESENT) : encodingKey.forKind(proto::orc::Stream_Kind_PRESENT);
  std::unique_ptr<dwio::common::SeekableInputStream> stream = stripe.getStream(id, false);
  if (stream) {
    notNullDecoder_ = createBooleanRleDecoder(std::move(stream), encodingKey);
  }

  // We always initialize indexStream_ because indices are needed as
  // soon as there is a single filter that can trigger row group skips
  // anywhere in the reader tree. This is not known at construct time
  // because the first filter can come from a hash join or other run
  // time pushdown.
  id = kFormat == DwrfFormat::kDwrf ? encodingKey.forKind(proto::Stream_Kind_ROW_INDEX) : encodingKey.forKind(proto::orc::Stream_Kind_ROW_INDEX);
  indexStream_ = stripe.getStream(
      encodingKey.forKind(proto::orc::Stream_Kind_ROW_INDEX), false);
}

[zuochunwei]

OK, let‘s reduce the duplications in a new PR, I will ask my partner to do this for me in the following days.

He is a new member of our team, so it's a good start for him to make such changes.

[zuochunwei]

Do we plan to open a PR to Velox upstream for this change?

Yes

[zhejiangxiaomai]

if(NOT DEFINED CMAKE_INSTALL_LIBDIR)
set(CMAKE_INSTALL_LIBDIR "lib")
endif()

[rui-mo]

Comment starts with capitalized letter and ends with .. Same for below.

[zuochunwei]

OK