ARROW-5902: [Java] Implement hash table and equals & hashCode API for dictionary encoding

[tianchen92]

As discussed in #4792

Implement a hash table to only store hash & index, meanwhile add check equal function in ValueVector API.

[tianchen92]

@emkornfield Hi Micah, I made a simple implementation for dictionary hash table.
1 Implement hash table to store dictionary vector hash & index, support lookup with given toEncode vector index.
2 add hashCode and equals methods in ValueVector API.
Please take a look and give some comments, thanks!

[tianchen92]

@liyafan82 Thanks for your comments, This PR is just a demo as I commented as WIP and this will work for VarCharVector type encoding in TestDictionaryVector to test correctness. default and memory copy already commented as TODO to make it perfect since we have to use memory level hashCode and implement equals in all type vectors.

[codecov-io]

Codecov Report

Merging #4846 into master will increase coverage by 2.16%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4846      +/-   ##
==========================================
+ Coverage   87.42%   89.58%   +2.16%     
==========================================
  Files         994      661     -333     
  Lines      140102    96617   -43485     
  Branches     1418        0    -1418     
==========================================
- Hits       122481    86555   -35926     
+ Misses      17259    10062    -7197     
+ Partials      362        0     -362

Impacted Files	Coverage Δ
cpp/src/gandiva/precompiled/decimal_ops.cc	97.17% <0%> (ø)	⬆️
cpp/src/gandiva/precompiled/decimal_ops_test.cc	97.43% <0%> (ø)	⬆️
r/src/recordbatch.cpp
r/R/Table.R
js/src/util/fn.ts
go/arrow/array/bufferbuilder.go
r/src/symbols.cpp
rust/datafusion/src/execution/projection.rs
rust/datafusion/src/execution/filter.rs
rust/arrow/src/csv/writer.rs
... and 326 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c350bba...2db7302. Read the comment docs.

[emkornfield]

this is a no-op I believe.

[tianchen92]

Thanks for comments, IMO, what dose this mean? :)

[tianchen92]

@emkornfield Could you please explain this a little more? thanks

[emkornfield]

I think Integer.hashCode(value) == value?

[tianchen92]

I see, you are right, thanks, fixed.

[emkornfield]

This looks like a good start do you see performance improvements? If so I think the next step would be to add unit tests.

Also, CC @praveenbingo and @pravindra to make sure they don't have concerns with the new interfaces.

[tianchen92]

This looks like a good start do you see performance improvements? If so I think the next step would be to add unit tests.

Also, CC @praveenbingo and @pravindra to make sure they don't have concerns with the new interfaces.

Sure, I have already added a DictionaryEncoderBenchmark and list its previous performance data in #4786.
The performance improvement with this PR is as below:
DictionaryEncoderBenchmarks#testEncode:
Before: 5 415703.943 ± 9258.049 ns/op
After: 5 386094.115 ± 19138.808 ns/op

And add what kind of UT? now TestDictionaryVector already support many types like Struct, List, Binary, Varchar etc.

[emkornfield]

Sorry forgot about the existing unit tests. LGTM.

[emkornfield]

+1, than you