-
Notifications
You must be signed in to change notification settings - Fork 6.7k
JNI Reader for Table Iterator #12511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@cbi42 @jowlyzhang Can you please review this? In ozone we depend on jni for rocksdb apis. I have just added JNI layer for the table_iterator written in your PR. |
52a2255 to
6b83480
Compare
6b83480 to
dcd5135
Compare
adamretter
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR. I have left some feedback on the Java changes which need some amendment - also a Java test needs to be added please for this new API. I will leave @ajkr @pdillinger and the RocksDB team to comment on the internal C++ changes to RocksDB table... as I cannot myself say if they are desirable or not for the RocksDB team.
| */ | ||
| class RawSstFileReaderIterator extends SstFileReaderIterator { | ||
|
|
||
| protected RawSstFileReaderIterator(SstFileReader reader, long nativeHandle) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please set all arguments and variables to final where possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| */ | ||
| class RawSstFileReaderIterator extends SstFileReaderIterator { | ||
|
|
||
| protected RawSstFileReaderIterator(SstFileReader reader, long nativeHandle) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class needs to implement some distinct form of disposeInternal to avoid leaking memory when it is finished with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It extends SstFileReaderIterator which clears off the required handle.
table/sst_file_reader.cc
Outdated
| return nullptr; | ||
| } | ||
| return std::make_unique<TableIterator>(internal_iter); | ||
| return std::make_unique<TableIterator>(internal_iter, &rep_->options); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ajkr @pdillinger Can you comment on this change in the internals for RocksDB please?
table/table_iterator.h
Outdated
|
|
||
| public: | ||
| explicit TableIterator(InternalIterator* iter) : iter_(iter) {} | ||
| explicit TableIterator(InternalIterator* iter, Options* options) : iter_(iter), options_(options) {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ajkr @pdillinger Can you comment on this change in the internals for RocksDB please?
table/table_iterator.h
Outdated
| void SeekToFirst() override { return iter_->SeekToFirst(); } | ||
| void SeekToLast() override { return iter_->SeekToLast(); } | ||
| void Seek(const Slice& target) override { return iter_->Seek(target); } | ||
| void Seek(const Slice& target) override { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ajkr @pdillinger Can you comment on this change in the internals for RocksDB please?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will review this. Thanks!
table/table_iterator.h
Outdated
| } | ||
| void SeekForPrev(const Slice& target) override { | ||
| return iter_->SeekForPrev(target); | ||
| std::string seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ajkr @pdillinger Can you comment on this change in the internals for RocksDB please?
table/table_iterator.h
Outdated
| void SeekToFirst() override { return iter_->SeekToFirst(); } | ||
| void SeekToLast() override { return iter_->SeekToLast(); } | ||
| void Seek(const Slice& target) override { return iter_->Seek(target); } | ||
| void Seek(const Slice& target) override { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will review this. Thanks!
table/table_iterator.h
Outdated
| void SeekToFirst() override { return iter_->SeekToFirst(); } | ||
| void SeekToLast() override { return iter_->SeekToLast(); } | ||
| void Seek(const Slice& target) override { return iter_->Seek(target); } | ||
| void Seek(const Slice& target) override { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The type of key taken as input for Seek and SeekForPrev should be consistent with the type of key produced by Iterator::key. Can you add jni APIs for the util methods GetInternalKeyForSeek etc instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't we change the definition of the TableIterator to seek and get only on the user_key instead? Isn't this a better way to define this class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The name TableIterator indicates it's iterating the raw table, and that makes internal key a better format of key for these APIs.
table/table_iterator.h
Outdated
| return Status::NotSupported("TableIterator does not support GetProperty."); | ||
| } | ||
|
|
||
| uint64_t SequenceNumber() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The public API is Iterator in rocksdb/includes/iterator.h, the TableIterator is an internal class that is not supposed to be publicly accessible. These SequenceNumber, and type APIs are not defined for Iterator.
We have added ParseEntry util methods to parse sequence number and type out of an internal key. You can have you own wrapper iterator class wrapping a RocksDB Iterator, and add the parsing logic in to the wrapper iterator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need all of these infos separately and storing the parsed entry would be much more optimal. Otherwise for each and every call we would have to parse the key over and over again. All I am doing here is changing the definition of the table iter which would seek over the same user key and return user key when getKey() is called. I have added 2 more functions on top of iterator, I believe that should be fine, since JNI would be called only when TableIterator is initialized.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jowlyzhang TableIterator itself is a wrapper for InternalIterator why have 2 wrappers? Should we consider making this public?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's worth having 2 wrappers because it will be your own wrapper iterator that you don't need to check into RocksDB codebase, you own those logic and you can modify it however you want. Parsing an entry every time over and over again isn't computationally heavy or performance wise concerning, you can just wrap it in your own wrapper iterator. It's beset for the RocksDB TableIterator to just do the most vanilla things of iterating raw entries. Others wishing to add their own iterating constraints can just add on top of it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jowlyzhang I have updated the code to add another jni functionality to convert internal Key to user key and vice versa
|
@jowlyzhang Can you check this PR? |
Thanks for making the changes. Since this PR now mostly have java changes, I'll leave it to @adamretter to make the call here. |
adamretter
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issues I think I have identified repeat throughout and should be addressed please
| kEntryWideColumnEntity((byte)0x7), | ||
| kEntryOther((byte)0x8); | ||
| private final byte value; | ||
| private static Map<Byte, EntryType> reverseLookupMap = new HashMap<>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need a map here? The collection is small enough to iterate I would have thought?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed the map and falling back to iterating through the values.
| private final byte value; | ||
| private static Map<Byte, EntryType> reverseLookupMap = new HashMap<>(); | ||
|
|
||
| EntryType(byte value) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be final param.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
java/rocksjni/parsed_entry_info.cc
Outdated
| jint len) { | ||
| const std::unique_ptr<char[]> target(new char[len]); | ||
| if (target == nullptr) { | ||
| jclass oom_class = env->FindClass("/lang/java/OutOfMemoryError"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this has been tested as the class name is incorrect. Also there are util methods in portal.h for throwing errors that could be used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added util method to also do operations on indirect byte array buffers
java/rocksjni/parsed_entry_info.cc
Outdated
| "Memory allocation failed in RocksDB JNI function"); | ||
| return; | ||
| } | ||
| env->GetByteArrayRegion(jtarget, off, len, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs error checking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
java/rocksjni/parsed_entry_info.cc
Outdated
| auto slice_size = key_slice.size(); | ||
| jsize copy_size = std::min(static_cast<uint32_t>(slice_size), | ||
| static_cast<uint32_t>(jlen)); | ||
| env->SetByteArrayRegion( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs error checking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
java/rocksjni/type_util.cc
Outdated
| auto slice_size = key_slice.size(); | ||
| jsize copy_size = std::min(static_cast<uint32_t>(slice_size), | ||
| static_cast<uint32_t>(int_key_len)); | ||
| env->SetByteArrayRegion( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs error checking.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
java/rocksjni/type_util.cc
Outdated
| reinterpret_cast<const ROCKSDB_NAMESPACE::Options *>(options_handle); | ||
| const std::unique_ptr<char[]> target(new char[user_key_len]); | ||
| if (target == nullptr) { | ||
| jclass oom_class = env->FindClass("/lang/java/OutOfMemoryError"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Class name and testing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
java/rocksjni/type_util.cc
Outdated
| "Memory allocation failed in RocksDB JNI function"); | ||
| return static_cast<jsize>(0); | ||
| } | ||
| env->GetByteArrayRegion(user_key, user_key_off, user_key_len, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs error checking.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
java/rocksjni/type_util.cc
Outdated
| auto slice_size = key_slice.size(); | ||
| jsize copy_size = std::min(static_cast<uint32_t>(slice_size), | ||
| static_cast<uint32_t>(int_key_len)); | ||
| env->SetByteArrayRegion( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs error checking.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
java/rocksjni/type_util.cc
Outdated
| // exception thrown: OutOfMemoryError | ||
| return nullptr; | ||
| } | ||
| env->SetByteArrayRegion( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs error checking.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
|
@adamretter Apologies for the late response and not following through addressing review comments on time. I have addressed all the review comments. Can you please take a look at it? I have added a Java test for the same as well and also added required util functions as well in portal.h. FYI these util functions can be used in other JNI classes as well. |
|
@adamretter Any updates on this? |
|
@adamretter Can you review this PR or is there anyone else who can review this PR if you are loaded with other stuff |
adamretter
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still some missing error handling that needs to be implemented.
java/rocksjni/portal.h
Outdated
| } | ||
| env->GetByteArrayRegion(jkey, jkey_off, jkey_len, | ||
| reinterpret_cast<jbyte*>(target.get())); | ||
| ROCKSDB_NAMESPACE::Slice target_slice(target.get(), jkey_len); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing error check for the preceding JNI call, e.g. if (env->ExceptionOccurred())
java/rocksjni/parsed_entry_info.cc
Outdated
| reinterpret_cast<const ROCKSDB_NAMESPACE::Options *>(options_handle); | ||
| auto *parsed_entry_info = | ||
| reinterpret_cast<ROCKSDB_NAMESPACE::ParsedEntryInfo *>(handle); | ||
| jbyte *target = env->GetByteArrayElements(jtarget, nullptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to consider the lifetime of target... should there be a corresponding call to ReleaseByteArrayElements?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah you are right thanks for pointing out the issue. I have added a boolean in ParsedEntryInfo struct to clear up on every parseEntry function or close
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here it would be much easier to use env->GetByteArrayElements to copy directly into charCopy and avoid having to use the GetByteArrayElements/ReleaseByteArrayElements pair.
java/rocksjni/portal.h
Outdated
| /* Helper for copying value in source into a byte array. | ||
| */ | ||
| template <class T> | ||
| static jint copyToByteArray(JNIEnv* env, T& source, jbyteArray jtarget, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are already some copyBytes functions in portal.h that are very similar apart from they allocate the jbyteArray. Could you create an overload of copyBytes that takes the jtarget, jtarget_off, jtarget_len, and then rename this copyBytes and refactor to remove code duplication?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done, renamed the function copyBytes. I don't think we can really reuse any of the existing function to reuse. I have called the existing copyBytes to create a new byte array instead of calling the copyToByteArray function
| case ROCKSDB_NAMESPACE::EntryType::kEntryOther: | ||
| return 0x8; | ||
| default: | ||
| return 0x9; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel uncomfortable with this approach that raises an exception for an invalid enum value. Is there a better approach perhaps?
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_indirect(getInternalKeySeek, env, user_key, | ||
| user_key_off, user_key_len); | ||
| ROCKSDB_NAMESPACE::Slice key_slice = seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing an error check to see if k_op_indirect raised a Java exception.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed in k_op_indirect function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same principle. Respect the k_op_indirect and check the status of ROCKSDB_NAMESPACE::GetInternalKeyForSeekForPrev, raising an exception if the status is bad.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_indirect(getInternalKeySeek, env, user_key, | ||
| user_key_off, user_key_len); | ||
| ROCKSDB_NAMESPACE::Slice key_slice = seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing an error check to see if k_op_indirect raised a Java exception.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed in k_op_indirect function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above, please respect the contract for k_op_indirect which allows a Java exception to be raised, and check the result of ROCKSDB_NAMESPACE::GetInternalKeyForSeekForPrev for a possible failure status, raising an exception if the status is an error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added an exception check here
| } | ||
| ROCKSDB_NAMESPACE::JniUtil::copyToByteArray( | ||
| env, key_slice, jkey, 0, static_cast<jint>(key_slice.size())); | ||
| return jkey; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing an error check to see if copyToByteArray raised a Java exception.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added a check in copyBytes function
java/rocksjni/type_util.cc
Outdated
| } | ||
| ROCKSDB_NAMESPACE::JniUtil::copyToByteArray( | ||
| env, key_slice, jkey, 0, static_cast<jint>(key_slice.size())); | ||
| return jkey; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing an error check to see if copyToByteArray raised a Java exception.
java/rocksjni/type_util.cc
Outdated
| } | ||
| ROCKSDB_NAMESPACE::JniUtil::copyToByteArray( | ||
| env, key_slice, jkey, 0, static_cast<jint>(key_slice.size())); | ||
| return jkey; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing an error check to see if copyToByteArray raised a Java exception.
|
@swamirishi Are you still working on this? |
|
|
@adamretter I have addressed most of the review comments to the best of my knowledge. I had a couple of doubts for few review comments and have left comments inline. |
|
@adamretter Can this be merged? |
|
@adamretter Do you think this can be merged? |
|
@swamirishi I have asked my colleague here @alanpaxton to take a look at this for you. He will come back to you shortly... |
alanpaxton
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have pointed out a few issues around memory management and lifetime, as well as some coding style/encapsulation issues. Happy to discuss further if that will be of assistance.
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_indirect(getInternalKeySeek, env, user_key, | ||
| user_key_off, user_key_len); | ||
| ROCKSDB_NAMESPACE::Slice key_slice = seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above, please respect the contract for k_op_indirect which allows a Java exception to be raised, and check the result of ROCKSDB_NAMESPACE::GetInternalKeyForSeekForPrev for a possible failure status, raising an exception if the status is an error.
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_direct(getInternalKeySeek, env, user_key, | ||
| user_key_off, user_key_len); | ||
| ROCKSDB_NAMESPACE::Slice key_slice = seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The contract for k_op_direct allows for a Java exception. Therefore you should check for it, to ensure encapsulation. Notice in fact that ROCKSDB_NAMESPACE::GetInternalKeyForSeekForPrev returns a status, so your lambda getInternalKeySeek should check that status and raise a Java exception from it, using a RocksDBExceptionJni::ThrowNew
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_indirect(getInternalKeySeek, env, user_key, | ||
| user_key_off, user_key_len); | ||
| ROCKSDB_NAMESPACE::Slice key_slice = seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same principle. Respect the k_op_indirect and check the status of ROCKSDB_NAMESPACE::GetInternalKeyForSeekForPrev, raising an exception if the status is bad.
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_direct(getInternalKeySeek, env, user_key, | ||
| user_key_off, user_key_len); | ||
| ROCKSDB_NAMESPACE::Slice key_slice = seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
k_op_direct contract etc..
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_indirect(getInternalKeySeek, env, user_key, | ||
| user_key_off, user_key_len); | ||
| ROCKSDB_NAMESPACE::Slice key_slice = seek_key_buf; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
k_op_direct contract etc..
java/rocksjni/type_util.cc
Outdated
| jbyteArray JNICALL Java_org_rocksdb_TypeUtil_getInternalKeyForPrevJni( | ||
| JNIEnv *env, jclass /*cls*/, jbyteArray user_key, jint user_key_len, | ||
| jlong options_handle) { | ||
| jbyte *target = env->GetByteArrayElements(user_key, nullptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There should also be a corresponding call to ReleaseByteArrayElements here - see https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html
java/rocksjni/parsed_entry_info.cc
Outdated
| reinterpret_cast<const ROCKSDB_NAMESPACE::Options *>(options_handle); | ||
| auto *parsed_entry_info = | ||
| reinterpret_cast<ROCKSDB_NAMESPACE::ParsedEntryInfo *>(handle); | ||
| jbyte *target = env->GetByteArrayElements(jtarget, nullptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here it would be much easier to use env->GetByteArrayElements to copy directly into charCopy and avoid having to use the GetByteArrayElements/ReleaseByteArrayElements pair.
| char* charCopy = new char[len]; | ||
| std::memcpy(charCopy, target, len); | ||
| env->ReleaseByteArrayElements(jtarget, target, JNI_ABORT); | ||
| ROCKSDB_NAMESPACE::Slice target_slice(reinterpret_cast<char *>(target), len); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be charCopy and not target in target_slice(... I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed the implementation to copy directly to target by using GetByteArrayRegion to avoid unnecessary copy from GetByteArray and release since we are going to be copy anyhow at the end.
| ROCKSDB_NAMESPACE::ParseEntry(target_slice, options->comparator, | ||
| parsed_entry_info); | ||
| }; | ||
| ROCKSDB_NAMESPACE::JniUtil::k_op_indirect(parse, env, jtarget, joff, jlen); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The slice data constructed by k_op_indirect has a lifetime which ends with the k_op_indirect call (std::unique_ptr), so the parsed_entry_info cannot be guaranteed to persist when it is returned. I wonder if it would be easier to make the ParsedEntryInfo a complete Java object, with members for key , type, sequence number, and fully instantiate this using JNI from a temporary C++ ParsedEntryInfo.
I also don't like the copied_user_key flag in ParsedEntryInfo, and I think if you refactored in the way I suggest, it would allow you to remove copied_user_key.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be another GC overhead on the jvm for each and every key we might end up creating an object which might be unoptimal if the number of SstFiles are going to be too many. With the byte buffer api we would have all the entries passed as it is. One possible way would be to have different byteBuffers being passed for different fields but this would mean we won't be able to reuse the SstFileReaderIterator object similar to the c++ implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the k_op_indirect callback function to return a boolean value to indicate if the target array can be released. This way we can avoid unnecessary copies.
https://github.com/swamirishi/rocksdb/blob/3a7bd0a7eee27dd18c116597246a9afac75e769e/java/rocksjni/portal.h#L2511-L2514
| } | ||
| } | ||
|
|
||
| private void assertInternalKey(final Options options, final ParsedEntryInfo parsedEntryInfo, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some of the parseEntry() variants are not being tested. Please ensure in particular that all the JNI methods you write get exercised.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done added a new test class SstTableReaderIteratorTest to test out all possible combinations.
…ests to cover all combinations
|
@alanpaxton Apologies for the delay in reverting back I got held up working on other items in Ozone. I have addressed all the review comments. Please check and let me know if this looks good. Actually this is a critical piece for Apache Ozone's rocksdb upgrade plan for performing efficient snapshot diff & Ozone Snapshot Defragmentation it would great if we can merge this change as soon as possible. |
|
@alanpaxton I have also added the api to read the table iterator in range. This would be a good api to have in JNI wherein a java code can do range iteration without having to do unnecessary copies to directByteBuffer/Heap |
Adding a Jni layer for table Iterator. Also changing the table iterator's interface preparse internal key so that SSTFileReaderIterator can be reused.