-
Notifications
You must be signed in to change notification settings - Fork 11.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[compiler-rt][ASan] Add function copying annotations #91702
[compiler-rt][ASan] Add function copying annotations #91702
Conversation
c5ac42d
to
42d3bfd
Compare
✅ With the latest revision this PR passed the C/C++ code formatter. |
cab5bc6
to
8ff3017
Compare
trivial relocation is also interesting for erasing object from a vector, so the builtins should probably be more akin to a |
8ff3017
to
7d3bc1a
Compare
LGTM for Asan however HWASAN story is not clear. It's probably fine, but I'd like to think about this more. |
I still have to fix failing CI, I can sit to it on Monday and, if necessary,
Should we change
Sure, let me know what are your conclusions! |
Yes. My plan is to change the |
Alright, I will update that PR accordingly in the coming week. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assume work in progress.
7725910
to
0ab9216
Compare
82a7c7e
to
93ab376
Compare
This PR adds a `__sanitizer_move_contiguous_container_annotations` function, which moves annotations from one memory area to another. Old area is unpoisoned at the end. New area is annotated in the same way as the old region at the beginning (within limitations of ASan). ```cpp void __sanitizer_move_contiguous_container_annotations( const void *old_storage_beg_p, const void *old_storage_end_p, const void *new_storage_beg_p, const void *new_storage_end_p) { ``` This function aims to help with short string annotations and similar container annotations. Right now we change trait types of `std::basic_string` when compiling with ASan. https://github.com/llvm/llvm-project/blob/87f3407856e61a73798af4e41b28bc33b5bf4ce6/libcxx/include/string#L738-L751 The goal is to not change `__trivially_relocatable` when compiling with ASan. If this function is accpeted and upstreamed, the next step is creating function like `__memcpy_with_asan` moving memory with ASan. And then using this function instead of `__builtin__memcpy` while moving trivially relocatable objects. NOTICE: I did not test it yet, so it's probably not compiling, but I don't expect big changes. PR is WIP until I test it. --- I'm thinking if there is a good way to address fact that in a container the new buffer is usually bigger than the previous one. We may add two more arguments to the functions to address it (the beginning and the end of the whole buffer. Another potential change is removing `new_storage_end_p` as it's redundant, because we require the same size.
Implemented correct container annotations copying function (non-overlapping containers only) Refactored the initial PoC to a fully functional implementation for copying container annotations without moving/copying the memory content itself. This function currently supports cases where the source and destination containers do not overlap. Handling overlapping containers would add complexity. The function is designed to work irrespective of whether the buffers are granule-aligned or the distance between them is granule-aligned. However, such scenarios may have an impact on performance. A Test case has been included to verify the correctness of the implementation. Removed unpoisoning oryginal buffer at the end, users can do it by themselves if they want to.
Added helper functions, removed unnecessary branches, simplified code.
Adds support for overlapping containers. Adds test case for that.
- VPrintf level changed to 3. - unique_ptr used in the test example.
For safety. Fix code formatting.
…rLastGranuleAnnotation
690e3a0
to
b31757b
Compare
This PR adds a `__sanitizer_copy_contiguous_container_annotations` function, which copies annotations from one memory area to another. New area is annotated in the same way as the old region at the beginning (within limitations of ASan). Overlapping case: The function supports overlapping containers, however no assumptions should be made outside of no false positives in new buffer area. (It doesn't modify old container annotations where it's not necessary, false negatives may happen in edge granules of the new container area.) I don't expect this function to be used with overlapping buffers, but it's designed to work with them and not result in incorrect ASan errors (false positives). If buffers have granularity-aligned distance between them (`old_beg % granularity == new_beg % granularity`), copying algorithm works faster. If the distance is not granularity-aligned, annotations are copied byte after byte. ```cpp void __sanitizer_copy_contiguous_container_annotations( const void *old_storage_beg_p, const void *old_storage_end_p, const void *new_storage_beg_p, const void *new_storage_end_p) { ``` This function aims to help with short string annotations and similar container annotations. Right now we change trait types of `std::basic_string` when compiling with ASan and this function purpose is reverting that change as soon as possible. https://github.com/llvm/llvm-project/blob/87f3407856e61a73798af4e41b28bc33b5bf4ce6/libcxx/include/string#L738-L751 The goal is to not change `__trivially_relocatable` when compiling with ASan. If this function is accepted and upstreamed, the next step is creating a function like `__memcpy_with_asan` moving memory with ASan. And then using this function instead of `__builtin__memcpy` while moving trivially relocatable objects. https://github.com/llvm/llvm-project/blob/11a6799740f824282650aa9ec249b55dcf1a8aae/libcxx/include/__memory/uninitialized_algorithms.h#L644-L646 --- I'm thinking if there is a good way to address fact that in a container the new buffer is usually bigger than the previous one. We may add two more arguments to the functions to address it (the beginning and the end of the whole buffer. Another potential change is removing `new_storage_end_p` as it's redundant, because we require the same size. Potential future work is creating a function `__asan_unsafe_memmove`, which will be basically memmove, but with turned off instrumentation (therefore it will allow copy data from poisoned area). --------- Co-authored-by: Vitaly Buka <vitalybuka@google.com>
This PR adds a `__sanitizer_copy_contiguous_container_annotations` function, which copies annotations from one memory area to another. New area is annotated in the same way as the old region at the beginning (within limitations of ASan). Overlapping case: The function supports overlapping containers, however no assumptions should be made outside of no false positives in new buffer area. (It doesn't modify old container annotations where it's not necessary, false negatives may happen in edge granules of the new container area.) I don't expect this function to be used with overlapping buffers, but it's designed to work with them and not result in incorrect ASan errors (false positives). If buffers have granularity-aligned distance between them (`old_beg % granularity == new_beg % granularity`), copying algorithm works faster. If the distance is not granularity-aligned, annotations are copied byte after byte. ```cpp void __sanitizer_copy_contiguous_container_annotations( const void *old_storage_beg_p, const void *old_storage_end_p, const void *new_storage_beg_p, const void *new_storage_end_p) { ``` This function aims to help with short string annotations and similar container annotations. Right now we change trait types of `std::basic_string` when compiling with ASan and this function purpose is reverting that change as soon as possible. https://github.com/llvm/llvm-project/blob/87f3407856e61a73798af4e41b28bc33b5bf4ce6/libcxx/include/string#L738-L751 The goal is to not change `__trivially_relocatable` when compiling with ASan. If this function is accepted and upstreamed, the next step is creating a function like `__memcpy_with_asan` moving memory with ASan. And then using this function instead of `__builtin__memcpy` while moving trivially relocatable objects. https://github.com/llvm/llvm-project/blob/11a6799740f824282650aa9ec249b55dcf1a8aae/libcxx/include/__memory/uninitialized_algorithms.h#L644-L646 --- I'm thinking if there is a good way to address fact that in a container the new buffer is usually bigger than the previous one. We may add two more arguments to the functions to address it (the beginning and the end of the whole buffer. Another potential change is removing `new_storage_end_p` as it's redundant, because we require the same size. Potential future work is creating a function `__asan_unsafe_memmove`, which will be basically memmove, but with turned off instrumentation (therefore it will allow copy data from poisoned area). --------- Co-authored-by: Vitaly Buka <vitalybuka@google.com>
This PR adds a `__sanitizer_copy_contiguous_container_annotations` function, which copies annotations from one memory area to another. New area is annotated in the same way as the old region at the beginning (within limitations of ASan). Overlapping case: The function supports overlapping containers, however no assumptions should be made outside of no false positives in new buffer area. (It doesn't modify old container annotations where it's not necessary, false negatives may happen in edge granules of the new container area.) I don't expect this function to be used with overlapping buffers, but it's designed to work with them and not result in incorrect ASan errors (false positives). If buffers have granularity-aligned distance between them (`old_beg % granularity == new_beg % granularity`), copying algorithm works faster. If the distance is not granularity-aligned, annotations are copied byte after byte. ```cpp void __sanitizer_copy_contiguous_container_annotations( const void *old_storage_beg_p, const void *old_storage_end_p, const void *new_storage_beg_p, const void *new_storage_end_p) { ``` This function aims to help with short string annotations and similar container annotations. Right now we change trait types of `std::basic_string` when compiling with ASan and this function purpose is reverting that change as soon as possible. https://github.com/llvm/llvm-project/blob/87f3407856e61a73798af4e41b28bc33b5bf4ce6/libcxx/include/string#L738-L751 The goal is to not change `__trivially_relocatable` when compiling with ASan. If this function is accepted and upstreamed, the next step is creating a function like `__memcpy_with_asan` moving memory with ASan. And then using this function instead of `__builtin__memcpy` while moving trivially relocatable objects. https://github.com/llvm/llvm-project/blob/11a6799740f824282650aa9ec249b55dcf1a8aae/libcxx/include/__memory/uninitialized_algorithms.h#L644-L646 --- I'm thinking if there is a good way to address fact that in a container the new buffer is usually bigger than the previous one. We may add two more arguments to the functions to address it (the beginning and the end of the whole buffer. Another potential change is removing `new_storage_end_p` as it's redundant, because we require the same size. Potential future work is creating a function `__asan_unsafe_memmove`, which will be basically memmove, but with turned off instrumentation (therefore it will allow copy data from poisoned area). --------- Co-authored-by: Vitaly Buka <vitalybuka@google.com>
This PR adds a
__sanitizer_copy_contiguous_container_annotations
function, which copies annotations from one memory area to another. New area is annotated in the same way as the old region at the beginning (within limitations of ASan).Overlapping case: The function supports overlapping containers, however no assumptions should be made outside of no false positives in new buffer area. (It doesn't modify old container annotations where it's not necessary, false negatives may happen in edge granules of the new container area.) I don't expect this function to be used with overlapping buffers, but it's designed to work with them and not result in incorrect ASan errors (false positives).
If buffers have granularity-aligned distance between them (
old_beg % granularity == new_beg % granularity
), copying algorithm works faster. If the distance is not granularity-aligned, annotations are copied byte after byte.This function aims to help with short string annotations and similar container annotations. Right now we change trait types of
std::basic_string
when compiling with ASan and this function purpose is reverting that change as soon as possible.llvm-project/libcxx/include/string
Lines 738 to 751 in 87f3407
The goal is to not change
__trivially_relocatable
when compiling with ASan. If this function is accepted and upstreamed, the next step is creating a function like__memcpy_with_asan
moving memory with ASan. And then using this function instead of__builtin__memcpy
while moving trivially relocatable objects.llvm-project/libcxx/include/__memory/uninitialized_algorithms.h
Lines 644 to 646 in 11a6799
I'm thinking if there is a good way to address fact that in a container the new buffer is usually bigger than the previous one. We may add two more arguments to the functions to address it (the beginning and the end of the whole buffer.
Another potential change is removing
new_storage_end_p
as it's redundant, because we require the same size.Potential future work is creating a function
__asan_unsafe_memmove
, which will be basically memmove, but with turned off instrumentation (therefore it will allow copy data from poisoned area).