Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Clang][Sema] placement new initializes typedef array with correct size #83124

Merged
merged 1 commit into from
Apr 16, 2024

Conversation

mahtohappy
Copy link
Contributor

@mahtohappy mahtohappy commented Feb 27, 2024

When in-place new-ing a local variable of an array of trivial type, the generated code calls 'memset' with the correct size of the array, earlier it was generating size (squared of the typedef array + size).

The cause: typedef TYPE TArray[8]; TArray x; The type of declarator is Tarray[8] and in SemaExprCXX.cpp::BuildCXXNew we check if it's of typedef and of constant size then we get the original type and it works fine for non-dependent cases.
But in case of template we do TreeTransform.h:TransformCXXNEWExpr and there we again check the allocated type which is TArray[8] and it stays that way, so ArraySize=(Tarray[8] type, alloc Tarray[8*type]) so the squared size allocation.

ArraySize gets calculated earlier in TreeTransform.h so that if(!ArraySize) condition was failing.
fix: I changed that condition to if(ArraySize).
fixes #41441

Copy link

Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be
notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write
permissions for the repository. In which case you can instead tag reviewers by
name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review
by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate
is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

@llvmbot llvmbot added clang Clang issues not falling into any other category clang:frontend Language frontend issues, e.g. anything involving "Sema" labels Feb 27, 2024
@llvmbot
Copy link
Member

llvmbot commented Feb 27, 2024

@llvm/pr-subscribers-backend-x86

@llvm/pr-subscribers-clang

Author: None (mahtohappy)

Changes

When in-place new-ing a local variable of an array of trivial type, the generated code calls 'memset' with the correct size of the array.
#fixes 41441


Full diff: https://github.com/llvm/llvm-project/pull/83124.diff

2 Files Affected:

  • (modified) clang/lib/Sema/TreeTransform.h (+1-1)
  • (added) clang/test/CodeGen/instantiate-new-placement-size.cpp (+19)
diff --git a/clang/lib/Sema/TreeTransform.h b/clang/lib/Sema/TreeTransform.h
index 7389a48fe56fcc..0a46deda194826 100644
--- a/clang/lib/Sema/TreeTransform.h
+++ b/clang/lib/Sema/TreeTransform.h
@@ -12731,7 +12731,7 @@ TreeTransform<Derived>::TransformCXXNewExpr(CXXNewExpr *E) {
   }
 
   QualType AllocType = AllocTypeInfo->getType();
-  if (!ArraySize) {
+  if (ArraySize) {
     // If no array size was specified, but the new expression was
     // instantiated with an array type (e.g., "new T" where T is
     // instantiated with "int[4]"), extract the outer bound from the
diff --git a/clang/test/CodeGen/instantiate-new-placement-size.cpp b/clang/test/CodeGen/instantiate-new-placement-size.cpp
new file mode 100644
index 00000000000000..eaf013a57a6176
--- /dev/null
+++ b/clang/test/CodeGen/instantiate-new-placement-size.cpp
@@ -0,0 +1,19 @@
+// RUN: %clang -S -emit-llvm -o - %s | FileCheck %s
+#include <new>
+
+// CHECK: call void @llvm.memset.p0.i64(ptr align 1 %x, i8 0, i64 8, i1 false)
+// CHECK: call void @llvm.memset.p0.i64(ptr align 16 %x, i8 0, i64 32, i1 false)
+template <typename TYPE>
+void f()
+{
+    typedef TYPE TArray[8];
+
+    TArray x;
+    new(&x) TArray();
+}
+
+int main()
+{
+    f<char>();
+    f<int>();
+}

@mahtohappy
Copy link
Contributor Author

It fixes this issue #41441
https://godbolt.org/z/8qoz3GM9h

@mahtohappy mahtohappy force-pushed the temp-placement-new branch 2 times, most recently from a4c9d01 to 4661010 Compare February 27, 2024 16:49

// CHECK: call void @llvm.memset.p0.i64(ptr align 1 %x, i8 0, i64 8, i1 false)
// CHECK: call void @llvm.memset.p0.i64(ptr align 16 %x, i8 0, i64 32, i1 false)
template <typename TYPE>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need more test cases to make sure we are not breaking other cases. Non-dependent Vs dependent, constant Vs dynamically sized, VLA etc

@shafik
Copy link
Collaborator

shafik commented Feb 27, 2024

Thank you for this fix.

Can you put more details in your summary. The approach I like to take is 1. what the problem is 2. what is the approach of the fix 3. any other important details.

@erichkeane
Copy link
Collaborator

I definitely agree with both of Shafik's comments!

The fix itself concerns me, the logic in the block that is having its condition inverted is specifically made for 'if no array size was specified' (see comment), so that makes me think this is an incorrect patch. It doesn't seem to follow the suggestion given in the original bug either.

@mahtohappy
Copy link
Contributor Author

I definitely agree with both of Shafik's comments!

The fix itself concerns me, the logic in the block that is having its condition inverted is specifically made for 'if no array size was specified' (see comment), so that makes me think this is an incorrect patch. It doesn't seem to follow the suggestion given in the original bug either.
I agree with that but this was simpler solution, and it didn't break any testcase so I pushed this solution.
I had a other solution where the check for typedef was separate condition and that was working as well. But this looked neat to me and didn't break anything so I pushed this one.

@mahtohappy
Copy link
Contributor Author

mahtohappy commented Feb 27, 2024

// If no array size was specified, but the new expression was
// instantiated with an array type (e.g., "new T" where T is
// instantiated with "int[4]"), extract the outer bound from the
// array type as our array size. We do this with constant and
// dependently-sized array types.
    typedef TYPE TArray[8];
    TArray x;

this is the same case. In TreeTransform.h we calculate the ArraySize earlier so that condition was failing.

@shafik
Copy link
Collaborator

shafik commented Feb 28, 2024

Can you explain which cases the if (!ArraySize) condition was catching before and why the change does not effect those cases? My fear is that we are breaking other cases but we don't have test coverage for those cases and we are missing those breaks.

@mahtohappy
Copy link
Contributor Author

@shafik I'm not aware of which cases it was catching before, I was hoping testing will reveal those testcases. I agree with your thinking that we might be breaking something and and don't have test coverage for it. I'll leave this condition as it is and add a separate check for this issue. It'll be more robust solution.

@mahtohappy mahtohappy force-pushed the temp-placement-new branch 3 times, most recently from 7de2ca7 to 991ca33 Compare February 28, 2024 14:08
@mahtohappy
Copy link
Contributor Author

Hi @shafik @cor3ntin @AaronBallman @erichkeane @Endilll Please review this. Thank you.

@@ -12669,6 +12669,19 @@ TreeTransform<Derived>::TransformCXXNewExpr(CXXNewExpr *E) {
ArraySize = NewArraySize.get();
}

// Per C++0x [expr.new]p5, the type being constructed may be a
// typedef of an array type.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment doesn't really match what is happening below? the 'if' is still based on 'ArraySize' (presumably you mean this to be a proxy for E->isArray?). But there is no standardeeze quoted here about why you'd pick up the size/alloc type from it like this, rather than in the E->isArray above.

Also, does this also handle cases where this is going to still be dependent?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right. It's proxy for E->isArray(), but the condition just before checks if it's Array and then calculates the ArraySize. I didn't want to check again and used the ArraySize calculated before. If the earlier condition E->isArray() fails then ArraySize won't be calculated and if(ArraySize) would also fail. In Typedef case Tarray x; the alloctype is Tarray for x, that's why using Tarray Element Type which is int, and using that as allocType. Yes, this is for dependent cases only.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it seems this should probably be inside of that E->isArray() block instead then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can be inside that block, but in the non-dependent it was outside the block to check for typedef so I followed the same thing here as well. It's in Sema::SemaExprCXX::BuildNEWCXX.

@mahtohappy
Copy link
Contributor Author

Ping @AaronBallman @cor3ntin @Endilll

@mahtohappy
Copy link
Contributor Author

Hi Please Review this. @Endilll @cor3ntin @AaronBallman

@cor3ntin
Copy link
Contributor

Can you:

  • add a changelog entry referencing the issue
  • rename the test file or reference the issue number in a comment in the test?

thanks!

@antmox
Copy link
Contributor

antmox commented Apr 16, 2024

hello, instantiate-new-placement-size.cpp also fails here :
https://lab.llvm.org/buildbot/#/builders/188/builds/44501

@mahtohappy
Copy link
Contributor Author

#88902 pushed a fix. Will update after build is done.

@dyung
Copy link
Collaborator

dyung commented Apr 16, 2024

Please either revert or get the fix committed. This test has been failing on bots for around 6 hours at this point.

erichkeane pushed a commit that referenced this pull request Apr 16, 2024
…ze (#88902)

Build Failure Fix
Fixes build failures due to #83124
@erichkeane
Copy link
Collaborator

I merged the fix, it looks good enough, and I trust @mahtohappy that it passed locally. I DID have 1 comment on the test.

It is unfortunate that our infrastructure CI takes so long, this was likely to have to wait another ~4-5 hrs for Windows CI.

erichkeane added a commit that referenced this pull request Apr 16, 2024
This just replaces an '#include<new>' with a declaration of array
placement new.
@Sterling-Augustine
Copy link
Contributor

Sterling-Augustine commented Apr 17, 2024

Unless the original test was subtly broken, the forward fix in 0a789ea seems erroneous.

The forward fix changes the test to have a different declaration of new. But I would not expect this original change to require source-code level changes. Or was that intended?

We are experiencing errors that look like this:

[foo.cc:1045] error: no matching member function for call to 'reset'
 1045 |   values_remaining_at_rank_of_width_.reset(new int64_t[ranks][65]);
      |   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~
[foo.cc:1784] note: in instantiation of member function '(anonymous namespace)::Writer<unsigned long>::InitStats' requested here
 1784 |   InitStats();
      |   ^
[foo.cc:2045] note: in instantiation of member function '(anonymous namespace)::Writer<unsigned long>::DoIt' requested here
 2045 |   writer.DoIt();
      |          ^
[.../toolchain/bin/../include/c++/v1/__memory/unique_ptr.h:487] note: candidate template ignored: requirement '_CheckArrayPointerConversion<long *>::value' was not satisfied [with _Pp = int64_t *]
  487 |   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX23 void reset(_Pp __p) _NOEXCEPT {
      |                                                            ^

metaflow added a commit that referenced this pull request Apr 17, 2024
This reverts commit 0a789ea.

Breaks builds, see discussion in #83124
metaflow added a commit that referenced this pull request Apr 17, 2024
…rrect size (#88902)"

This reverts commit 5c6af60.

Breaks builds, see discussion in #83124
metaflow added a commit that referenced this pull request Apr 17, 2024
…rrect size (#83124)"

This reverts commit c309dc6.

Breaks builds, see discussion in #83124
@metaflow
Copy link
Contributor

The problem is still present. I have reverted this chain of commits.

mahtohappy added a commit to mahtohappy/llvm-project that referenced this pull request Apr 17, 2024
…ze (llvm#83124)

When in-place new-ing a local variable of an array of trivial type, the
generated code calls 'memset' with the correct size of the array,
earlier it was generating size (squared of the typedef array + size).

The cause: `typedef TYPE TArray[8]; TArray x;` The type of declarator is
Tarray[8] and in `SemaExprCXX.cpp::BuildCXXNew` we check if it's of
typedef and of constant size then we get the original type and it works
fine for non-dependent cases.
But in case of template we do `TreeTransform.h:TransformCXXNEWExpr` and
there we again check the allocated type which is TArray[8] and it stays
that way, so ArraySize=(Tarray[8] type, alloc Tarray[8*type]) so the
squared size allocation.

ArraySize gets calculated earlier in `TreeTransform.h` so that
`if(!ArraySize)` condition was failing.
fix: I changed that condition to `if(ArraySize)`. 


Fixes llvm#41441
mahtohappy added a commit to mahtohappy/llvm-project that referenced this pull request Apr 17, 2024
mahtohappy pushed a commit to mahtohappy/llvm-project that referenced this pull request Apr 17, 2024
This just replaces an '#include<new>' with a declaration of array
placement new.
@erichkeane
Copy link
Collaborator

Unless the original test was subtly broken, the forward fix in 0a789ea seems erroneous.

The forward fix changes the test to have a different declaration of new. But I would not expect this original change to require source-code level changes. Or was that intended?

We are experiencing errors that look like this:

[foo.cc:1045] error: no matching member function for call to 'reset'
 1045 |   values_remaining_at_rank_of_width_.reset(new int64_t[ranks][65]);
      |   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~
[foo.cc:1784] note: in instantiation of member function '(anonymous namespace)::Writer<unsigned long>::InitStats' requested here
 1784 |   InitStats();
      |   ^
[foo.cc:2045] note: in instantiation of member function '(anonymous namespace)::Writer<unsigned long>::DoIt' requested here
 2045 |   writer.DoIt();
      |          ^
[.../toolchain/bin/../include/c++/v1/__memory/unique_ptr.h:487] note: candidate template ignored: requirement '_CheckArrayPointerConversion<long *>::value' was not satisfied [with _Pp = int64_t *]
  487 |   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX23 void reset(_Pp __p) _NOEXCEPT {
      |                                                            ^

That forward-fix was just a test change, doing a #include in our tests isn't something we can do, as it makes them fail with "file not found". As far as your errors, it shouldn't be because of those, but could definitely be a problem wiht the originnal patch, hopefully @mahtohappy can get a chance to look at this.

@mahtohappy
Copy link
Contributor Author

@Sterling-Augustine please provide a reproducer script for your errors.

@slackito
Copy link
Collaborator

The following program fails to build after this change:

#include <cstdint>
#include <memory>

template<typename T>
struct X {
  void f() {
    values_.reset(new int64_t[123][65]);
  }

  std::unique_ptr<int64_t[][65]> values_;
};

int main() {
  X<unsigned char> x;
  x.f();
}

you get similar errors with both libstdc++:

jgorbe@gorbe:~/llvm-build/bin$ ./clang++ -o /dev/null ~/repro.cc
/usr/local/google/home/jgorbe/repro.cc:7:13: error: no matching member function for call to 'reset'
    7 |     values_.reset(new int64_t[123][65]);
      |     ~~~~~~~~^~~~~
/usr/local/google/home/jgorbe/repro.cc:15:5: note: in instantiation of member function 'X<unsigned char>::f' requested here
   15 |   x.f();
      |     ^
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:779:7: note: candidate template ignored: requirement '__and_<std::__or_<std::is_same<long *, long (*)[65]>, std::__and_<std::is_same<long (*)[65], long (*)[65]>, std::is_pointer<long *>, std::is_convertible<long (*)[], long (*)[][65]>>>>::value' was not satisfied [with _Up = int64_t *]
  779 |       reset(_Up __p) noexcept
      |       ^
1 error generated.

and libc++

jgorbe@gorbe:~/llvm-build/bin$ ./clang++ -o /dev/null ~/repro.cc -stdlib=libc++
/usr/local/google/home/jgorbe/repro.cc:7:13: error: no matching member function for call to 'reset'
    7 |     values_.reset(new int64_t[123][65]);
      |     ~~~~~~~~^~~~~
/usr/local/google/home/jgorbe/repro.cc:15:5: note: in instantiation of member function 'X<unsigned char>::f' requested here
   15 |   x.f();
      |     ^
/usr/local/google/home/jgorbe/llvm-build/bin/../include/c++/v1/__memory/unique_ptr.h:457:60: note: candidate template ignored: requirement '_CheckArrayPointerConversion<long *>::value' was not satisfied [with _Pp = int64_t *]
  457 |   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX23 void reset(_Pp __p) _NOEXCEPT {
      |                                                            ^
1 error generated.

@dwblaikie
Copy link
Collaborator

Simplified a bit more:

template <typename T>
void f1() {
  int (*x)[1] = new int[1][1]; // fails
}
template void f1<char>();
void f2() {
  int (*x)[1] = new int[1][1]; // passes
}

https://godbolt.org/z/MhaYbvefG - yeah seems pretty clear this is a regression. Compiles with GCC, fails with clang, and the non-dependent context passes with either.

erichkeane added a commit that referenced this pull request Apr 23, 2024
…orrect size (#83124)" (#89036)

When in-place new-ing a local variable of an array of trivial type, the
generated code calls 'memset' with the correct size of the array,
earlier it was generating size (squared of the typedef array + size).

The cause: typedef TYPE TArray[8]; TArray x; The type of declarator is
Tarray[8] and in SemaExprCXX.cpp::BuildCXXNew we check if it's of
typedef and of constant size then we get the original type and it works
fine for non-dependent cases.
But in case of template we do TreeTransform.h:TransformCXXNEWExpr and
there we again check the allocated type which is TArray[8] and it stays
that way, so ArraySize=(Tarray[8] type, alloc Tarray[8*type]) so the
squared size allocation.

ArraySize gets calculated earlier in TreeTransform.h so that
if(!ArraySize) condition was failing.
fix: I changed that condition to if(ArraySize).
fixes #41441

---------

Co-authored-by: erichkeane <ekeane@nvidia.com>
@pranavk
Copy link
Contributor

pranavk commented Apr 23, 2024

This commit is still problematic. Minimal reproducer (https://godbolt.org/z/hE7M8EfT1).

We should get this reverted again.

@pranavk
Copy link
Contributor

pranavk commented Apr 23, 2024

In file included from /usr/local/foo/home/prka/wip/unique/test.cpp:1:
In file included from /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/memory:78:
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:1085:14: error: no matching conversion for functional-style cast from 'unsigned short *' to 'unique_ptr<unsigned short[][256]>'
 1085 |     { return unique_ptr<_Tp>(new remove_extent_t<_Tp>[__num]()); }
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/foo/home/prka/wip/unique/test.cpp:10:47: note: in instantiation of function template specialization 'std::make_unique<unsigned short[][256]>' requested here
   10 |   const std::unique_ptr<Node[]> nodes_ = std::make_unique<Node[]>(max_nodes_);
      |                                               ^
/usr/local/foo/home/prka/wip/unique/test.cpp:15:38: note: in instantiation of member function 'Base<256>::Base' requested here
   15 |   explicit Child(size_t max_bytes) : Base<256>(max_bytes) {
      |                                      ^
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:648:7: note: candidate constructor not viable: no known conversion from 'unsigned short *' to 'unique_ptr<unsigned short[][256]>' for 1st argument
  648 |       unique_ptr(unique_ptr&&) = default;
      |       ^          ~~~~~~~~~~~~
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:652:12: note: candidate constructor template not viable: no known conversion from 'unsigned short *' to 'nullptr_t' (aka 'std::nullptr_t') for 1st argument
  652 |         constexpr unique_ptr(nullptr_t) noexcept
      |                   ^          ~~~~~~~~~
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:796:7: note: candidate constructor not viable: no known conversion from 'unsigned short *' to 'const unique_ptr<unsigned short[][256]>' for 1st argument
  796 |       unique_ptr(const unique_ptr&) = delete;
      |       ^          ~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:603:2: note: candidate template ignored: requirement '__and_<std::__or_<std::__or_<std::is_same<unsigned short *, unsigned short (*)[256]>, std::is_same<unsigned short *, std::nullptr_t>>, std::__and_<std::is_pointer<unsigned short *>, std::is_same<unsigned short (*)[256], unsigned short (*)[256]>, std::is_convertible<unsigned short (*)[], unsigned short (*)[][256]>>>>::value' was not satisfied [with _Up = unsigned short *, _Vp = std::default_delete<unsigned short[][256]>, $2 = _DeleterConstraint<default_delete<unsigned short[][256]>>]
  603 |         unique_ptr(_Up __p) noexcept
      |         ^
/usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:662:2: note: candidate template ignored: could not match 'unique_ptr<_Up, _Ep>' against 'unsigned short *'
  662 |         unique_ptr(unique_ptr<_Up, _Ep>&& __u) noexcept

We get error messages like above with this commit. This compiles fine without this commit or with gcc.

pranavk added a commit that referenced this pull request Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend:X86 clang:frontend Language frontend issues, e.g. anything involving "Sema" clang Clang issues not falling into any other category
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Inplace new-ing an array overwrites the square of the memory