Do copy[_nonoverlapping]/swap[_nonoverlapping] do typed copies?

[gnzlbg]

The following example (godbolt):

#[repr(align(128))] #[derive(Copy, Clone)] pub struct A(u8);

pub unsafe fn foo(x: &A) -> A { 
    *x
}

pub unsafe fn bar(x: &A) -> A { 
    let mut y: A = std::mem::uninitialized();
    std::ptr::copy_nonoverlapping(
        x as *const A, &mut y as *mut A, 1
    );
    y
}

produces the following LLVM-IR:

%A = type { [0 x i8], i8, [127 x i8] }

define void @_ZN7example3foo17h89c067fa0b9a17bcE(%A* noalias nocapture sret dereferenceable(128), %A* noalias nocapture readonly align 128 dereferenceable(128) %x) unnamed_addr #0 {
  %1 = getelementptr inbounds %A, %A* %0, i64 0, i32 0, i64 0
  %2 = getelementptr inbounds %A, %A* %x, i64 0, i32 0, i64 0
  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 128 %1, i8* nonnull align 128 %2, i64 128, i1 false)
  ret void
}

define void @_ZN7example3bar17hd5d27715385ba486E(%A* noalias nocapture sret dereferenceable(128), %A* noalias nocapture readonly align 128 dereferenceable(128) %x) unnamed_addr #0 {
  %1 = getelementptr inbounds %A, %A* %x, i64 0, i32 0, i64 0
  %2 = getelementptr inbounds %A, %A* %0, i64 0, i32 0, i64 0
  call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 128 %2, i8* nonnull align 128 %1, i64 128, i1 false) #2
  ret void
}

and machine code:

example::foo:
        mov     rax, rdi
        movaps  xmm0, xmmword ptr [rsi + 112]
        movaps  xmmword ptr [rdi + 112], xmm0
        movaps  xmm0, xmmword ptr [rsi + 96]
        movaps  xmmword ptr [rdi + 96], xmm0
        movaps  xmm0, xmmword ptr [rsi + 80]
        movaps  xmmword ptr [rdi + 80], xmm0
        movaps  xmm0, xmmword ptr [rsi + 64]
        movaps  xmmword ptr [rdi + 64], xmm0
        movaps  xmm0, xmmword ptr [rsi]
        movaps  xmm1, xmmword ptr [rsi + 16]
        movaps  xmm2, xmmword ptr [rsi + 32]
        movaps  xmm3, xmmword ptr [rsi + 48]
        movaps  xmmword ptr [rdi + 48], xmm3
        movaps  xmmword ptr [rdi + 32], xmm2
        movaps  xmmword ptr [rdi + 16], xmm1
        movaps  xmmword ptr [rdi], xmm0
        ret

example::bar:
        mov     rax, rdi
        movaps  xmm0, xmmword ptr [rsi + 112]
        movaps  xmmword ptr [rdi + 112], xmm0
        movaps  xmm0, xmmword ptr [rsi + 96]
        movaps  xmmword ptr [rdi + 96], xmm0
        movaps  xmm0, xmmword ptr [rsi + 80]
        movaps  xmmword ptr [rdi + 80], xmm0
        movaps  xmm0, xmmword ptr [rsi + 64]
        movaps  xmmword ptr [rdi + 64], xmm0
        movaps  xmm0, xmmword ptr [rsi]
        movaps  xmm1, xmmword ptr [rsi + 16]
        movaps  xmm2, xmmword ptr [rsi + 32]
        movaps  xmm3, xmmword ptr [rsi + 48]
        movaps  xmmword ptr [rdi + 48], xmm3
        movaps  xmmword ptr [rdi + 32], xmm2
        movaps  xmmword ptr [rdi + 16], xmm1
        movaps  xmmword ptr [rdi], xmm0
        ret

where 128 bytes are copied every time a value of type A is moved/copied/read/...

However, one actually only has to copy a single byte, since all other bytes are trailing padding. The expected machine code is (godbolt):

example::foo:
        mov     al, byte ptr [rdi]
        ret

example::bar:
        mov     al, byte ptr [rdi]
        ret

cc @nikic @rkruppe

[hanna-kruppe]

The least gross and special-cased way I can think of achieving this would be extending llvm.memcpy (and related intrinsics) with some extra info about bytes which don't need to be copied, e.g. a bitset in metadata. It would discarded when lowering to a memcpy call but could hypothetically be used when expanding to a sequence of loads and stores inline. That's still a lot of implementation work for what appears to be a somewhat artificial case, though.

A slightly more hacky, and more limited, fix for this specific case would be to trim the size copied by the amount of trailing padding (in this case 128 -> 1) if we can see statically that it's a typed copy of a single element (intrinsics::copy* call with constant 1 as length, MIR Copy/Move). However, this might regress performance of e.g. (u8, u8, u8) because it would at first glance disallow copying it with a 32 bit load/store (similar issue exists with the proposed stride/size distinction).

[gnzlbg]

The least gross and special-cased way I can think of achieving this would be extending llvm.memcpy (and related intrinsics) with some extra info about bytes which don't need to be copied, e.g. a bitset in metadata.

Instead of extending memcpy, could the Rust front end generate single loads for each of the fields, and would LLVM be able to merge them as appropriate?

That is, for the OP, I'd expect Rust to emit an 8-bit load. For something like (u8, u16) I'd expect Rust to emit two loads (8-bit and 16-bit), and for (u8, u8, u8) I'd expect Rust to emit three 8-bit loads. For (u8, u16) and for (u8, u8, u8) I do however expect LLVM to merge the loads into a single 32-bit load (EDIT: if possible).

[hanna-kruppe]

I don't think that can work. Leaving aside the huge inefficiencies of emitting so many instructions and praying that LLVM merges them, if we just emit loads and stores for the non-padding bytes and don't say anything about the padding, then LLVM has no indication that it's even allowed to clobber those parts.

[nikic]

LLVM already supports specifying padding through !tbaa.struct metadata: https://llvm.org/docs/LangRef.html#tbaa-struct-metadata

We currently don't use it because we generally aren't interested in TBAA. I also haven't checked whether the padding specified there is actually taking into account.

[RalfJung]

I am not at all convinced that this is a legal optimization, and I think making it legal makes the operational semantics much more annoying than it should be.

copy_nonoverlapping says

Copies count * size_of::() bytes from src to dst

To me this looks like the description of an untyped, memcpy-like operation that just copies a bunch of bytes. I feel fairly strongly that behavior of copy_nonoverlapping should not depend on the type of the pointers.

If you had used y = *x, that would be a totally different situation.

[RalfJung]

In particular, the docs also say that copy_nonoverlapping is equivalent to memcpy, which in turn is documented as:

Copies count characters from the object pointed to by src to the object pointed to by dest. Both objects are interpreted as arrays of unsigned char.

Given this spec, I see no way to justify not copying padding bytes. As in, I think we have to copy all bytes to comply with the spec, and the proposed optimization is illegal.

[nikic]

@RalfJung Not for copy_nonoverlapping possibly, but we should be able to elide padding copies for implicit memcpy's, as in the foo example (copy of return value into sret).

[RalfJung]

Ah, yes, I agree foo can be optimized as described in the OP. This can be explained, for example, by my proposed definition for a "typed copy".

[gnzlbg]

@RalfJung

In particular, the docs also say that copy_nonoverlapping is equivalent to memcpy, which in turn is documented as:

https://rust.godbolt.org/z/Ka2VLG (A C call to memcpy does not need to call the memcpy function).

[gnzlbg]

If you had used y = *x, that would be a totally different situation.

I did use y = *x in my original comment, e.g., see fn foo.