From fdfa5c59c64e403122dd91e64fd0319be0d8be6b Mon Sep 17 00:00:00 2001 From: Josh Liebow-Feeser Date: Sun, 18 Oct 2026 01:49:20 +0000 Subject: [PATCH] Optimize DST `size_of_val`, `align_of_val`, and field projection codegen Previously, `rustc_codegen_ssa::size_of_val::size_and_align_of_dst` (and its counterpart in `rustc_codegen_cranelift`) used a naive recursive algorithm that simultaneously computed both size and alignment at every level of struct/tuple nesting. This caused several codegen inefficiencies: 1. Calling `size_of_val` or `align_of_val` computed both values and discarded half the work. In `Layout::for_value_raw` (used when dropping `Box`, `Rc`, and `Arc`), calling `size_of_val_raw` followed by `align_of_val_raw` emitted the entire vtable load and alignment-rounding sequence twice (#152773). 2. Nested slice-tailed DSTs emitted `O(n)` sequential alignment-rounding steps (`add`, `sub`, `neg`, `and`) instead of folding the layout hierarchy at compile time. 3. Zero-offset `align(1)` wrappers around `dyn Trait` (such as `ManuallyDrop`, `UnsafeCell`, and `Cell`) loaded `vtable.align` and rounded `vtable.size` up to `vtable.align` even though `vtable.size` is already guaranteed to be a multiple of `vtable.align`. 4. `PlaceRef::project_field` called `size_and_align_of_dst` and emitted dynamic alignment-rounding IR whenever projecting to any unsized field at a non-zero offset, even for slice-tailed DST fields whose alignment (and therefore field offset) is a compile-time constant. Replace `size_and_align_of_dst` with decoupled `size_of_dst`, `align_of_dst`, and `dst_field_offset` routines in `rustc_codegen_ssa` and `rustc_codegen_cranelift`: - Decouple `align_of_dst` from `size_of_dst`: - For slice-tailed DSTs (`[T]`, `str`), `layout.align.abi` already incorporates the element alignment alongside all outer fields, `#[repr(align)]`, and `#[repr(packed)]`, so `align_of_dst` returns it as a compile-time constant. - For `dyn Trait`-tailed DSTs, by distributivity of `max` and `min` on power-of-two alignments, the alignment of an arbitrarily nested wrapper chain collapses to `max(layout.align.abi, min(vtable.align, min_pack))`. When `min_pack <= layout.align.abi`, this is statically `layout.align.abi` with no vtable load. - Distill slice-tailed DSTs into `O(1)` parameters in `size_of_dst`: - Walk the wrapper chain from the `[T]`/`str` tail outward at compile time, folding steps into `outer_offset + align_to(base + offset, align)` via the identities `align_to(c + align_to(x, a1), a2) == align_to(align_to(c, a1) + x, a2)` (when `a1 <= a2`), `align_to(c + align_to(x, a1), a2) == align_to(c, a2) + align_to(x, a1)` (when `a2 < a1`), and divisibility by `elem_size`. - Emit at most one `mul nuw nsw` (omitted when `elem_size == 1`), one pre-mask `add nuw` (or `add nuw nsw` when no bitmask is needed, and omitted when the addend is 0), one bitwise `and` (omitted when `elem_size % align == 0`), and (only when an outer `#[repr(packed)]` struct places a dynamically-padded inner DST at an unaligned offset) one post-mask `add nuw nsw`. When `elem_size == 0`, fold the entire DST size to a compile-time constant. - Because DST sizes fit in `isize` and `isize::MAX + 1` is a multiple of every valid alignment, `base + offset + (align - 1) <= isize::MAX`. Emitting `unchecked_uadd` (`add nuw`) before the bitmask (and `unchecked_suadd` when no bitmask is needed) lets LLVM prove `size > 0` for DSTs with a non-zero header and eliminate the dead `size == 0` branch before `__rust_dealloc` in `Box::drop`, `Rc::drop`, and `Arc::drop`, while avoiding `nsw` before the mask so LLVM's `InstCombine` does not strip the sign bit from `-align` and regress compact sign-extended immediate masks (`andq $-8`) on x86_64. - Collapse `dyn Trait` wrapper chains in `size_of_dst`: - Stream steps from innermost to outermost, skipping inner `offset == 0 && align == 1` wrappers (`ManuallyDrop`, `UnsafeCell`, etc.) so `size_of_val` emits a single `vtable.size` load (a 2.18x speedup at `-O3`). - Merge adjacent wrappers when `prev.align <= cur.align` and either `cur.offset == 0 && cur.pack == prev.pack` or both steps have statically capped alignment (`pack <= align`). - When the innermost step has `offset == 1 && align == 1` (e.g. `struct S(u8, dyn Trait)`), `full_align` divides `vtable.align` and thus divides `vtable.size`, so `align_to(1 + vtable.size, full_align)` simplifies to `vtable.size + full_align` without a bitmask. - Unify unsized field offset calculation into `dst_field_offset`: - Return `None` (using the static `layout.fields.offset(i)`) for all slice-tailed DST fields and for `dyn Trait` fields whose static offset is already a multiple of `min_pack`. - When dynamic rounding is needed for a `dyn Trait` field, exploit the fact that `offset` is already aligned to `field.align.abi`: when `offset == field.align.abi`, `align_to(offset, effective_align)` is `effective_align` itself; when `offset > field.align.abi`, all bits below `field.align.abi` in `-offset` are already 0, so clamping `vtable.align` up to `field.align.abi` before computing `offset + (-offset & (align - 1))` is unnecessary. - Extend MIR `InstSimplify` (`simplify_size_or_align_of_val`): - Fold `align_of_val` on `str` and single-field unaligned/unpacked wrapper structs and 1-tuples around `[T]` or `str` (such as `Path`, `OsStr`, and `CStr`) to `align_of::()` in MIR (a 1.50x speedup on `Layout::for_value::` at `-O3`). Fixes rust-lang/rust#152773 Validation: ./x test --stage 1 tests/codegen-llvm tests/mir-opt tests/ui/packed tests/ui/extern tests/ui/dst tests/ui/layout Agent-Authored-By: AI agent acting on Josh Liebow-Feeser's behalf --- .../src/intrinsics/mod.rs | 4 +- .../rustc_codegen_cranelift/src/unsize.rs | 400 ++++++++---- .../src/value_and_place.rs | 41 +- .../rustc_codegen_ssa/src/mir/intrinsic.rs | 4 +- compiler/rustc_codegen_ssa/src/mir/place.rs | 137 +--- compiler/rustc_codegen_ssa/src/size_of_val.rs | 589 +++++++++++++----- .../rustc_mir_transform/src/instsimplify.rs | 32 +- tests/codegen-llvm/dst-offset.rs | 4 +- .../intrinsics/size_and_align_of_val.rs | 158 ++++- ...al_str.InstSimplify-after-simplifycfg.diff | 22 + ...rapper.InstSimplify-after-simplifycfg.diff | 22 + tests/mir-opt/instsimplify/align_of_slice.rs | 17 + tests/ui/dst/dst-nested-size-align-offset.rs | 225 +++++++ 13 files changed, 1228 insertions(+), 427 deletions(-) create mode 100644 tests/mir-opt/instsimplify/align_of_slice.of_val_str.InstSimplify-after-simplifycfg.diff create mode 100644 tests/mir-opt/instsimplify/align_of_slice.of_val_wrapper.InstSimplify-after-simplifycfg.diff create mode 100644 tests/ui/dst/dst-nested-size-align-offset.rs diff --git a/compiler/rustc_codegen_cranelift/src/intrinsics/mod.rs b/compiler/rustc_codegen_cranelift/src/intrinsics/mod.rs index 61e165ef272a5..83f100900ebb1 100644 --- a/compiler/rustc_codegen_cranelift/src/intrinsics/mod.rs +++ b/compiler/rustc_codegen_cranelift/src/intrinsics/mod.rs @@ -461,7 +461,7 @@ fn codegen_regular_intrinsic_call<'tcx>( } else { None }; - let (size, _align) = crate::unsize::size_and_align_of(fx, layout, meta); + let size = crate::unsize::size_of_dst(fx, layout, meta); ret.write_cvalue(fx, CValue::by_val(size, usize_layout)); } sym::align_of_val => { @@ -475,7 +475,7 @@ fn codegen_regular_intrinsic_call<'tcx>( } else { None }; - let (_size, align) = crate::unsize::size_and_align_of(fx, layout, meta); + let align = crate::unsize::align_of_dst(fx, layout, meta); ret.write_cvalue(fx, CValue::by_val(align, usize_layout)); } diff --git a/compiler/rustc_codegen_cranelift/src/unsize.rs b/compiler/rustc_codegen_cranelift/src/unsize.rs index 48fb0f6c7d4a8..bd488be979096 100644 --- a/compiler/rustc_codegen_cranelift/src/unsize.rs +++ b/compiler/rustc_codegen_cranelift/src/unsize.rs @@ -164,131 +164,307 @@ pub(crate) fn coerce_unsized_into<'tcx>( // Adapted from https://github.com/rust-lang/rust/blob/2a663555ddf36f6b041445894a8c175cd1bc718c/src/librustc_codegen_ssa/glue.rs -pub(crate) fn size_and_align_of<'tcx>( +use rustc_codegen_ssa::size_of_val::unsized_tail_and_min_pack; + +fn codegen_foreign_panic<'tcx>(fx: &mut FunctionCx<'_, '_, 'tcx>, ty: Ty<'tcx>) -> Value { + let trap_block = fx.bcx.create_block(); + let true_ = fx.bcx.ins().iconst(types::I8, 1); + let next_block = fx.bcx.create_block(); + fx.bcx.ins().brif(true_, trap_block, &[], next_block, &[]); + fx.bcx.seal_block(trap_block); + fx.bcx.seal_block(next_block); + fx.bcx.switch_to_block(trap_block); + + // `extern` type. We cannot compute the size or alignment, so panic. + let msg_str = with_no_visible_paths!({ + with_no_trimmed_paths!({ + format!("attempted to compute the size or alignment of extern type `{ty}`") + }) + }); + + codegen_panic_nounwind(fx, &msg_str, fx.mir.span); + + fx.bcx.switch_to_block(next_block); + + // This function does not return so we can now return whatever we want. + fx.bcx.ins().iconst(fx.pointer_type, 42) +} + +fn clamp_dyn_align( + fx: &mut FunctionCx<'_, '_, '_>, + mut dyn_align: Value, + static_align: u64, + min_pack: u64, +) -> Value { + if min_pack < u64::MAX { + let packed = fx.bcx.ins().iconst(fx.pointer_type, min_pack as i64); + let cmp = fx.bcx.ins().icmp(IntCC::UnsignedLessThan, dyn_align, packed); + dyn_align = fx.bcx.ins().select(cmp, dyn_align, packed); + } + if static_align > 1 { + let sized_align = fx.bcx.ins().iconst(fx.pointer_type, static_align as i64); + let cmp = fx.bcx.ins().icmp(IntCC::UnsignedGreaterThan, sized_align, dyn_align); + dyn_align = fx.bcx.ins().select(cmp, sized_align, dyn_align); + } + dyn_align +} + +pub(crate) fn dst_field_offset<'tcx>( + fx: &mut FunctionCx<'_, '_, 'tcx>, + parent: TyAndLayout<'tcx>, + field: TyAndLayout<'tcx>, + offset: Size, + meta: Option, +) -> Option { + if field.is_sized() || offset.bytes() == 0 { + return None; + } + + let (tail_ty, min_pack) = unsized_tail_and_min_pack(fx.tcx, fx.typing_env(), parent.ty); + match tail_ty.kind() { + ty::Slice(_) | ty::Str => None, + ty::Dynamic(..) if offset.bytes().is_multiple_of(min_pack) => None, + ty::Dynamic(..) => { + let vtable_align = crate::vtable::align_of_obj(fx, meta.unwrap()); + let field_static_align = field.align.bytes(); + let offset_bytes = offset.bytes(); + debug_assert!(offset_bytes.is_multiple_of(field_static_align)); + debug_assert!(field_static_align < min_pack); + if offset_bytes == field_static_align { + Some(clamp_dyn_align(fx, vtable_align, field_static_align, min_pack)) + } else { + let align = clamp_dyn_align(fx, vtable_align, 1, min_pack); + let align_minus_1 = fx.bcx.ins().iadd_imm_s(align, -1); + let mask = fx.bcx.ins().iconst(fx.pointer_type, offset_bytes.wrapping_neg() as i64); + let pad = fx.bcx.ins().band(align_minus_1, mask); + Some(fx.bcx.ins().iadd_imm_u(pad, offset_bytes as i64)) + } + } + ty::Foreign(_) => Some(codegen_foreign_panic(fx, tail_ty)), + _ => bug!("dst_field_offset: unexpected unsized tail {tail_ty:?}"), + } +} + +pub(crate) fn align_of_dst<'tcx>( fx: &mut FunctionCx<'_, '_, 'tcx>, layout: TyAndLayout<'tcx>, info: Option, -) -> (Value, Value) { +) -> Value { if layout.is_sized() { - return ( - fx.bcx.ins().iconst(fx.pointer_type, layout.size.bytes() as i64), - fx.bcx.ins().iconst(fx.pointer_type, layout.align.bytes() as i64), - ); + return fx.bcx.ins().iconst(fx.pointer_type, layout.align.bytes() as i64); } - let ty = layout.ty; - match ty.kind() { + let (tail_ty, min_pack) = unsized_tail_and_min_pack(fx.tcx, fx.typing_env(), layout.ty); + match tail_ty.kind() { + ty::Slice(_) | ty::Str => fx.bcx.ins().iconst(fx.pointer_type, layout.align.bytes() as i64), + ty::Foreign(_) => codegen_foreign_panic(fx, tail_ty), ty::Dynamic(..) => { - // load size/align from vtable - ( - crate::vtable::size_of_obj(fx, info.unwrap()), - crate::vtable::align_of_obj(fx, info.unwrap()), - ) - } - ty::Slice(_) | ty::Str => { - let unit = layout.field(fx, 0); - // The info in this case is the length of the str, so the size is that - // times the unit size. - ( - fx.bcx.ins().imul_imm_u(info.unwrap(), unit.size.bytes() as i64), - fx.bcx.ins().iconst(fx.pointer_type, unit.align.bytes() as i64), - ) - } - ty::Foreign(_) => { - let trap_block = fx.bcx.create_block(); - let true_ = fx.bcx.ins().iconst(types::I8, 1); - let next_block = fx.bcx.create_block(); - fx.bcx.ins().brif(true_, trap_block, &[], next_block, &[]); - fx.bcx.seal_block(trap_block); - fx.bcx.seal_block(next_block); - fx.bcx.switch_to_block(trap_block); - - // `extern` type. We cannot compute the size, so panic. - let msg_str = with_no_visible_paths!({ - with_no_trimmed_paths!({ - format!("attempted to compute the size or alignment of extern type `{ty}`") - }) - }); - - codegen_panic_nounwind(fx, &msg_str, fx.mir.span); - - fx.bcx.switch_to_block(next_block); - - // This function does not return so we can now return whatever we want. - let size = fx.bcx.ins().iconst(fx.pointer_type, 42); - let align = fx.bcx.ins().iconst(fx.pointer_type, 42); - (size, align) + let static_align = layout.align.bytes(); + if min_pack <= static_align { + return fx.bcx.ins().iconst(fx.pointer_type, static_align as i64); + } + let dyn_align = crate::vtable::align_of_obj(fx, info.unwrap()); + clamp_dyn_align(fx, dyn_align, static_align, min_pack) } - ty::Adt(..) | ty::Tuple(..) => { - // First get the size of all statically known fields. - // Don't use size_of because it also rounds up to alignment, which we - // want to avoid, as the unsized field's alignment could be smaller. - assert!(!layout.ty.is_simd()); - - let i = layout.fields.count() - 1; - let unsized_offset_unadjusted = layout.fields.offset(i).bytes(); - let unsized_offset_unadjusted = - fx.bcx.ins().iconst(fx.pointer_type, unsized_offset_unadjusted as i64); - let sized_align = layout.align.bytes(); - let sized_align = fx.bcx.ins().iconst(fx.pointer_type, sized_align as i64); - - // Recurse to get the size of the dynamically sized field (must be - // the last field). - let field_layout = layout.field(fx, i); - let (unsized_size, mut unsized_align) = size_and_align_of(fx, field_layout, info); - - // # First compute the dynamic alignment - - // For packed types, we need to cap the alignment. - if let ty::Adt(def, _) = ty.kind() { - if let Some(packed) = def.repr().pack { - if packed.bytes() == 1 { - // We know this will be capped to 1. - unsized_align = fx.bcx.ins().iconst(fx.pointer_type, 1); + _ => bug!("align_of_dst: {} (tail {tail_ty}) not supported", layout.ty), + } +} + +#[derive(Copy, Clone)] +struct DstChainStep { + offset: u64, + align: u64, + pack: u64, +} + +fn emit_aligned_static_size( + fx: &mut FunctionCx<'_, '_, '_>, + base: Value, + base_multiple: u64, + offset: u64, + align: u64, +) -> Value { + let (addend, mask_align) = if base_multiple.is_multiple_of(align) { + (offset.next_multiple_of(align), 1) + } else { + (offset + (align - 1), align) + }; + let sum = if addend == 0 { base } else { fx.bcx.ins().iadd_imm_u(base, addend as i64) }; + if mask_align == 1 { + sum + } else { + let mask = fx.bcx.ins().iconst(fx.pointer_type, mask_align.wrapping_neg() as i64); + fx.bcx.ins().band(sum, mask) + } +} + +fn emit_dyn_step( + fx: &mut FunctionCx<'_, '_, '_>, + cur_size: Value, + base_multiple: Option, + step: DstChainStep, + vtable_align: &mut Option, + vtable: Value, +) -> Value { + if step.pack <= step.align { + return emit_aligned_static_size( + fx, + cur_size, + base_multiple.unwrap_or(1), + step.offset, + step.align, + ); + } + + let raw_dyn_align = + *vtable_align.get_or_insert_with(|| crate::vtable::align_of_obj(fx, vtable)); + let full_align = clamp_dyn_align(fx, raw_dyn_align, step.align, step.pack); + + if base_multiple.is_none() && step.offset == 1 && step.align == 1 { + return fx.bcx.ins().iadd(cur_size, full_align); + } + + let sum = if step.offset == 0 { + let addend = fx.bcx.ins().iadd_imm_s(full_align, -1); + fx.bcx.ins().iadd(cur_size, addend) + } else if step.offset == 1 { + fx.bcx.ins().iadd(cur_size, full_align) + } else { + let s = fx.bcx.ins().iadd_imm_u(cur_size, (step.offset - 1) as i64); + fx.bcx.ins().iadd(s, full_align) + }; + let neg = fx.bcx.ins().ineg(full_align); + fx.bcx.ins().band(sum, neg) +} + +pub(crate) fn size_of_dst<'tcx>( + fx: &mut FunctionCx<'_, '_, 'tcx>, + layout: TyAndLayout<'tcx>, + info: Option, +) -> Value { + if layout.is_sized() { + return fx.bcx.ins().iconst(fx.pointer_type, layout.size.bytes() as i64); + } + + let mut steps = Vec::new(); + let mut cur_layout = layout; + loop { + let cur_ty = cur_layout.ty; + match cur_ty.kind() { + ty::Adt(..) | ty::Tuple(..) => { + assert!(!cur_ty.is_simd()); + let i = cur_layout.fields.count() - 1; + let offset = cur_layout.fields.offset(i).bytes(); + let align = cur_layout.align.bytes(); + let pack = match cur_ty.kind() { + ty::Adt(def, _) => def.repr().pack.map_or(u64::MAX, |p| p.bytes()), + _ => u64::MAX, + }; + steps.push(DstChainStep { offset, align, pack }); + cur_layout = cur_layout.field(fx, i); + } + ty::Pat(..) => { + cur_layout = cur_layout.field(fx, 0); + } + ty::Slice(_) | ty::Str => { + let unit = cur_layout.field(fx, 0); + let elem_size = unit.size.bytes(); + + if elem_size == 0 { + let size = steps + .iter() + .rev() + .fold(0u64, |size, step| (step.offset + size).next_multiple_of(step.align)); + return fx.bcx.ins().iconst(fx.pointer_type, size as i64); + } + + let len = info.unwrap(); + let base = if elem_size == 1 { + len + } else { + fx.bcx.ins().imul_imm_u(len, elem_size as i64) + }; + + let mut offset = 0u64; + let mut outer_offset = 0u64; + let mut align = 1u64; + + for step in steps.iter().rev() { + if align <= step.align { + offset += (outer_offset + step.offset).next_multiple_of(align); + outer_offset = 0; + align = step.align; + if elem_size.is_multiple_of(align) { + offset = offset.next_multiple_of(align); + align = 1; + } } else { - // We have to dynamically compute `min(unsized_align, packed)`. - let packed = fx.bcx.ins().iconst(fx.pointer_type, packed.bytes() as i64); - let cmp = fx.bcx.ins().icmp(IntCC::UnsignedLessThan, unsized_align, packed); - unsized_align = fx.bcx.ins().select(cmp, unsized_align, packed); + let c = (outer_offset + step.offset).next_multiple_of(step.align); + offset += c & !(align - 1); + outer_offset = c & (align - 1); } } + + let aligned = emit_aligned_static_size(fx, base, elem_size, offset, align); + return if outer_offset == 0 { + aligned + } else { + fx.bcx.ins().iadd_imm_u(aligned, outer_offset as i64) + }; } + ty::Dynamic(..) => { + let vtable = info.unwrap(); + let mut cur_size = crate::vtable::size_of_obj(fx, vtable); + let mut vtable_align: Option = None; + let mut base_multiple: Option = None; + let mut min_pack = u64::MAX; + let mut pending: Option = None; - // Choose max of two known alignments (combined value must - // be aligned according to more restrictive of the two). - let cmp = fx.bcx.ins().icmp(IntCC::UnsignedGreaterThan, sized_align, unsized_align); - let full_align = fx.bcx.ins().select(cmp, sized_align, unsized_align); - - // # Then compute the dynamic size - - // The full formula for the size would be: - // let unsized_offset_adjusted = unsized_offset_unadjusted.align_to(unsized_align); - // let full_size = (unsized_offset_adjusted + unsized_size).align_to(full_align); - // However, `unsized_size` is a multiple of `unsized_align`. - // Therefore, we can equivalently do the `align_to(unsized_align)` *after* adding `unsized_size`: - // let full_size = (unsized_offset_unadjusted + unsized_size).align_to(unsized_align).align_to(full_align); - // Furthermore, `align >= unsized_align`, and therefore we only need to do: - // let full_size = (unsized_offset_unadjusted + unsized_size).align_to(full_align); - - let full_size = fx.bcx.ins().iadd(unsized_offset_unadjusted, unsized_size); - - // Issue #27023: must add any necessary padding to `size` - // (to make it a multiple of `align`) before returning it. - // - // Namely, the returned size should be, in C notation: - // - // `size + ((size & (align-1)) ? align : 0)` - // - // emulated via the semi-standard fast bit trick: - // - // `(size + (align-1)) & -align` - let addend = fx.bcx.ins().iadd_imm_s(full_align, -1); - let add = fx.bcx.ins().iadd(full_size, addend); - let neg = fx.bcx.ins().ineg(full_align); - let full_size = fx.bcx.ins().band(add, neg); - - (full_size, full_align) + for step in steps.iter().rev() { + min_pack = min_pack.min(step.pack); + let cur = + DstChainStep { offset: step.offset, align: step.align, pack: min_pack }; + + let Some(prev) = &mut pending else { + if cur.offset == 0 && cur.align == 1 { + continue; + } + pending = Some(cur); + continue; + }; + + if prev.align <= cur.align + && ((cur.offset == 0 && cur.pack == prev.pack) + || (prev.pack <= prev.align && cur.pack <= cur.align)) + { + debug_assert!(cur.offset.is_multiple_of(prev.align)); + prev.offset += cur.offset; + prev.align = cur.align; + prev.pack = cur.pack; + continue; + } + + cur_size = emit_dyn_step( + fx, + cur_size, + base_multiple, + *prev, + &mut vtable_align, + vtable, + ); + base_multiple = Some(prev.align); + *prev = cur; + } + + if let Some(last) = pending { + cur_size = + emit_dyn_step(fx, cur_size, base_multiple, last, &mut vtable_align, vtable); + } + + return cur_size; + } + ty::Foreign(_) => return codegen_foreign_panic(fx, cur_ty), + _ => bug!("size_of_dst: {} (tail {cur_ty}) not supported", layout.ty), } - _ => bug!("size_and_align_of_dst: {ty} not supported"), } } diff --git a/compiler/rustc_codegen_cranelift/src/value_and_place.rs b/compiler/rustc_codegen_cranelift/src/value_and_place.rs index a4af73cbc3f20..31795b3086dc1 100644 --- a/compiler/rustc_codegen_cranelift/src/value_and_place.rs +++ b/compiler/rustc_codegen_cranelift/src/value_and_place.rs @@ -18,41 +18,14 @@ fn codegen_field<'tcx>( let field_offset = layout.fields.offset(field.index()); let field_layout = layout.field(&*fx, field.index()); - let simple = |fx: &mut FunctionCx<'_, '_, '_>| { - (base.offset_i64(fx, i64::try_from(field_offset.bytes()).unwrap()), field_layout) + let ptr = if let Some(dyn_offset) = + crate::unsize::dst_field_offset(fx, layout, field_layout, field_offset, extra) + { + base.offset_value(fx, dyn_offset) + } else { + base.offset_i64(fx, i64::try_from(field_offset.bytes()).unwrap()) }; - - if field_layout.is_sized() { - return simple(fx); - } - match field_layout.ty.kind() { - ty::Slice(..) | ty::Str => simple(fx), - _ => { - let unaligned_offset = field_offset.bytes(); - - // Get the alignment of the field - let (_, mut unsized_align) = crate::unsize::size_and_align_of(fx, field_layout, extra); - - // For packed types, we need to cap alignment. - if let ty::Adt(def, _) = layout.ty.kind() { - if let Some(packed) = def.repr().pack { - let packed = fx.bcx.ins().iconst(fx.pointer_type, packed.bytes() as i64); - let cmp = fx.bcx.ins().icmp(IntCC::UnsignedLessThan, unsized_align, packed); - unsized_align = fx.bcx.ins().select(cmp, unsized_align, packed); - } - } - - // Bump the unaligned offset up to the appropriate alignment - let one = fx.bcx.ins().iconst(fx.pointer_type, 1); - let align_sub_1 = fx.bcx.ins().isub(unsized_align, one); - let and_lhs = fx.bcx.ins().iadd_imm_u(align_sub_1, unaligned_offset as i64); - let zero = fx.bcx.ins().iconst(fx.pointer_type, 0); - let and_rhs = fx.bcx.ins().isub(zero, unsized_align); - let offset = fx.bcx.ins().band(and_lhs, and_rhs); - - (base.offset_value(fx, offset), field_layout) - } - } + (ptr, field_layout) } /// A read-only value diff --git a/compiler/rustc_codegen_ssa/src/mir/intrinsic.rs b/compiler/rustc_codegen_ssa/src/mir/intrinsic.rs index d930ac8e686da..98865c453395a 100644 --- a/compiler/rustc_codegen_ssa/src/mir/intrinsic.rs +++ b/compiler/rustc_codegen_ssa/src/mir/intrinsic.rs @@ -165,13 +165,13 @@ impl<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>> FunctionCx<'a, 'tcx, Bx> { sym::size_of_val => { let tp_ty = fn_args.type_at(0); let (_, meta) = args[0].val.pointer_parts(); - let (llsize, _) = size_of_val::size_and_align_of_dst(bx, tp_ty, meta, span); + let llsize = size_of_val::size_of_dst(bx, tp_ty, meta, span); OperandValue::Immediate(llsize) } sym::align_of_val => { let tp_ty = fn_args.type_at(0); let (_, meta) = args[0].val.pointer_parts(); - let (_, llalign) = size_of_val::size_and_align_of_dst(bx, tp_ty, meta, span); + let llalign = size_of_val::align_of_dst(bx, tp_ty, meta, span); OperandValue::Immediate(llalign) } sym::vtable_size | sym::vtable_align => { diff --git a/compiler/rustc_codegen_ssa/src/mir/place.rs b/compiler/rustc_codegen_ssa/src/mir/place.rs index ce166fbe59fd9..6d872aace9ec3 100644 --- a/compiler/rustc_codegen_ssa/src/mir/place.rs +++ b/compiler/rustc_codegen_ssa/src/mir/place.rs @@ -6,14 +6,13 @@ use rustc_abi::{ use rustc_middle::mir; use rustc_middle::mir::PlaceTy; use rustc_middle::mir::interpret::Scalar; +use rustc_middle::ty::Ty; use rustc_middle::ty::layout::{HasTyCtxt, HasTypingEnv, LayoutOf, TyAndLayout}; -use rustc_middle::ty::{self, Ty}; -use rustc_span::{DUMMY_SP, bug}; +use rustc_span::bug; use tracing::{debug, instrument}; use super::operand::OperandValue; use super::{FunctionCx, LocalRef}; -use crate::common::IntPredicate; use crate::size_of_val; use crate::traits::*; @@ -181,79 +180,29 @@ impl<'a, 'tcx, V: CodegenObject> PlaceRef<'tcx, V> { let offset = self.layout.fields.offset(ix); let effective_field_align = self.val.align.restrict_for_offset(offset); - // `simple` is called when we don't need to adjust the offset to - // the dynamic alignment of the field. - let mut simple = || { - let llval = if offset.bytes() == 0 { - self.val.llval - } else { - bx.inbounds_ptradd(self.val.llval, bx.const_usize(offset.bytes())) - }; - let val = PlaceValue { - llval, - llextra: if bx.cx().tcx().type_has_metadata(field.ty, bx.cx().typing_env()) { - self.val.llextra - } else { - None - }, - align: effective_field_align, - }; - val.with_type(field) - }; + let tcx = bx.cx().tcx(); + let typing_env = bx.cx().typing_env(); - // Simple cases, which don't need DST adjustment: - // * known alignment - sized types, `[T]`, `str` - // * offset 0 -- rounding up to alignment cannot change the offset - // Note that looking at `field.align` is incorrect since that is not necessarily equal - // to the dynamic alignment of the type. - match field.ty.kind() { - _ if field.is_sized() => return simple(), - ty::Slice(..) | ty::Str => return simple(), - _ if offset.bytes() == 0 => return simple(), - _ => {} - } - - // We need to get the pointer manually now. - // We do this by casting to a `*i8`, then offsetting it by the appropriate amount. - // We do this instead of, say, simply adjusting the pointer from the result of a GEP - // because the field may have an arbitrary alignment in the LLVM representation. - // - // To demonstrate: - // - // struct Foo { - // x: u16, - // y: T - // } - // - // The type `Foo>` is represented in LLVM as `{ u16, { u16, u8 }}`, meaning that - // the `y` field has 16-bit alignment. - - let meta = self.val.llextra; - - let unaligned_offset = bx.cx().const_usize(offset.bytes()); - - // Get the alignment of the field. No span is available here to blame a layout error on. - let (_, mut unsized_align) = - size_of_val::size_and_align_of_dst(bx, field.ty, meta, DUMMY_SP); - - // For packed types, we need to cap alignment. - if let ty::Adt(def, _) = self.layout.ty.kind() - && let Some(packed) = def.repr().pack + let llval = if let Some(dyn_offset) = + size_of_val::dst_field_offset(bx, self.layout, field, offset, self.val.llextra) { - let packed = bx.const_usize(packed.bytes()); - let cmp = bx.icmp(IntPredicate::IntULT, unsized_align, packed); - unsized_align = bx.select(cmp, unsized_align, packed) - } - - // Bump the unaligned offset up to the appropriate alignment - let offset = round_up_const_value_to_alignment(bx, unaligned_offset, unsized_align); - - debug!("struct_field_ptr: DST field offset: {:?}", offset); + debug!("struct_field_ptr: DST field offset: {:?}", dyn_offset); + bx.inbounds_ptradd(self.val.llval, dyn_offset) + } else if offset.bytes() == 0 { + self.val.llval + } else { + bx.inbounds_ptradd(self.val.llval, bx.const_usize(offset.bytes())) + }; - // Adjust pointer. - let ptr = bx.inbounds_ptradd(self.val.llval, offset); - let val = - PlaceValue { llval: ptr, llextra: self.val.llextra, align: effective_field_align }; + let val = PlaceValue { + llval, + llextra: if tcx.type_has_metadata(field.ty, typing_env) { + self.val.llextra + } else { + None + }, + align: effective_field_align, + }; val.with_type(field) } @@ -428,48 +377,6 @@ impl<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>> FunctionCx<'a, 'tcx, Bx> { } } -fn round_up_const_value_to_alignment<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( - bx: &mut Bx, - value: Bx::Value, - align: Bx::Value, -) -> Bx::Value { - // In pseudo code: - // - // if value & (align - 1) == 0 { - // value - // } else { - // (value & !(align - 1)) + align - // } - // - // Usually this is written without branches as - // - // (value + align - 1) & !(align - 1) - // - // But this formula cannot take advantage of constant `value`. E.g. if `value` is known - // at compile time to be `1`, this expression should be optimized to `align`. However, - // optimization only holds if `align` is a power of two. Since the optimizer doesn't know - // that `align` is a power of two, it cannot perform this optimization. - // - // Instead we use - // - // value + (-value & (align - 1)) - // - // Since `align` is used only once, the expression can be optimized. For `value = 0` - // its optimized to `0` even in debug mode. - // - // NB: The previous version of this code used - // - // (value + align - 1) & -align - // - // Even though `-align == !(align - 1)`, LLVM failed to optimize this even for - // `value = 0`. Bug report: https://bugs.llvm.org/show_bug.cgi?id=48559 - let one = bx.const_usize(1); - let align_minus_1 = bx.sub(align, one); - let neg_value = bx.neg(value); - let offset = bx.and(neg_value, align_minus_1); - bx.add(value, offset) -} - /// Calculates the value that needs to be stored to mark the discriminant. /// /// This might be `None` for a `struct` or a niched variant (like `Some(&3)`). diff --git a/compiler/rustc_codegen_ssa/src/size_of_val.rs b/compiler/rustc_codegen_ssa/src/size_of_val.rs index 2b8427c159ba4..e40e4611796bd 100644 --- a/compiler/rustc_codegen_ssa/src/size_of_val.rs +++ b/compiler/rustc_codegen_ssa/src/size_of_val.rs @@ -1,185 +1,460 @@ -//! Computing the size and alignment of a value. +//! Computing the size and alignment of a dynamically-sized value. +//! +//! For slice-tailed DSTs (`[T]`, `str`, and arbitrarily nested structs/tuples wrapping them, +//! including `#[repr(align)]` and `#[repr(packed)]`), we distill the entire nesting hierarchy at +//! compile time into an $O(1)$ parameter representation `outer_offset + align_to(base + offset, align)` +//! and emit at most a single multiplication, pre-mask addition, alignment mask (only when +//! `elem_size % align != 0`), and (only when an outer `#[repr(packed)]` struct places a +//! dynamically-padded inner DST at an unaligned offset) post-mask addition. +//! +//! For trait-object-tailed DSTs (`dyn Trait`), we collapse no-op inner wrappers (such as +//! `ManuallyDrop` and `UnsafeCell`), merge adjacent compatible wrappers, and +//! fold statically capped `#[repr(packed)]` wrappers so that only necessary vtable loads and +//! alignment masks are emitted. -use rustc_abi::{Align, WrappingRange}; +use rustc_abi::{Align, Size, WrappingRange}; use rustc_attr_ir::lang_items::LangItem; +use rustc_middle::ty::layout::TyAndLayout; use rustc_middle::ty::print::{with_no_trimmed_paths, with_no_visible_paths}; -use rustc_middle::ty::{self, Ty}; +use rustc_middle::ty::{self, Ty, TyCtxt}; use rustc_span::{DUMMY_SP, Span, bug}; +use smallvec::SmallVec; use tracing::{debug, trace}; use crate::common::IntPredicate; use crate::traits::*; use crate::{common, meth}; -pub fn size_and_align_of_dst<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( +fn codegen_foreign_panic<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>(bx: &mut Bx, t: Ty<'tcx>) { + // `extern` type. We cannot compute the size or alignment, so panic. + let msg_str = with_no_visible_paths!({ + with_no_trimmed_paths!({ + format!("attempted to compute the size or alignment of extern type `{t}`") + }) + }); + let msg = bx.const_str(&msg_str); + + // Obtain the panic entry point. + let (fn_abi, llfn, _instance) = common::build_langcall(bx, DUMMY_SP, LangItem::PanicNounwind); + + // Generate the call. Cannot use `do_call` since we don't have a MIR terminator so we + // can't create a `TerminationCodegenHelper`. (But we are in good company, this code is + // duplicated plenty of times.) + let fn_ty = bx.fn_decl_backend_type(fn_abi); + + bx.call( + fn_ty, + /* fn_attrs */ None, + Some(fn_abi), + llfn, + ReturnSlot::Direct, // we know the ABI here + &[msg.0, msg.1], + None, + None, + ); +} + +/// Walks from an unsized type `t` down to its unsized tail (`Slice`, `Str`, `Dynamic`, or +/// `Foreign`), returning `(tail_ty, min_pack)` where `min_pack` is the minimum `#[repr(packed)]` +/// alignment cap in bytes encountered along the struct chain (`u64::MAX` if uncapped). +pub fn unsized_tail_and_min_pack<'tcx>( + tcx: TyCtxt<'tcx>, + typing_env: ty::TypingEnv<'tcx>, + mut t: Ty<'tcx>, +) -> (Ty<'tcx>, u64) { + let mut min_pack = u64::MAX; + loop { + match *t.kind() { + ty::Adt(def, args) if def.is_struct() => { + if let Some(pack) = def.repr().pack { + min_pack = min_pack.min(pack.bytes()); + } + match def.non_enum_variant().tail_opt() { + Some(field) => { + t = tcx.normalize_erasing_regions(typing_env, field.ty(tcx, args)); + } + None => break, + } + } + ty::Tuple(tys) if let Some(&last_ty) = tys.last() => { + t = last_ty; + } + ty::Pat(inner, _) => { + t = inner; + } + _ => break, + } + } + (t, min_pack) +} + +fn load_vtable_align<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( + bx: &mut Bx, + vtable: Bx::Value, +) -> Bx::Value { + let align = + meth::VirtualIndex::from_index(ty::COMMON_VTABLE_ENTRIES_ALIGN).get_usize(bx, vtable); + // Alignment is always a power of two, thus 1..=0x800…000, + // but also bounded by the maximum we support in type layout. + let align_bound = Align::max_for_target(bx.data_layout()).bytes().into(); + bx.range_metadata(align, WrappingRange { start: 1, end: align_bound }); + align +} + +/// Computes `max(static_align, min(dyn_align, min_pack))`. +fn clamp_dyn_align<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( + bx: &mut Bx, + mut dyn_align: Bx::Value, + static_align: u64, + min_pack: u64, +) -> Bx::Value { + if min_pack < u64::MAX { + let packed = bx.const_usize(min_pack); + let cmp = bx.icmp(IntPredicate::IntULT, dyn_align, packed); + dyn_align = bx.select(cmp, dyn_align, packed); + } + if static_align > 1 { + let sized_align = bx.const_usize(static_align); + let cmp = bx.icmp(IntPredicate::IntUGT, sized_align, dyn_align); + dyn_align = bx.select(cmp, sized_align, dyn_align); + } + dyn_align +} + +/// If projecting to `field` at static `offset` inside `parent` requires a dynamic alignment +/// adjustment (or panics on an `extern type` tail), emits and returns `Some(dynamic_offset)`. +/// Returns `None` when the field sits at its static `offset` (`0` or `offset.bytes()`). +pub fn dst_field_offset<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( + bx: &mut Bx, + parent: TyAndLayout<'tcx>, + field: TyAndLayout<'tcx>, + offset: Size, + meta: Option, +) -> Option { + if field.is_sized() || offset.bytes() == 0 { + return None; + } + + // Only the unsized tail field of `parent` can be unsized, so walking from `parent.ty` + // accumulates `parent`'s `#[repr(packed)]` cap alongside all inner wrapper caps. + let (tail_ty, min_pack) = unsized_tail_and_min_pack(bx.tcx(), bx.typing_env(), parent.ty); + match tail_ty.kind() { + ty::Slice(_) | ty::Str => None, + ty::Dynamic(..) if offset.bytes().is_multiple_of(min_pack) => None, + ty::Dynamic(..) => { + // `offset` is aligned to `min(field.align, parent_pack)`. Since `min_pack <= parent_pack` + // and `offset` is not a multiple of `min_pack`, we must have `field.align < min_pack <= parent_pack`. + let vtable_align = load_vtable_align(bx, meta.unwrap()); + let field_static_align = field.align.bytes(); + let offset_bytes = offset.bytes(); + debug_assert!(offset_bytes.is_multiple_of(field_static_align)); + debug_assert!(field_static_align < min_pack); + if offset_bytes == field_static_align { + // Since `0 < offset_bytes == field_static_align <= effective_align`, rounding + // `offset_bytes` up to `effective_align` is simply `effective_align` itself. + Some(clamp_dyn_align(bx, vtable_align, field_static_align, min_pack)) + } else { + // `offset_bytes` is already a multiple of `field_static_align`, so all bits below + // `field_static_align` in `-offset_bytes` are 0. Thus `max(field_static_align, ..)` + // is unnecessary when computing `offset + (-offset & (align - 1))`. + let align = clamp_dyn_align(bx, vtable_align, 1, min_pack); + let align_minus_1 = bx.sub(align, bx.const_usize(1)); + let neg_offset = bx.const_int(bx.type_isize(), offset_bytes.wrapping_neg() as i64); + let pad = bx.and(neg_offset, align_minus_1); + Some(bx.add(bx.const_usize(offset_bytes), pad)) + } + } + ty::Foreign(_) => { + codegen_foreign_panic(bx, tail_ty); + Some(bx.const_usize(offset.bytes())) + } + _ => bug!("dst_field_offset: unexpected unsized tail {tail_ty:?}"), + } +} + +/// Computes the alignment of a (possibly unsized) type `t`. +pub fn align_of_dst<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( bx: &mut Bx, t: Ty<'tcx>, info: Option, span: Span, -) -> (Bx::Value, Bx::Value) { +) -> Bx::Value { let layout = bx.spanned_layout_of(t, span); - trace!("size_and_align_of_dst(ty={}, info={:?}): layout: {:?}", t, info, layout); + trace!("align_of_dst(ty={}, info={:?}): layout: {:?}", t, info, layout); if layout.is_sized() { - let size = bx.const_usize(layout.size.bytes()); - let align = bx.const_usize(layout.align.bytes()); - return (size, align); + return bx.const_usize(layout.align.bytes()); } - match t.kind() { - ty::Dynamic(..) => { - // Load size/align from vtable. - let vtable = info.unwrap(); - let size = meth::VirtualIndex::from_index(ty::COMMON_VTABLE_ENTRIES_SIZE) - .get_usize(bx, vtable); - let align = meth::VirtualIndex::from_index(ty::COMMON_VTABLE_ENTRIES_ALIGN) - .get_usize(bx, vtable); - - // Size is always <= isize::MAX. - let size_bound = bx.data_layout().ptr_sized_integer().signed_max() as u128; - bx.range_metadata(size, WrappingRange { start: 0, end: size_bound }); - // Alignment is always a power of two, thus 1..=0x800…000, - // but also bounded by the maximum we support in type layout. - let align_bound = Align::max_for_target(bx.data_layout()).bytes().into(); - bx.range_metadata(align, WrappingRange { start: 1, end: align_bound }); - - (size, align) - } + + let (tail_ty, min_pack) = unsized_tail_and_min_pack(bx.tcx(), bx.typing_env(), t); + match tail_ty.kind() { ty::Slice(_) | ty::Str => { - let unit = layout.field(bx, 0); - // The info in this case is the length of the str, so the size is that - // times the unit size. - ( - // All slice sizes must fit into `isize`, so this multiplication cannot - // wrap -- neither signed nor unsigned. - bx.unchecked_sumul(info.unwrap(), bx.const_usize(unit.size.bytes())), - bx.const_usize(unit.align.bytes()), - ) + // The static `layout.align.abi` of any slice-tailed DST already incorporates the + // element alignment as well as all outer fields, `#[repr(align)]`, and `#[repr(packed)]`. + bx.const_usize(layout.align.bytes()) } ty::Foreign(_) => { - // `extern` type. We cannot compute the size, so panic. - let msg_str = with_no_visible_paths!({ - with_no_trimmed_paths!({ - format!("attempted to compute the size or alignment of extern type `{t}`") - }) - }); - let msg = bx.const_str(&msg_str); - - // Obtain the panic entry point. - let (fn_abi, llfn, _instance) = - common::build_langcall(bx, DUMMY_SP, LangItem::PanicNounwind); - - // Generate the call. Cannot use `do_call` since we don't have a MIR terminator so we - // can't create a `TerminationCodegenHelper`. (But we are in good company, this code is - // duplicated plenty of times.) - let fn_ty = bx.fn_decl_backend_type(fn_abi); - - bx.call( - fn_ty, - /* fn_attrs */ None, - Some(fn_abi), - llfn, - ReturnSlot::Direct, // we know the ABI here - &[msg.0, msg.1], - None, - None, - ); - - // This function does not return so we can now return whatever we want. - let size = bx.const_usize(layout.size.bytes()); - let align = bx.const_usize(layout.align.bytes()); - (size, align) + codegen_foreign_panic(bx, tail_ty); + bx.const_usize(layout.align.bytes()) } - ty::Adt(..) | ty::Tuple(..) => { - // First get the size of all statically known fields. - // Don't use size_of because it also rounds up to alignment, which we - // want to avoid, as the unsized field's alignment could be smaller. - assert!(!t.is_simd()); - debug!("DST {} layout: {:?}", t, layout); - - let i = layout.fields.count() - 1; - let unsized_offset_unadjusted = layout.fields.offset(i).bytes(); - let sized_align = layout.align.bytes(); - debug!( - "DST {} offset of dyn field: {}, statically sized align: {}", - t, unsized_offset_unadjusted, sized_align - ); - let unsized_offset_unadjusted = bx.const_usize(unsized_offset_unadjusted); - let sized_align = bx.const_usize(sized_align); - - // Recurse to get the size of the dynamically sized field (must be - // the last field). - let field_ty = layout.field(bx, i).ty; - let (unsized_size, mut unsized_align) = size_and_align_of_dst(bx, field_ty, info, span); - - // # First compute the dynamic alignment - - // For packed types, we need to cap the alignment. - if let ty::Adt(def, _) = t.kind() - && let Some(packed) = def.repr().pack - { - if packed.bytes() == 1 { - // We know this will be capped to 1. - unsized_align = bx.const_usize(1); - } else { - // We have to dynamically compute `min(unsized_align, packed)`. - let packed = bx.const_usize(packed.bytes()); - let cmp = bx.icmp(IntPredicate::IntULT, unsized_align, packed); - unsized_align = bx.select(cmp, unsized_align, packed); - } + ty::Dynamic(..) => { + // By distributivity of lattice operations (`max` and `min`) on power-of-two alignments, + // the alignment of a chain of structs/tuples wrapping `dyn Trait` collapses to: + // `max(layout.align.abi, min(vtable.align, min_pack))`. + // In particular, if `min_pack <= layout.align.abi`, the dynamic term cannot exceed + // `layout.align.abi`, so the alignment is statically `layout.align.abi`. + let static_align = layout.align.bytes(); + if min_pack <= static_align { + return bx.const_usize(static_align); + } + let dyn_align = load_vtable_align(bx, info.unwrap()); + clamp_dyn_align(bx, dyn_align, static_align, min_pack) + } + _ => bug!("align_of_dst: {t} (tail {tail_ty}) not supported"), + } +} + +#[derive(Copy, Clone)] +struct DstChainStep { + offset: u64, + align: u64, + pack: u64, +} + +/// Computes `align_to(base + offset, align)` where `base` is known to be a multiple of +/// `base_multiple` and the final aligned result is guaranteed to fit in `isize`. +fn emit_aligned_static_size<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( + bx: &mut Bx, + base: Bx::Value, + base_multiple: u64, + offset: u64, + align: u64, +) -> Bx::Value { + if base_multiple.is_multiple_of(align) { + // Since `base` is a multiple of `align`, `align_to(base + offset, align)` is simply + // `base + offset.next_multiple_of(align)` with no bitmask needed. + let addend = offset.next_multiple_of(align); + return if addend == 0 { base } else { bx.unchecked_suadd(base, bx.const_usize(addend)) }; + } + + // `align_to(base + offset, align) == (base + offset + (align - 1)) & -align`. + // Because the aligned result is `<= isize::MAX` and `isize::MAX + 1` is a multiple of + // every valid power-of-two `align`, the unmasked sum `base + offset + (align - 1)` is + // also `<= isize::MAX` and cannot wrap. We emit `unchecked_uadd` (`add nuw`) rather than + // `unchecked_suadd` (`add nuw nsw`) before the bitmask: if `nsw` were present and `base` + // had a known-zero sign bit (e.g. from `mul nuw nsw`), LLVM's `InstCombine` would strip + // the sign bit from `-align` (turning e.g. `-8` into `0x7fff_ffff_ffff_fff8`), preventing + // compact sign-extended immediate encoding (`andq $-8`) when `SelectionDAG` cannot + // re-prove `SignBitIsZero`. `add nuw` still allows LLVM's `ConstantRange` analysis to + // prove that the masked result is non-zero whenever `offset > 0` (eliminating dead + // `size == 0` checks in `Box`/`Rc`/`Arc` deallocation). + let addend = offset + (align - 1); + let sum = bx.unchecked_uadd(base, bx.const_usize(addend)); + let mask = bx.const_int(bx.type_isize(), align.wrapping_neg() as i64); + bx.and(sum, mask) +} + +fn emit_dyn_step<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( + bx: &mut Bx, + cur_size: Bx::Value, + base_multiple: Option, + step: DstChainStep, + vtable_align: &mut Option, + vtable: Bx::Value, +) -> Bx::Value { + if step.pack <= step.align { + // Alignment at this step is the compile-time constant `step.align`. + return emit_aligned_static_size( + bx, + cur_size, + base_multiple.unwrap_or(1), + step.offset, + step.align, + ); + } + + let raw_dyn_align = *vtable_align.get_or_insert_with(|| load_vtable_align(bx, vtable)); + let full_align = clamp_dyn_align(bx, raw_dyn_align, step.align, step.pack); + + // At the innermost step (`base_multiple.is_none()`), `cur_size == vtable_size`. When + // `step.align == 1`, `full_align` divides `vtable_align` and therefore divides `vtable_size`. + // If `step.offset == 1`, `align_to(1 + vtable_size, full_align)` is simply `vtable_size + full_align`. + if base_multiple.is_none() && step.offset == 1 && step.align == 1 { + return bx.unchecked_suadd(cur_size, full_align); + } + + let sum = if step.offset == 0 { + let addend = bx.sub(full_align, bx.const_usize(1)); + bx.unchecked_uadd(cur_size, addend) + } else if step.offset == 1 { + bx.unchecked_uadd(cur_size, full_align) + } else { + let s = bx.unchecked_uadd(cur_size, bx.const_usize(step.offset - 1)); + bx.unchecked_uadd(s, full_align) + }; + let neg = bx.neg(full_align); + bx.and(sum, neg) +} + +fn emit_dyn_dst_size<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( + bx: &mut Bx, + vtable: Bx::Value, + steps: &[DstChainStep], +) -> Bx::Value { + let mut cur_size = + meth::VirtualIndex::from_index(ty::COMMON_VTABLE_ENTRIES_SIZE).get_usize(bx, vtable); + // Size is always <= isize::MAX. + let size_bound = bx.data_layout().ptr_sized_integer().signed_max() as u128; + bx.range_metadata(cur_size, WrappingRange { start: 0, end: size_bound }); + + let mut vtable_align: Option = None; + let mut base_multiple: Option = None; + let mut min_pack = u64::MAX; + let mut pending: Option = None; + + // Walk from innermost wrapper (`steps.iter().rev()`) to outermost (`steps[0]`), collapsing + // no-op and adjacent compatible steps before emitting IR. + for step in steps.iter().rev() { + min_pack = min_pack.min(step.pack); + let cur = DstChainStep { offset: step.offset, align: step.align, pack: min_pack }; + + let Some(prev) = &mut pending else { + // While `pending` is `None`, the current size is `vtable.size`, which is already a + // multiple of `vtable.align` (and thus also of `min(vtable.align, p)` for any `p`). + // Any inner wrapper with `offset == 0` and `align == 1` (e.g., `ManuallyDrop`, + // `UnsafeCell`, `Cell`, `Pin`) is a complete no-op on `vtable.size`. + if cur.offset == 0 && cur.align == 1 { + continue; + } + pending = Some(cur); + continue; + }; + + // Whenever `prev.align <= cur.align`, `cur` cannot have `pack < prev.align`, so `cur.offset` + // is guaranteed by layout calculation to be a multiple of `prev.align`. We can merge `prev` + // and `cur` into a single step when either: + // - `cur.offset == 0 && cur.pack == prev.pack` (`cur`'s dynamic alignment is a multiple of + // `prev`'s for all `vtable.align`), or + // - `prev.pack <= prev.align && cur.pack <= cur.align` (both alignments are static constants). + if prev.align <= cur.align + && ((cur.offset == 0 && cur.pack == prev.pack) + || (prev.pack <= prev.align && cur.pack <= cur.align)) + { + debug_assert!(cur.offset.is_multiple_of(prev.align)); + prev.offset += cur.offset; + prev.align = cur.align; + prev.pack = cur.pack; + continue; + } + + cur_size = emit_dyn_step(bx, cur_size, base_multiple, *prev, &mut vtable_align, vtable); + base_multiple = Some(prev.align); + *prev = cur; + } + + if let Some(last) = pending { + cur_size = emit_dyn_step(bx, cur_size, base_multiple, last, &mut vtable_align, vtable); + } + + cur_size +} + +/// Computes the size of a (possibly unsized) type `t`. +pub fn size_of_dst<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>( + bx: &mut Bx, + t: Ty<'tcx>, + info: Option, + span: Span, +) -> Bx::Value { + let layout = bx.spanned_layout_of(t, span); + trace!("size_of_dst(ty={}, info={:?}): layout: {:?}", t, info, layout); + if layout.is_sized() { + return bx.const_usize(layout.size.bytes()); + } + + let mut steps = SmallVec::<[DstChainStep; 4]>::new(); + let mut cur_layout = layout; + loop { + let cur_ty = cur_layout.ty; + match cur_ty.kind() { + ty::Adt(..) | ty::Tuple(..) => { + assert!(!cur_ty.is_simd()); + debug!("DST {} layout: {:?}", cur_ty, cur_layout); + + let i = cur_layout.fields.count() - 1; + let offset = cur_layout.fields.offset(i).bytes(); + let align = cur_layout.align.bytes(); + let pack = match cur_ty.kind() { + ty::Adt(def, _) => def.repr().pack.map_or(u64::MAX, |p| p.bytes()), + _ => u64::MAX, + }; + debug!( + "DST {} offset of dyn field: {}, statically sized align: {}, pack: {}", + cur_ty, offset, align, pack + ); + steps.push(DstChainStep { offset, align, pack }); + cur_layout = cur_layout.field(bx, i); + } + ty::Pat(..) => { + cur_layout = cur_layout.field(bx, 0); } + ty::Slice(_) | ty::Str => { + let unit = cur_layout.field(bx, 0); + let elem_size = unit.size.bytes(); - // Choose max of two known alignments (combined value must - // be aligned according to more restrictive of the two). - let full_align = match ( - bx.const_to_opt_u128(sized_align, false), - bx.const_to_opt_u128(unsized_align, false), - ) { - (Some(sized_align), Some(unsized_align)) => { - // If both alignments are constant, (the sized_align should always be), then - // pick the correct alignment statically. - bx.const_usize(std::cmp::max(sized_align, unsized_align) as u64) + if elem_size == 0 { + let size = steps + .iter() + .rev() + .fold(0u64, |size, step| (step.offset + size).next_multiple_of(step.align)); + return bx.const_usize(size); } - _ => { - let cmp = bx.icmp(IntPredicate::IntUGT, sized_align, unsized_align); - bx.select(cmp, sized_align, unsized_align) + + let len = info.unwrap(); + let base = if elem_size == 1 { + len + } else { + // All slice sizes must fit into `isize`, so this multiplication cannot + // wrap -- neither signed nor unsigned. + bx.unchecked_sumul(len, bx.const_usize(elem_size)) + }; + + // Walk from innermost wrapper (`steps.iter().rev()`) to outermost (`steps[0]`), + // distilling the chain into `outer_offset + align_to(base + offset, align)`. + let mut offset = 0u64; + let mut outer_offset = 0u64; + let mut align = 1u64; + + for step in steps.iter().rev() { + if align <= step.align { + offset += (outer_offset + step.offset).next_multiple_of(align); + outer_offset = 0; + align = step.align; + if elem_size.is_multiple_of(align) { + offset = offset.next_multiple_of(align); + align = 1; + } + } else { + let c = (outer_offset + step.offset).next_multiple_of(step.align); + offset += c & !(align - 1); + outer_offset = c & (align - 1); + } } - }; - - // # Then compute the dynamic size - - // The full formula for the size would be: - // let unsized_offset_adjusted = unsized_offset_unadjusted.align_to(unsized_align); - // let full_size = (unsized_offset_adjusted + unsized_size).align_to(full_align); - // However, `unsized_size` is a multiple of `unsized_align`. Therefore, we can - // equivalently do the `align_to(unsized_align)` *after* adding `unsized_size`: - // - // let full_size = - // (unsized_offset_unadjusted + unsized_size) - // .align_to(unsized_align) - // .align_to(full_align); - // - // Furthermore, `align >= unsized_align`, and therefore we only need to do: - // let full_size = (unsized_offset_unadjusted + unsized_size).align_to(full_align); - - let full_size = bx.add(unsized_offset_unadjusted, unsized_size); - - // Issue #27023: must add any necessary padding to `size` - // (to make it a multiple of `align`) before returning it. - // - // Namely, the returned size should be, in C notation: - // - // `size + ((size & (align-1)) ? align : 0)` - // - // emulated via the semi-standard fast bit trick: - // - // `(size + (align-1)) & -align` - let one = bx.const_usize(1); - let addend = bx.sub(full_align, one); - let add = bx.add(full_size, addend); - let neg = bx.neg(full_align); - let full_size = bx.and(add, neg); - - (full_size, full_align) + + let aligned = emit_aligned_static_size(bx, base, elem_size, offset, align); + return if outer_offset == 0 { + aligned + } else { + bx.unchecked_suadd(aligned, bx.const_usize(outer_offset)) + }; + } + ty::Dynamic(..) => { + return emit_dyn_dst_size(bx, info.unwrap(), &steps); + } + ty::Foreign(_) => { + codegen_foreign_panic(bx, cur_ty); + return bx.const_usize(layout.size.bytes()); + } + _ => bug!("size_of_dst: {t} (tail {cur_ty}) not supported"), } - _ => bug!("size_and_align_of_dst: {t} not supported"), } } diff --git a/compiler/rustc_mir_transform/src/instsimplify.rs b/compiler/rustc_mir_transform/src/instsimplify.rs index 11ce25f2f5d7d..aee213c329af8 100644 --- a/compiler/rustc_mir_transform/src/instsimplify.rs +++ b/compiler/rustc_mir_transform/src/instsimplify.rs @@ -292,7 +292,7 @@ impl<'tcx> InstSimplifyContext<'_, 'tcx> { let ty = if generic_ty.is_sized(self.tcx, self.typing_env) { generic_ty } else if let LangItem::AlignOf = lang_item - && let ty::Slice(elem_ty) = *generic_ty.kind() + && let Some(elem_ty) = self.statically_known_unsized_align_ty(generic_ty) { elem_ty } else { @@ -317,6 +317,36 @@ impl<'tcx> InstSimplifyContext<'_, 'tcx> { } } + /// If `ty` is an unsized type whose alignment is guaranteed to equal that of a known + /// `Sized` type `U` (without computing layouts or risking recursive-type cycles), returns + /// `Some(U)`. + fn statically_known_unsized_align_ty(&self, mut ty: Ty<'tcx>) -> Option> { + for _ in 0..8 { + match *ty.kind() { + ty::Slice(elem_ty) => return Some(elem_ty), + ty::Str => return Some(self.tcx.types.u8), + ty::Tuple(tys) if let [elem_ty] = &tys[..] => { + ty = *elem_ty; + } + ty::Adt(def, args) if def.is_struct() => { + let repr = def.repr(); + if repr.align.is_some() || repr.pack.is_some() { + return None; + } + let [field] = &def.non_enum_variant().fields.raw[..] else { + return None; + }; + ty = field.ty(self.tcx, args).skip_normalization(); + } + ty::Pat(inner, _) => { + ty = inner; + } + _ => return None, + } + } + None + } + /// Simplify `raw_eq` intrinsic calls to `Eq` when the type has the size of a primitive. /// /// For example, replace `raw_eq::<[u8; 4]>(a, b)` with `Eq(Transmute(a), Transmute(b))`. diff --git a/tests/codegen-llvm/dst-offset.rs b/tests/codegen-llvm/dst-offset.rs index 2cf5fa9fac6ab..387a0883071aa 100644 --- a/tests/codegen-llvm/dst-offset.rs +++ b/tests/codegen-llvm/dst-offset.rs @@ -23,10 +23,8 @@ struct Dst { #[no_mangle] pub fn dst_dyn_trait_offset(s: &Dst) -> &dyn Drop { // The alignment of dyn trait is unknown, so we compute the offset based on align from the - // vtable. + // vtable (without loading the unused size from the vtable). - // CHECK: [[SIZE_PTR:%[0-9]+]] = getelementptr inbounds i8, ptr [[VTABLE_PTR]] - // CHECK: load [[USIZE]], ptr [[SIZE_PTR]] // CHECK: [[ALIGN_PTR:%[0-9]+]] = getelementptr inbounds i8, ptr [[VTABLE_PTR]] // CHECK: load [[USIZE]], ptr [[ALIGN_PTR]] diff --git a/tests/codegen-llvm/intrinsics/size_and_align_of_val.rs b/tests/codegen-llvm/intrinsics/size_and_align_of_val.rs index 2ae45aec5a623..24e586515ce72 100644 --- a/tests/codegen-llvm/intrinsics/size_and_align_of_val.rs +++ b/tests/codegen-llvm/intrinsics/size_and_align_of_val.rs @@ -29,7 +29,6 @@ pub unsafe fn size_of_array(x: &[u16; 7]) -> usize { #[no_mangle] pub unsafe fn align_of_slice(x: &[u16]) -> usize { // CHECK: start: - // CHECK-NEXT: [[SIZE:%.+]] = mul nuw nsw i64 %x.1, 2 // CHECK-NEXT: ret i64 2 align_of_val(x) } @@ -42,3 +41,160 @@ pub unsafe fn size_of_slice(x: &[u16]) -> usize { // CHECK-NEXT: ret i64 [[SIZE]] size_of_val(x) } + +// CHECK-LABEL: @size_of_u8_slice( +#[no_mangle] +pub unsafe fn size_of_u8_slice(x: &[u8]) -> usize { + // CHECK: start: + // CHECK-NEXT: ret i64 %x.1 + size_of_val(x) +} + +#[repr(C)] +pub struct Level0([u8]); +#[repr(C)] +pub struct Level1(u16, Level0); +#[repr(C)] +pub struct Level2(u32, Level1); +#[repr(C)] +pub struct Level3(u64, Level2); + +// CHECK-LABEL: @size_of_nested_slice_dst( +#[no_mangle] +pub unsafe fn size_of_nested_slice_dst(x: &Level3) -> usize { + // CHECK: start: + // CHECK-NEXT: [[UNALIGNED:%.+]] = add nuw i64 %x.1, 21 + // CHECK-NEXT: [[SIZE:%.+]] = and i64 [[UNALIGNED]], -8 + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +// CHECK-LABEL: @align_of_nested_slice_dst( +#[no_mangle] +pub unsafe fn align_of_nested_slice_dst(x: &Level3) -> usize { + // CHECK: start: + // CHECK-NEXT: ret i64 8 + align_of_val(x) +} + +#[repr(C)] +pub struct Unpadded0([u64]); +#[repr(C)] +pub struct Unpadded1(u64, Unpadded0); +#[repr(C)] +pub struct Unpadded2(u64, Unpadded1); +#[repr(C)] +pub struct Unpadded3(u64, Unpadded2); + +// CHECK-LABEL: @size_of_unpadded_slice_dst( +#[no_mangle] +pub unsafe fn size_of_unpadded_slice_dst(x: &Unpadded3) -> usize { + // CHECK: start: + // CHECK-NEXT: [[SLICE_BYTES:%.+]] = mul nuw nsw i64 %x.1, 8 + // CHECK-NEXT: [[SIZE:%.+]] = add nuw nsw i64 [[SLICE_BYTES]], 24 + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +#[repr(C)] +pub struct AlignedInner(u32, u8, [u8]); +#[repr(C, packed(2))] +pub struct PackedAroundAligned(u8, AlignedInner); +#[repr(C, align(8))] +pub struct AlignedAroundPacked(u8, PackedAroundAligned); + +// CHECK-LABEL: @size_of_packed_around_aligned( +#[no_mangle] +pub unsafe fn size_of_packed_around_aligned(x: &PackedAroundAligned) -> usize { + // CHECK: start: + // CHECK-NEXT: [[UNALIGNED:%.+]] = add nuw i64 %x.1, 8 + // CHECK-NEXT: [[ALIGNED:%.+]] = and i64 [[UNALIGNED]], -4 + // CHECK-NEXT: [[SIZE:%.+]] = add nuw nsw i64 [[ALIGNED]], 2 + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +// CHECK-LABEL: @size_of_aligned_around_packed( +#[no_mangle] +pub unsafe fn size_of_aligned_around_packed(x: &AlignedAroundPacked) -> usize { + // CHECK: start: + // CHECK-NEXT: [[UNALIGNED:%.+]] = add nuw i64 %x.1, 16 + // CHECK-NEXT: [[SIZE:%.+]] = and i64 [[UNALIGNED]], -8 + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +#[repr(C)] +pub struct NonPow2ElemSliceDst(u64, [[u8; 3]]); + +// CHECK-LABEL: @size_of_non_pow2_elem_slice_dst( +#[no_mangle] +pub unsafe fn size_of_non_pow2_elem_slice_dst(x: &NonPow2ElemSliceDst) -> usize { + // CHECK: start: + // CHECK-NEXT: [[SLICE_BYTES:%.+]] = mul nuw nsw i64 %x.1, 3 + // CHECK-NEXT: [[UNALIGNED:%.+]] = add nuw i64 [[SLICE_BYTES]], 15 + // CHECK-NEXT: [[SIZE:%.+]] = and i64 [[UNALIGNED]], -8 + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +#[repr(C)] +pub struct ZstSliceDst(u32, u8, [()]); + +// CHECK-LABEL: @size_of_zst_slice_dst( +#[no_mangle] +pub unsafe fn size_of_zst_slice_dst(x: &ZstSliceDst) -> usize { + // CHECK: start: + // CHECK-NEXT: ret i64 8 + size_of_val(x) +} + +pub trait Trait {} + +// CHECK-LABEL: @size_of_manually_drop_dyn( +#[no_mangle] +pub unsafe fn size_of_manually_drop_dyn(x: &std::mem::ManuallyDrop) -> usize { + // CHECK: start: + // CHECK-NEXT: [[VTABLE_SIZE_PTR:%.+]] = getelementptr inbounds i8, ptr %x.1, i64 8 + // CHECK-NEXT: [[SIZE:%.+]] = load i64, ptr [[VTABLE_SIZE_PTR]], align 8, !range [[RANGE:![0-9]+]], !invariant.load + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +#[repr(C)] +pub struct DynWrap0(u8, T); + +// CHECK-LABEL: @size_of_dyn_wrap0( +#[no_mangle] +pub unsafe fn size_of_dyn_wrap0(x: &DynWrap0) -> usize { + // CHECK: start: + // CHECK-NEXT: [[VTABLE_SIZE_PTR:%.+]] = getelementptr inbounds i8, ptr %x.1, i64 8 + // CHECK-NEXT: [[DYN_SIZE:%.+]] = load i64, ptr [[VTABLE_SIZE_PTR]], align 8, !range {{![0-9]+}}, !invariant.load + // CHECK-NEXT: [[VTABLE_ALIGN_PTR:%.+]] = getelementptr inbounds i8, ptr %x.1, i64 16 + // CHECK-NEXT: [[ALIGN:%.+]] = load i64, ptr [[VTABLE_ALIGN_PTR]], align 8, !range {{![0-9]+}}, !invariant.load + // CHECK-NEXT: [[SIZE:%.+]] = add nuw nsw i64 [[DYN_SIZE]], [[ALIGN]] + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +#[repr(C, packed)] +pub struct PackedDyn(u8, std::mem::ManuallyDrop); + +// CHECK-LABEL: @size_of_packed_dyn( +#[no_mangle] +pub unsafe fn size_of_packed_dyn(x: &PackedDyn) -> usize { + // CHECK: start: + // CHECK-NEXT: [[VTABLE_SIZE_PTR:%.+]] = getelementptr inbounds i8, ptr %x.1, i64 8 + // CHECK-NEXT: [[DYN_SIZE:%.+]] = load i64, ptr [[VTABLE_SIZE_PTR]], align 8, !range {{![0-9]+}}, !invariant.load + // CHECK-NEXT: [[SIZE:%.+]] = add nuw nsw i64 [[DYN_SIZE]], 1 + // CHECK-NEXT: ret i64 [[SIZE]] + size_of_val(x) +} + +// CHECK-LABEL: @align_of_packed_dyn( +#[no_mangle] +pub unsafe fn align_of_packed_dyn(x: &PackedDyn) -> usize { + // CHECK: start: + // CHECK-NEXT: ret i64 1 + align_of_val(x) +} diff --git a/tests/mir-opt/instsimplify/align_of_slice.of_val_str.InstSimplify-after-simplifycfg.diff b/tests/mir-opt/instsimplify/align_of_slice.of_val_str.InstSimplify-after-simplifycfg.diff new file mode 100644 index 0000000000000..4c920b454873a --- /dev/null +++ b/tests/mir-opt/instsimplify/align_of_slice.of_val_str.InstSimplify-after-simplifycfg.diff @@ -0,0 +1,22 @@ +- // MIR for `of_val_str` before InstSimplify-after-simplifycfg ++ // MIR for `of_val_str` after InstSimplify-after-simplifycfg + + fn of_val_str(_1: &str) -> usize { + debug s => _1; + let mut _0: usize; + let mut _2: *const str; + + bb0: { + StorageLive(_2); + _2 = &raw const (*_1); +- _0 = std::intrinsics::align_of_val::(move _2) -> [return: bb1, unwind unreachable]; ++ _0 = const ::ALIGN; ++ goto -> bb1; + } + + bb1: { + StorageDead(_2); + return; + } + } + diff --git a/tests/mir-opt/instsimplify/align_of_slice.of_val_wrapper.InstSimplify-after-simplifycfg.diff b/tests/mir-opt/instsimplify/align_of_slice.of_val_wrapper.InstSimplify-after-simplifycfg.diff new file mode 100644 index 0000000000000..9f2aa95293a53 --- /dev/null +++ b/tests/mir-opt/instsimplify/align_of_slice.of_val_wrapper.InstSimplify-after-simplifycfg.diff @@ -0,0 +1,22 @@ +- // MIR for `of_val_wrapper` before InstSimplify-after-simplifycfg ++ // MIR for `of_val_wrapper` after InstSimplify-after-simplifycfg + + fn of_val_wrapper(_1: &Wrapper<[T]>) -> usize { + debug w => _1; + let mut _0: usize; + let mut _2: *const Wrapper<[T]>; + + bb0: { + StorageLive(_2); + _2 = &raw const (*_1); +- _0 = std::intrinsics::align_of_val::>(move _2) -> [return: bb1, unwind unreachable]; ++ _0 = const ::ALIGN; ++ goto -> bb1; + } + + bb1: { + StorageDead(_2); + return; + } + } + diff --git a/tests/mir-opt/instsimplify/align_of_slice.rs b/tests/mir-opt/instsimplify/align_of_slice.rs index 99a68e444fe5d..5fe311c4033e0 100644 --- a/tests/mir-opt/instsimplify/align_of_slice.rs +++ b/tests/mir-opt/instsimplify/align_of_slice.rs @@ -10,3 +10,20 @@ pub fn of_val_slice(slice: &[T]) -> usize { // CHECK: _0 = const ::ALIGN; unsafe { core::intrinsics::align_of_val(slice) } } + +// EMIT_MIR align_of_slice.of_val_str.InstSimplify-after-simplifycfg.diff +pub fn of_val_str(s: &str) -> usize { + // CHECK-LABEL: fn of_val_str(_1: &str) + // CHECK: _0 = const ::ALIGN; + unsafe { core::intrinsics::align_of_val(s) } +} + +pub struct Wrapper(WrapperInner); +pub struct WrapperInner((T,)); + +// EMIT_MIR align_of_slice.of_val_wrapper.InstSimplify-after-simplifycfg.diff +pub fn of_val_wrapper(w: &Wrapper<[T]>) -> usize { + // CHECK-LABEL: fn of_val_wrapper(_1: &Wrapper<[T]>) + // CHECK: _0 = const ::ALIGN; + unsafe { core::intrinsics::align_of_val(w) } +} diff --git a/tests/ui/dst/dst-nested-size-align-offset.rs b/tests/ui/dst/dst-nested-size-align-offset.rs new file mode 100644 index 0000000000000..02b1eb4acae2a --- /dev/null +++ b/tests/ui/dst/dst-nested-size-align-offset.rs @@ -0,0 +1,225 @@ +//@ run-pass +#![allow(dead_code)] + +use std::cell::{Cell, UnsafeCell}; +use std::mem::{ + ManuallyDrop, align_of, align_of_val, align_of_val_raw, offset_of, size_of, size_of_val, + size_of_val_raw, +}; +use std::ptr::{addr_of, slice_from_raw_parts}; + +// --- Slice-tailed DST hierarchy --- + +#[repr(C)] +struct AlignedInner { + a: u32, + b: u8, + tail: T, +} + +#[repr(C, packed(2))] +struct Packed2AroundAligned { + z: u8, + inner: ManuallyDrop>, +} + +#[repr(C, packed)] +struct Packed1AroundAligned { + z: u8, + inner: ManuallyDrop>, +} + +#[repr(C, align(8))] +struct Align8AroundPacked2 { + w: u8, + outer: Packed2AroundAligned, +} + +#[repr(C, packed(2))] +struct Packed2AroundAlign8 { + v: u8, + aligned: ManuallyDrop>, +} + +#[repr(C)] +struct DeepReprC { + a: u64, + b: (u32, (u16, AlignedInner)), +} + +macro_rules! check_slice_dst_len { + ($len:expr, $elem:ty) => {{ + unsafe { + let base = std::ptr::without_provenance::<()>(0x1000); + let raw_slice = slice_from_raw_parts(base, $len); + + let p = raw_slice as *const AlignedInner<[$elem]>; + assert_eq!(size_of_val_raw(p), size_of::>()); + assert_eq!(align_of_val_raw(p), align_of::>()); + assert_eq!( + addr_of!((*p).tail).cast::().offset_from(p.cast::()) as usize, + offset_of!(AlignedInner<[$elem; $len]>, tail), + ); + + let p = raw_slice as *const Packed2AroundAligned<[$elem]>; + assert_eq!(size_of_val_raw(p), size_of::>()); + assert_eq!(align_of_val_raw(p), align_of::>()); + + let p = raw_slice as *const Packed1AroundAligned<[$elem]>; + assert_eq!(size_of_val_raw(p), size_of::>()); + assert_eq!(align_of_val_raw(p), align_of::>()); + + let p = raw_slice as *const Align8AroundPacked2<[$elem]>; + assert_eq!(size_of_val_raw(p), size_of::>()); + assert_eq!(align_of_val_raw(p), align_of::>()); + assert_eq!( + addr_of!((*p).outer).cast::().offset_from(p.cast::()) as usize, + offset_of!(Align8AroundPacked2<[$elem; $len]>, outer), + ); + + let p = raw_slice as *const Packed2AroundAlign8<[$elem]>; + assert_eq!(size_of_val_raw(p), size_of::>()); + assert_eq!(align_of_val_raw(p), align_of::>()); + + let p = raw_slice as *const DeepReprC<[$elem]>; + assert_eq!(size_of_val_raw(p), size_of::>()); + assert_eq!(align_of_val_raw(p), align_of::>()); + assert_eq!( + addr_of!((*p).b).cast::().offset_from(p.cast::()) as usize, + offset_of!(DeepReprC<[$elem; $len]>, b), + ); + } + }}; +} + +macro_rules! check_slice_dst_elem { + ($elem:ty) => {{ + check_slice_dst_len!(0, $elem); + check_slice_dst_len!(1, $elem); + check_slice_dst_len!(2, $elem); + check_slice_dst_len!(3, $elem); + check_slice_dst_len!(4, $elem); + check_slice_dst_len!(5, $elem); + check_slice_dst_len!(7, $elem); + check_slice_dst_len!(8, $elem); + }}; +} + +// --- `dyn Trait`-tailed DST hierarchy --- + +#[repr(align(1))] +struct A1([u8; 3]); +#[repr(align(2))] +struct A2([u8; 6]); +#[repr(align(4))] +struct A4([u8; 12]); +#[repr(align(8))] +struct A8([u8; 24]); +#[repr(align(16))] +struct A16([u8; 16]); + +#[repr(C)] +struct DynInner(u8, ManuallyDrop>>); + +#[repr(C)] +struct DynOffset2([u8; 2], ManuallyDrop); + +#[repr(C)] +struct DynOffset6([u8; 6], ManuallyDrop); + +#[repr(C, packed(2))] +struct DynPacked2(u8, ManuallyDrop); + +#[repr(C, packed(4))] +struct DynPacked4Offset2([u8; 2], ManuallyDrop); + +#[repr(C, packed(4))] +struct DynPacked4Offset3([u8; 3], ManuallyDrop); + +#[repr(C, packed(2))] +struct DynPacked2AroundInner(u8, ManuallyDrop>); + +#[repr(C, align(8))] +struct DynAlign8AroundPacked2(u8, DynPacked2AroundInner); + +macro_rules! check_dyn_dst { + ($val:expr, $ty:ty) => {{ + let md = ManuallyDrop::new($val); + let md_sized: &ManuallyDrop<$ty> = &md; + let md_dyn: &ManuallyDrop = md_sized; + assert_eq!(size_of_val(md_sized), size_of_val(md_dyn)); + assert_eq!(align_of_val(md_sized), align_of_val(md_dyn)); + + let inner = DynInner(1, ManuallyDrop::new(UnsafeCell::new(Cell::new($val)))); + let inner_sized: &DynInner<$ty> = &inner; + let inner_dyn: &DynInner = inner_sized; + assert_eq!(size_of_val(inner_sized), size_of_val(inner_dyn)); + assert_eq!(align_of_val(inner_sized), align_of_val(inner_dyn)); + assert_eq!(addr_of!(inner_sized.1).cast::(), addr_of!(inner_dyn.1).cast::()); + + let o2 = DynOffset2([1, 2], ManuallyDrop::new($val)); + let o2_sized: &DynOffset2<$ty> = &o2; + let o2_dyn: &DynOffset2 = o2_sized; + assert_eq!(size_of_val(o2_sized), size_of_val(o2_dyn)); + assert_eq!(align_of_val(o2_sized), align_of_val(o2_dyn)); + assert_eq!(addr_of!(o2_sized.1).cast::(), addr_of!(o2_dyn.1).cast::()); + + let o6 = DynOffset6([1, 2, 3, 4, 5, 6], ManuallyDrop::new($val)); + let o6_sized: &DynOffset6<$ty> = &o6; + let o6_dyn: &DynOffset6 = o6_sized; + assert_eq!(size_of_val(o6_sized), size_of_val(o6_dyn)); + assert_eq!(align_of_val(o6_sized), align_of_val(o6_dyn)); + assert_eq!(addr_of!(o6_sized.1).cast::(), addr_of!(o6_dyn.1).cast::()); + + let p2 = DynPacked2(1, ManuallyDrop::new($val)); + let p2_sized: &DynPacked2<$ty> = &p2; + let p2_dyn: &DynPacked2 = p2_sized; + assert_eq!(size_of_val(p2_sized), size_of_val(p2_dyn)); + assert_eq!(align_of_val(p2_sized), align_of_val(p2_dyn)); + assert_eq!(addr_of!(p2_sized.1).cast::(), addr_of!(p2_dyn.1).cast::()); + + let p4_2 = DynPacked4Offset2([1, 2], ManuallyDrop::new($val)); + let p4_2_sized: &DynPacked4Offset2<$ty> = &p4_2; + let p4_2_dyn: &DynPacked4Offset2 = p4_2_sized; + assert_eq!(size_of_val(p4_2_sized), size_of_val(p4_2_dyn)); + assert_eq!(align_of_val(p4_2_sized), align_of_val(p4_2_dyn)); + assert_eq!(addr_of!(p4_2_sized.1).cast::(), addr_of!(p4_2_dyn.1).cast::()); + + let p4_3 = DynPacked4Offset3([1, 2, 3], ManuallyDrop::new($val)); + let p4_3_sized: &DynPacked4Offset3<$ty> = &p4_3; + let p4_3_dyn: &DynPacked4Offset3 = p4_3_sized; + assert_eq!(size_of_val(p4_3_sized), size_of_val(p4_3_dyn)); + assert_eq!(align_of_val(p4_3_sized), align_of_val(p4_3_dyn)); + assert_eq!(addr_of!(p4_3_sized.1).cast::(), addr_of!(p4_3_dyn.1).cast::()); + + let a8 = DynAlign8AroundPacked2( + 1, + DynPacked2AroundInner( + 2, + ManuallyDrop::new(DynInner(3, ManuallyDrop::new(UnsafeCell::new(Cell::new($val))))), + ), + ); + let a8_sized: &DynAlign8AroundPacked2<$ty> = &a8; + let a8_dyn: &DynAlign8AroundPacked2 = a8_sized; + assert_eq!(size_of_val(a8_sized), size_of_val(a8_dyn)); + assert_eq!(align_of_val(a8_sized), align_of_val(a8_dyn)); + assert_eq!(addr_of!(a8_sized.1).cast::(), addr_of!(a8_dyn.1).cast::()); + }}; +} + +fn main() { + check_slice_dst_elem!(()); + check_slice_dst_elem!(u8); + check_slice_dst_elem!(u16); + check_slice_dst_elem!([u8; 3]); + check_slice_dst_elem!(u32); + check_slice_dst_elem!([u16; 3]); + check_slice_dst_elem!(u64); + + check_dyn_dst!((), ()); + check_dyn_dst!(A1([0; 3]), A1); + check_dyn_dst!(A2([0; 6]), A2); + check_dyn_dst!(A4([0; 12]), A4); + check_dyn_dst!(A8([0; 24]), A8); + check_dyn_dst!(A16([0; 16]), A16); +}