LocalGroupedGEMM
Single-card grouped GEMM with runtime row offsets and support for empty groups
Operator definition
c[offsets[e]:offsets[e+1]] = a[offsets[e]:offsets[e+1]] @ b[e]
Interface
local_grouped_gemm(Tensor a, Tensor b, Tensor row_offsets) -> Tensor c
Data ranges
a:[T,K],b:[E,K,N],row_offsets:[E+1],c:[T,N]; T=1..1048576,E=2..256,K,N=128..32768 (multiples of 64)
Single-die stress scale
One physical die only (nominal 64 GiB HBM). Functional cases remain small and exhaustive; the independent three-tier stress suite exercises large bandwidth, compute and random-access workloads. Input + output tensors must total <=48 GiB, candidate workspace <=8 GiB; reserve >=8 GiB for runtime. Shape bounds are not a Cartesian guarantee: combinations exceeding the byte budget are invalid. Pressure correctness compares complete offline binary standards outside timing. Public stress runs are debugging only; formal pressure scoring starts only after all functional checks pass. Host wall time includes CPU work, dispatch and device synchronization, not only kernel time. No other die or CPU computation fallback.
Value ranges and coverage
a, b: [-256, 256]; row_offsets: [0, T]. Finite inputs only. Coverage includes zero, small magnitudes down to 2^-12, large magnitudes, positive/negative-only data, mixed scales, sparse data and representable boundaries. Task-specific tests also cover cancellation, BF16 ties and saturated softmax. NaN/Inf are excluded; subnormal preservation is not required. Not every tensor must span the entire range.
Parameters
Inputs
- a: ['bfloat16'] — Matrix of token rows
- b: ['bfloat16'] — Per-group matrices
- row_offsets: ['int32'] — Runtime NPU group boundaries: first 0, last T, nondecreasing; repeats allowed
Outputs
- c: ['bfloat16'] — Output preserving the original row order
Attrs
Numerical and implementation requirements
All inputs are contiguous and on the same card. a and b contain finite values in [-256, 256]. row_offsets is an NPU int32 tensor, starts at 0, ends at T and is nondecreasing. Empty groups are valid. Do not call .cpu(), .item() or .tolist() on offsets, or hard-code routing for known shapes. Accumulate in FP32, then cast to BF16.
Outputs must remain on the input device, be contiguous, leave all inputs unchanged, and not alias any input. One physical die only; using the sibling die or multi-card communication is prohibited. Core computation, data movement and conversions must use custom NPU kernels. Built-in ACLNN operators, torch delegation, CPU fallback and cached answers are prohibited.
a, b: [-256, 256]; row_offsets: [0, T]. Finite inputs only. Coverage includes zero, small magnitudes down to 2^-12, large magnitudes, positive/negative-only data, mixed scales, sparse data and representable boundaries. Task-specific tests also cover cancellation, BF16 ties and saturated softmax. NaN/Inf are excluded; subnormal preservation is not required. Not every tensor must span the entire range.
One physical die only (nominal 64 GiB HBM). Functional cases remain small and exhaustive; the independent three-tier stress suite exercises large bandwidth, compute and random-access workloads. Input + output tensors must total <=48 GiB, candidate workspace <=8 GiB; reserve >=8 GiB for runtime. Shape bounds are not a Cartesian guarantee: combinations exceeding the byte budget are invalid. Pressure correctness compares complete offline binary standards outside timing. Public stress runs are debugging only; formal pressure scoring starts only after all functional checks pass. Host wall time includes CPU work, dispatch and device synchronization, not only kernel time. No other die or CPU computation fallback.
Scoring: accuracy 20%, performance 80%; compilation has no bonus. All accuracy checks must pass before pressure evaluation.