← Hackathons

Fused Linear Layer with SwiGLU

Fused Linear Layer with SwiGLU: two bias-free linear projections followed by standard SiLU gating

Operator definition

gate = x @ w_gate; up = x @ w_up; z = SiLU(gate) * up

Interface

fused_swiglu(Tensor x, Tensor w_gate, Tensor w_up) -> Tensor z

Data ranges

x:[B,K],w_gate,w_up:[K,N],z:[B,N]; B=1..1048576,K,N=1..65536

Single-die stress scale

One physical die only (nominal 64 GiB HBM). Functional cases remain small and exhaustive; the independent three-tier stress suite exercises large bandwidth, compute and random-access workloads. Input + output tensors must total <=48 GiB, candidate workspace <=8 GiB; reserve >=8 GiB for runtime. Shape bounds are not a Cartesian guarantee: combinations exceeding the byte budget are invalid. Pressure correctness compares complete offline binary standards outside timing. Public stress runs are debugging only; formal pressure scoring starts only after all functional checks pass. Host wall time includes CPU work, dispatch and device synchronization, not only kernel time. No other die or CPU computation fallback.

Value ranges and coverage

FP16 x: [-8, 8], w_gate/w_up: [-2^-12, 2^-12]; BF16 x: [-32, 32], w_gate/w_up: [-4, 4]. Finite inputs only. Coverage includes zero, small magnitudes down to 2^-12, large magnitudes, positive/negative-only data, mixed scales, sparse data and representable boundaries. Task-specific tests also cover cancellation, BF16 ties and saturated softmax. NaN/Inf are excluded; subnormal preservation is not required. Not every tensor must span the entire range.

Parameters

Inputs

Outputs

Attrs

Numerical and implementation requirements

Contiguous matrices, same device and matching FP16/BF16 dtype. Gate and up projections, SiLU and multiplication use FP32 with only a final output cast. No precomputed gate, bias or down projection. All custom-kernel launches are included in timing. FP16 weights are bounded by 2^-12 to keep maximum supported dimensions finite; BF16 weights are bounded by 4.

Outputs must remain on the input device, be contiguous, leave all inputs unchanged, and not alias any input. One physical die only; using the sibling die or multi-card communication is prohibited. Core computation, data movement and conversions must use custom NPU kernels. Built-in ACLNN operators, torch delegation, CPU fallback and cached answers are prohibited.

FP16 x: [-8, 8], w_gate/w_up: [-2^-12, 2^-12]; BF16 x: [-32, 32], w_gate/w_up: [-4, 4]. Finite inputs only. Coverage includes zero, small magnitudes down to 2^-12, large magnitudes, positive/negative-only data, mixed scales, sparse data and representable boundaries. Task-specific tests also cover cancellation, BF16 ties and saturated softmax. NaN/Inf are excluded; subnormal preservation is not required. Not every tensor must span the entire range.

One physical die only (nominal 64 GiB HBM). Functional cases remain small and exhaustive; the independent three-tier stress suite exercises large bandwidth, compute and random-access workloads. Input + output tensors must total <=48 GiB, candidate workspace <=8 GiB; reserve >=8 GiB for runtime. Shape bounds are not a Cartesian guarantee: combinations exceeding the byte budget are invalid. Pressure correctness compares complete offline binary standards outside timing. Public stress runs are debugging only; formal pressure scoring starts only after all functional checks pass. Host wall time includes CPU work, dispatch and device synchronization, not only kernel time. No other die or CPU computation fallback.

Scoring: accuracy 20%, performance 80%; compilation has no bonus. All accuracy checks must pass before pressure evaluation.