【FR】Bug: add operator performance degradation on nvidia (google/gemma-4-E2B-it)

Bug Report: Bug: Operator performance degradation on nvidia (google/gemma-4-E2B-it)

Description

During performance benchmarking on model google/gemma-4-E2B-it, operator elimination search identified the following operator as causing significant performance degradation:

• ADD (flag_gems.ops.add.add): When all 70 FlagGems operators are enabled, throughput drops to 69.0% of native baseline. Disabling only the ADD operator recovers throughput to 96.3% of native baseline.

Environment

Item	Value
Hardware	nvidia
GPU	H20-3e
PyTorch	2.10.0+cu128
FlagGems	5.0.1.rc.0
vLLM	0.19.0

Steps to Reproduce

1. Set up environment on nvidia
2. Install dependencies (PyTorch + Triton + FlagGems + vLLM)
3. Run performance benchmark with V1 (native) and V2 (FlagGems enabled)
4. Compare V1 vs V2 throughput, ratio below 80% at specific concurrency levels
5. Run operator search optimization to identify problematic operators

Expected Behavior

All FlagGems operators should maintain performance at >=80% of native baseline at each concurrency level.

Actual Behavior

The ADD operator causes severe performance degradation:

• Native baseline: 33,798 tok/s
• All 70 FlagGems operators enabled: 23,313 tok/s (69.0% of baseline)
• After disabling ADD only: 32,551 tok/s (96.3% of baseline)

The ADD operator alone accounts for a ~27% throughput drop. All other 69 operators run within acceptable performance range.

FlagGems Integration Code

File: /usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py Enable call: flag_gems.enable(record=True, once=True, path="/root/gems.txt")

Current gems.txt (66 operators, path: /root/gems.txt):

[DEBUG] flag_gems.ops.zeros.zeros: GEMS ZEROS
[DEBUG] flag_gems.ops.full.full: GEMS FULL
[DEBUG] flag_gems.ops.zeros.zero_: GEMS ZERO_
[DEBUG] flag_gems.ops.ones.ones: GEMS ONES
[DEBUG] flag_gems.ops.to.to_copy: GEMS _TO_COPY
[DEBUG] flag_gems.ops.div.true_divide: GEMS TRUE_DIVIDE
[DEBUG] flag_gems.ops.pow.pow_scalar: GEMS POW_SCALAR
[DEBUG] flag_gems.ops.reciprocal.reciprocal: GEMS RECIPROCAL
[DEBUG] flag_gems.ops.mul.mul: GEMS MUL
[DEBUG] flag_gems.ops.copy.copy_: GEMS COPY_
[DEBUG] flag_gems.ops.normal.normal_: GEMS NORMAL_
[DEBUG] flag_gems.ops.fill.fill_scalar_: GEMS FILL_SCALAR_
[DEBUG] flag_gems.ops.arange.arange_start: GEMS ARANGE
[DEBUG] flag_gems.ops.exp.exp: GEMS EXP
[DEBUG] flag_gems.ops.rsqrt.rsqrt: GEMS RSQRT
[DEBUG] flag_gems.ops.cos.cos: GEMS COS
[DEBUG] flag_gems.ops.sin.sin: GEMS SIN
[DEBUG] flag_gems.ops.cat.cat: GEMS CAT
[DEBUG] flag_gems.ops.resolve_conj.resolve_conj: GEMS RESOLVE_CONJ
[DEBUG] flag_gems.ops.resolve_neg.resolve_neg: GEMS RESOLVE_NEG
[DEBUG] flag_gems.ops.eq.eq_scalar: GEMS EQ SCALAR
[DEBUG] flag_gems.ops.all.all_dim: GEMS ALL DIM
[DEBUG] flag_gems.ops.sub.sub: GEMS SUB
[DEBUG] flag_gems.runtime.backend._nvidia.hopper.ops.mm.general_mm: GEMS MM-hopper, [mm scenario]: general, [shape info]: [-, 630, 768, 768](batch, M, N, K), [A column-major]: False, [B column-major]: True
[DEBUG] flag_gems.ops.clamp.clamp: GEMS CLAMP
[DEBUG] flag_gems.ops.one_hot.one_hot: GEMS ONE_HOT
[DEBUG] flag_gems.ops.bmm.bmm: GEMS BMM
[DEBUG] flag_gems.ops.sum.sum_dim: GEMS SUM_DIM
[DEBUG] flag_gems.ops.where.where_self: GEMS WHERE_SELF
[DEBUG] flag_gems.ops.where.where_self_out: GEMS WHERE_SELF_OUT
[DEBUG] flag_gems.ops.bitwise_not.bitwise_not: GEMS BITWISE NOT
[DEBUG] flag_gems.ops.all.all: GEMS ALL
[DEBUG] flag_gems.ops.ge.ge_scalar: GEMS GE SCALAR
[DEBUG] flag_gems.ops.index.index: GEMS INDEX
[DEBUG] flag_gems.ops.pow.pow_tensor_scalar: GEMS POW_TENSOR_SCALAR
[DEBUG] flag_gems.ops.mean.mean_dim: GEMS MEAN_DIM (wrapper)
[DEBUG] flag_gems.ops.mean.mean_dim_comm: GEMS MEAN_DIM
[DEBUG] flag_gems.ops.clamp.clamp_tensor: GEMS CLAMP TENSOR
[DEBUG] flag_gems.ops.neg.neg: GEMS NEG
[DEBUG] flag_gems.ops.pad.pad: GEMS CONSTANT PAD ND
[DEBUG] flag_gems.ops.gelu.gelu: GEMS GELU FORWARD
[DEBUG] flag_gems.ops.masked_fill.masked_fill: GEMS MASKED FILL
[DEBUG] flag_gems.ops.max.max_dim: GEMS MAX DIM
[DEBUG] flag_gems.ops.div.div_mode: GEMS DIV_MODE
[DEBUG] flag_gems.ops.div.floor_divide: GEMS FLOOR_DIVIDE
[DEBUG] flag_gems.ops.logical_not.logical_not: GEMS LOGICAL_NOT
[DEBUG] flag_gems.ops.mul.mul_: GEMS MUL_
[DEBUG] flag_gems.ops.nonzero.nonzero: GEMS NONZERO
[DEBUG] flag_gems.ops.cumsum.cumsum: GEMS CUMSUM
[DEBUG] flag_gems.ops.rand_like.rand_like: GEMS RAND_LIKE
[DEBUG] flag_gems.ops.tanh.tanh: GEMS TANH FORWARD
[DEBUG] flag_gems.ops.argmax.argmax: GEMS ARGMAX
[DEBUG] flag_gems.ops.lt.lt_scalar: GEMS LT SCALAR
[DEBUG] flag_gems.ops.div.true_divide_: GEMS TRUE_DIVIDE_
[DEBUG] flag_gems.ops.softmax.softmax: GEMS SOFTMAX
[DEBUG] flag_gems.ops.exponential_.exponential_: GEMS EXPONENTIAL_
[DEBUG] flag_gems.ops.zeros_like.zeros_like: GEMS ZEROS_LIKE
[DEBUG] flag_gems.ops.logical_and.logical_and: GEMS LOGICAL_AND
[DEBUG] flag_gems.ops.embedding.embedding: GEMS EMBEDDING FORWARD
[DEBUG] flag_gems.ops.sort.sort: GEMS SORT
[DEBUG] flag_gems.ops.sort.sort_stable: GEMS SORT.STABLE
[DEBUG] flag_gems.ops.gather.gather: GEMS GATHER
[DEBUG] flag_gems.ops.lt.lt: GEMS LT
[DEBUG] flag_gems.ops.cumsum.cumsum_out: GEMS CUMSUM_OUT
[DEBUG] flag_gems.ops.le.le: GEMS LE
[DEBUG] flag_gems.ops.scatter.scatter_: GEMS SCATTER_

Additional Context

• Operator elimination search: 2 rounds, 307s elapsed
• Search mode: elimination, direction: forward
• 70 operators total in search space, single-elimination of ADD was sufficient
• Accuracy evaluation: PASSED (V1=36%, V2=34%, drop=2.0%, threshold=5.0%)
• Previous run crashed during startup (service_crash=true), current run recovered

Possible Directions

• Profile the flag_gems.ops.add.add Triton kernel on H20-3e (Hopper architecture) to identify the performance bottleneck
• Check if the ADD kernel has suboptimal tile/block configuration for this GPU
• Compare native PyTorch ADD kernel vs FlagGems ADD Triton kernel launch overhead
• Verify dtype handling (BF16/FP16) in the ADD operator implementation