← sources

Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp

source: reddit (reddit:LocalLLaMA)published 2026-10-22 18:16ingested 2026-10-23 02:42

retained text

This made a huge improvement to speculate decoding speeds on my B70 running SYCL.

Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2.

NOTE: Due to the significantly reduced cost of token verification, the n-max was increased which is why the acceptance rate is lower.


Before:

```
  code_python        pred= 192 draft= 127 acc= 106 rate=0.835 tok/s=38.0
  code_cpp           pred= 192 draft= 175 acc= 113 rate=0.646 tok/s=36.2
  explain_concept    pred= 192 draft= 135 acc= 116 rate=0.859 tok/s=41.2
  summarize          pred= 188 draft= 170 acc= 142 rate=0.835 tok/s=50.1
  qa_factual         pred= 192 draft= 145 acc= 107 rate=0.738 tok/s=37.0
  translation        pred= 192 draft= 175 acc= 133 rate=0.760 tok/s=45.1
  creative_short     pred= 192 draft= 130 acc= 105 rate=0.808 tok/s=37.1
  stepwise_math      pred= 192 draft= 167 acc= 145 rate=0.868 tok/s=53.2
  long_code_review   pred= 192 draft= 159 acc= 118 rate=0.742 tok/s=38.4

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1724,
  "total_draft": 1383,
  "total_draft_accepted": 1085,
  "aggregate_accept_rate": 0.7845,
  "wall_s_total": 46.86
}
```

After:

```
  code_python        pred= 192 draft= 324 acc= 143 rate=0.441 tok/s=73.0
  code_cpp           pred= 192 draft= 424 acc= 140 rate=0.330 tok/s=68.5
  explain_concept    pred= 192 draft= 330 acc= 141 rate=0.427 tok/s=70.7
  summarize          pred= 192 draft= 416 acc= 151 rate=0.363 tok/s=85.4
  qa_factual         pred= 192 draft= 399 acc= 133 rate=0.333 tok/s=60.3
  translation        pred= 192 draft= 356 acc= 152 rate=0.427 tok/s=87.7
  creative_short     pred= 192 draft= 379 acc= 125 rate=0.330 tok/s=53.6
  stepwise_math      pred= 192 draft= 462 acc= 148 rate=0.320 tok/s=77.6
  long_code_review   pred= 192 draft= 414 acc= 139 rate=0.336 tok/s=64.2

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 3504,
  "total_draft_accepted": 1272,
  "aggregate_accept_rate": 0.363,
  "wall_s_total": 26.68
}
```

document fetched 2026-10-23 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xcqb3p/massive_b70_specdecoding_improvement_intel_xmx