Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp
retained text
This made a huge improvement to speculate decoding speeds on my B70 running SYCL.
Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2.
NOTE: Due to the significantly reduced cost of token verification, the n-max was increased which is why the acceptance rate is lower.
Before:
```
code_python pred= 192 draft= 127 acc= 106 rate=0.835 tok/s=38.0
code_cpp pred= 192 draft= 175 acc= 113 rate=0.646 tok/s=36.2
explain_concept pred= 192 draft= 135 acc= 116 rate=0.859 tok/s=41.2
summarize pred= 188 draft= 170 acc= 142 rate=0.835 tok/s=50.1
qa_factual pred= 192 draft= 145 acc= 107 rate=0.738 tok/s=37.0
translation pred= 192 draft= 175 acc= 133 rate=0.760 tok/s=45.1
creative_short pred= 192 draft= 130 acc= 105 rate=0.808 tok/s=37.1
stepwise_math pred= 192 draft= 167 acc= 145 rate=0.868 tok/s=53.2
long_code_review pred= 192 draft= 159 acc= 118 rate=0.742 tok/s=38.4
Aggregate: {
"n_requests": 9,
"total_predicted": 1724,
"total_draft": 1383,
"total_draft_accepted": 1085,
"aggregate_accept_rate": 0.7845,
"wall_s_total": 46.86
}
```
After:
```
code_python pred= 192 draft= 324 acc= 143 rate=0.441 tok/s=73.0
code_cpp pred= 192 draft= 424 acc= 140 rate=0.330 tok/s=68.5
explain_concept pred= 192 draft= 330 acc= 141 rate=0.427 tok/s=70.7
summarize pred= 192 draft= 416 acc= 151 rate=0.363 tok/s=85.4
qa_factual pred= 192 draft= 399 acc= 133 rate=0.333 tok/s=60.3
translation pred= 192 draft= 356 acc= 152 rate=0.427 tok/s=87.7
creative_short pred= 192 draft= 379 acc= 125 rate=0.330 tok/s=53.6
stepwise_math pred= 192 draft= 462 acc= 148 rate=0.320 tok/s=77.6
long_code_review pred= 192 draft= 414 acc= 139 rate=0.336 tok/s=64.2
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 3504,
"total_draft_accepted": 1272,
"aggregate_accept_rate": 0.363,
"wall_s_total": 26.68
}
```document fetched 2026-10-23 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xcqb3p/massive_b70_specdecoding_improvement_intel_xmx