I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)
retained text
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vilcil/i_got_tired_of_my_300gb_model_loads_taking_5min/"> <img alt="I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)" src="https://preview.redd.it/076z2bhul2ih1.png?width=140&height=140&crop=1:1,smart&auto=webp&s=05d148744010519fa5dc2d11fc72b59d972240de" title="I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)" /> </a> </td><td> <!-- SC_OFF --><div class="md"><p>- On b10173 - "state":"loading" 4min54sec.<br /> - With this PR and GGML_RPC_LOAD_THREADS 12 - "state":"loading" 1min38sec</p> <p>The PR is close to ready, will need a docs change if they want to keep the new GGML_RPC_LOAD_THREADS variable.. and hopefully they take it with the client-side being solved and someone else cant take on the remaining server effort. Getting sub minute model loads at this size would be amazing, and its not that far. </p> <p>It was mildly amusing that I developed this on a potato hardware; and the guy testing it is probably paying per hour what my whole setup costs. </p> <p>But in the Sovereign AI Wars, I code for the little guy running on 2-3 gaming PCs</p> </div><!-- SC_ON -->   submitted by   <a href="https://www.reddit.com/user/Chuyito"> /u/Chuyito </a> <br /> <span><a href="https://www.reddit.com/gallery/1vilcil">[link]</a></span>   <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vilcil/i_got_tired_of_my_300gb_model_loads_taking_5min/">[comments]</a></span> </td></tr></table>
document fetched 2026-08-08 06:58 · canonical https://reddit.com/r/LocalLLaMA/comments/1vilcil/i_got_tired_of_my_300gb_model_loads_taking_5min