Transformers now supports Mixtral GPTQs and I've updated my READMEs accordingly. It was awesome working with @_marcsun and @younesbelkada of @huggingface on this!
Credit to LaaZa for coding the AutoGPTQ quant and inference implementation which enabled me to get GPTQs out fast!
Announcing 4-bit Mixtral 8x7B on 🤗Transformers!
Run the new Mistal MoE with minimal performance degradation on your local computer (24Go) 🔥
Stay tuned as more quants are coming soon using AWQ. We are also looking into sparsification with @Tim_Dettmers
huggingface.co/TheBloke/Mixtr…




