I have built and successfully used llama.cpp for Haiku. With 32 GB of RAM and a Rizen7 CPU, these models work fine for me:
- Vikhr-Gemma-2B-instruct-Q5_K_M.gguf
- vikhr-7b-instruct_0.4.Q4_1.gguf
- gemma-1.1-7b-it.Q4_K_M.gguf
- gemma-2-27b-it-Q4_K_L.gguf
3 Likes