please remember i’m a noob, i’m sure many people on this website can make it work but for the novice how realistic is it for AI to run on haiku ?
You will need to be more specific.
What are you trying that doesn’t works? A website, a command line program? Did you tried something already, or you are asking just in advance?
hi bipolar
thank you for the reply
i have not tried anything, i was just posing the question.
i would be interest in hearing from anyone that already is running AI on haiku !!
I don’t think we understand the question, actually. What do you mean by running AI on Haiku? Are you talking about opening up ChatGPT in a browser? Because we have browsers that will do that. Or are you talking about using haiku to run the models themselves as standalone programs?
hi dakota
yes i’m referring to running ai models offline standalone
1 Like
Offline stand-alone AI models are often heavily dependent on GPU acceleration to make the speed at least tolerable. Haiku has no graphics acceleration at this point so plan on waiting a day or two for each reply from the AI model if you even get it to work.
1 Like
Good day,
Well, partly true. I’m using Llama 3 offline on a laptop at work with 8 GB RAM and an Intel Core i5 4core 4thread, on Linux running off an external NVMe with USB 3.2, just using CPU, no GPU whatsoever.
It’s true that it’s not lightning fast, sometimes slow but then again, I used to connect to internet with 4800 baud modem back in the day. It would run better on a better processor and with more RAM, of course, it’ll depend on what you use it for. Certainly would improve speed using GPU, the better the GPU the better the performance.
On Haiku, I’ve tried to run some models also but always got stuck on the python dependencies, most of them non installable through pip and not available in the repos. But if I recall correctly, someone here in the forums has already compiled the Llama 3 model.
Nonetheless, I’ll keep on trying this on Haiku too.
Regards,
RR
1 Like
“On Haiku, I’ve tried to run some models also but always got stuck on the python dependencies, most of them non installable through pip and not available in the repos. But if I recall correctly, someone here in the forums has already compiled the Llama 3 model.
Nonetheless, I’ll keep on trying this on Haiku too.”
thank you, thats the answer i was looking for.
Maybe you could try LlamaCpp or WhisperCpp which are the ones that could give the best performance inside Haiku since they can run more optimally on CPU than their counterparts written in Python.
As for their practical use it would be debatable, but for the love of art it would be something interesting to see.
I think a plugin for Genio or some Haiku IDE that supports Github Copilot would be more useful, with this code autocompletion plugin if a model could be fine-tuned that supports the Haiku / BeOS API for example, which would help write code for Haiku or help programmers to better understand the APIs, their operation and implementation. Like asking BeBook or Haiku Book a problem and having it summarize it.
(For those who come out with the thought of “We don’t want AI for programming because that’s for weak and lazy people blah, blah, blah” or “We don’t want GitHub because it’s from the devil and steals our souls every time it lights up with the flash”… Sincerely “Vayan a ver quien los pisa”)
I think this part of your message is unwarranted. No one has so far expressed such views on this thread. Preemptively reacting to things that didn’t (and might not even) happen other than in your mind… sounds a bit off.
BTW, I’m a native Castilian speaker, and I have no clue about what "Vayan a ver quien los pisa” means. Suggestion… if you want to say something, say it explicitly, don’t hide it behind local idioms others might not understand.
2 Likes
This happens in every thread where someone wants to port something “mainstream” to Haiku. Those who consider themselves purists of computing, and see Operating Systems almost as a religion or a dogma.
I assume that they will appear in this thread because it is about “Artificial Intelligence” which is a controversial topic between old school programmers, and new programmers.
More so if I mention GitHub Copilot, since… GitHub = Microsoft = Devil.
Personally, I would like to see some AI tool integrated into Genio, some language model tuned with Haiku/BeOS documents, it would improve the integration of new programmers with the ecosystem, it would serve to increase interest in people for whom this is only a hobby.
Well, my suggestion is… try at least to wait until such comments actually show up, before you let your “righteous outrage” come out. Otherwise, you’re are only contributing to creating such discussion/outrage that might never happen otherwise.
4 Likes
To me it sounds like you are trying to actually incite such controversy. Please refrain from doing so. We have enough heated discussion around here without anyone actively trying to incite them.
7 Likes
Yes, you are absolutely right. 
1 Like
thank you for the response, but your answer was to technical for noobs
I’m trying to get GLiNER.cpp working to integrate entity recognition into my little editor SENity for my project SEN. For using it with actual models though it requires the ONNX library, which is a bit more hassle to get working.
With the help of @Begasus we almost have a working port now but it’s not done yet and complains about missing functions on initialization.
I hope to get back to it early next year, hopefully with support from @Begasus , since I want to focus more on the editor and SEN and this is just a use case for me…
Hey, I’m not the developer here! 
But happy to help out where I can.
Had some talks the other day with @coolcoder613 about Ollama, already got a recipe for it, and a Alpaka recipe, but never got anything working with them 
1 Like
I just got the openai and ollama python libraries working on Haiku, to connect to the openai api, or ollama running on a different computer.
What I did was build wheels for dependencies of openai/ollama.
Here they are: https://0x0.st/8rHR.zip
2 Likes
I have built and successfully used llama.cpp for Haiku. With 32 GB of RAM and a Rizen7 CPU, these models work fine for me:
- Vikhr-Gemma-2B-instruct-Q5_K_M.gguf
- vikhr-7b-instruct_0.4.Q4_1.gguf
- gemma-1.1-7b-it.Q4_K_M.gguf
- gemma-2-27b-it-Q4_K_L.gguf
2 Likes
