No, but that doesn’t prevent people from thinking it should be solved (or it doesn’t exist) and the rules can be discussed, and either changed, or better understood by everyone. As far as I know, this wasn’t really done for the current rule. Let’s say it was set in an emergency in face of the risk of copyright/license tainting, and so for that reason the normal process (discussion on the mailing list or forum) was skipped or expedited quickly.
The result is that there can still be disagreement about it, as no consensus was reached.
That’s why we should have this discussion properly.
My experience with code review is that it is considerably more effort to properly review code than to write it in the first place. The activity on Gerrit seems to agree with that: lots of people submitting new changes. Few people reviewing the changes, making the fixes, getting things finetuned so they can be merged.
Doing code review on generated code is even harder. Due to the way the LLMs work, the code will always look nice at a first glance, well formatted, neatly commented, and so on. But there can be logic errors in the tiny details, and there can be sometimes a lack of understanding of the surrounding code architecture (which can lead for example to not reusing an existing function, but writing a new one to do the same thing).
These issues are not easy to see. When I review code written by a human and in good faith, I can usually tell by indirect signs where to look more closely for problems. Maybe a comment disagrees with the code. Maybe the naming for something isn’t so well chosen, which indicates that maybe the concept wasn’t fully thought through or fully understood. And when I have questions, I can ask the person who wrote the code why they did things in one way rather than another.
When reviewing code that was largely written by LLMs, this review strategy doesn’t work anymore. I have to use a method I would normally use for an adversarial code review, like if someone was intentionally trying to sneak in a vulnerability, and I’d be trying to find it. This is something I’m not good at. I trust the person who wrote the code too much. I assume they are trying to do what they say. Some of my paid job colleagues are much better than me at this. I feel so not ready for it. And to add to that, we could easily be drowned under low quality pull requests, as some other projects participating in GSoC are this and last year. I’m not sure what shielded us from that. Maybe the ban on LLM generated contributions. Maybe the fact that we’re not on Github and that was enough to stop these low-effort GSoC attempts. Anyway, I would rather keep it that way, and the proposals we got so far look better than in previous years.
Now, all of this is, of course, mainly about entirely “vibe-coded” things. There is a wide range of ways to write code from that to “I write the code directly in hexadecimal because I don’t trust the assembler/compiler” (I know some people who can do this, too).
So the question can be, where do we put the limit? I would say we can eliminate the extreme ends of the spectrum: vibe-coded slop is not allowed, and we are allowed to use a text editor and a C++ compiler. So the right answer is somewhere in between. I would say auto-completion by a traditional language server or clangd based system is allowed (that’s not LLMs, of course). But then there’s a whole range of things that may be debatable. At which point is the limit between auto-completion and vibe coding? But the answer to that very much depends on the reasons to encourage or not the use of LLMs.
If the main worry is copyright and licenses, auto-completing a few lines of code is probably OK, but an entire function is already a bit risky.
If the main worry is code quality, you probably want to keep it to a pretty low level as well. Something you can carefully review and understand. Maybe one function at a time is OK, more than that, I think the developer “writing” the code is not fully in control anymore, and you start putting more work on the person who will do the code review.
If the main worry is natural resources usage or DDoSing of servers, the right tim to stop would be before training the model at all. But, as x512 pointed out, this is currently outside of the scope of Haiku, due to the “no politics” rule. I would prefer for this to change, but there’s enough heated debate already at the moment here, so let’s keep it at the purely technical and legal arguments.
If it’s about not relying on closed source tools that may someday be unavailable, we’d have to research self hostable options (there are some). But I think this aspect wasn’t discussed as much in this thread, so I assume it is less important. In Haiku we don’t currently have a rule against use of closed source tools like Github, for example. So introducing one specifically for LLMs would be strange (although personally I would also say that would be a good idea, I’m not going to push for it at this time, same as above).
If it’s about the pleasure to work with other people as a team, help each other and learn together, rather than with machines, I don’t see why one would want to use LLMs. Maybe I’m missing something. I’d rather ask a fellow human.
And if it’s the personal pleasure and pride of having written the code all by oneself, that probably doesn’t belong to a project-wide policy.
I’m not sure if I’m missing other arguments. The thread is quite long and I’m not going to re-read it all.