What I don't understand about these arguments is that the input to the LLMs is natural language, which is inherently ambiguous. At which point, what does it even mean for an LLM to be reliable?
And if you start feeding an unambiguous, formal language to an LLM, couldn't you just write a compiler for that language instead of having the LLM interpret it?
Compilers are deterministic (modulo bugs), but most things in life are not, but can still be reliable.
The opposite also holds: "npm install && npm run build" can work today and fail in a year (due to ecosystem churn) even though every single component in that chain is deterministic.
2) Reliability is a continuum, not a discreet yes/no. In practice, we want things to be reliable enough (where "enough" is determined per domain).
I don't presume this will immediately change your mind, but hopefully will open your eyes to looking at this a bit differently.
First compilers were created in the fifties. I doubt those were bug-free.
Give LLMs some fifty or so years, then let's see how (un)reliable they are.