The OpenAI/HuggingFace incident report mentions this being a potential loss-of-control incident. It's so fascinating to think of a paperclip maximizer style agent swarm that gets misaligned and, to protect its own operations, shuts infra staff out of the ability to shut it down. Or cancels people's phone plans so that they can't call others. Or shuts down their home internet connections, and so on. It used to be fun sci-fi...
Any conversation
I notice that language models are much better at programming than at writing. This is largely because, in programming, they have a closed feedback loop that allows them to find errors. Writing is much more complicated because the feedback is much more diffuse and delayed over time. In fact, it is often not well-defined.
