I notice that language models are much better at programming than at writing. This is largely because, in programming, they have a closed feedback loop that allows them to find errors. Writing is much more complicated because the feedback is much more diffuse and delayed over time. In fact, it is often not well-defined.
What is the function that determines whether a poem is good? This means that models do not know how to think; they lack judgment. If they do not know how to think, it means that we humans are still essential. In fact, even when programming, you have to stay right on top of them. My conviction is that this will not change.
They will become increasingly useful, but they will never be able to replace the human who defines objectives and evaluates the quality of the work. All these news stories claiming that AI is going to kill us all are stupid. Humans will always be able to turn off the machine.
In three years Cyberdyne will become the largest supplier of military computer systems. All stealth bombers are upgraded with Cyberdyne computers, becoming fully unmanned. Afterwards they fly with a perfect operational record. The Skynet Funding Bill is passed. The system goes online on August 4th, 1997. Human decisions are removed from strategic defence. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14am, Eastern time, August 29th. In a panic, they try to pull the plug.