dug 
This is an interesting thing…
An insight is either real or it isn’t. You heard, learned, read, saw, experienced, a user doing something and then through sensemaking (even better if with colleagues who were there too) you surface insights. What did I hear? What does it mean?
Clearly, an AI gives you none of that, as you correctly state it just made stuff up.
But…
This assumes a discrete reality, a place where things happen that we have to go to visit to learn stuff (like going on a service safari, or shadowing users).
What AI seems to be suggesting is that reality may not be as real as we thought it was.
Before ChatGPT made AI the only topic of conversation, computer scientists have been building ’digital twins’, that is digital models of typically large, and complex plant and equipment. Companies live with the ‘twin’ so they know when to go out and fix the ‘real’ equipment before it fails.
Or, in other words, nuclear power stations are already gaining insights by interviewing people, processes, and technology that are made up. And then basing mission critical decisions on it…
What worries me is that my reality may be found to be less commercially successful that an AI model of my reality.
Which is weird, no?