I suppose I'm suggesting that, aside from personal scripts, we shouldn't assume too much about who our users are and what they're trying to do. The principle of least power tells us to use the dumbest format that will work, eg. plain text.
Anything we add on top of that, eg. ANSI colour codes, will be useful to some but harmful to others. The tricky part is working out which of those categories the current user is in.
So is your proposed solution not to have colored prompts? (I vehemently disagree.) Or not to put them in a .bashrc? (I still disagree, but only strongly.) Or something else?
To be precise, what you're suggesting is that we have prompts which are allowed to be colored except for the $/# at the end, because you can't color those without following them by escape characters. And that prompts must have a $/# at the end.
Anything we add on top of that, eg. ANSI colour codes, will be useful to some but harmful to others. The tricky part is working out which of those categories the current user is in.