I wouldn't be surprised to see an increasing number of (express-type) roads or perhaps dedicated lanes where human drivers were not allowed on though, after self-driving capabilities become the norm. (yes I realise that's an assumption).
I think people underestimate the cultural impact that self-driving vehicles will have - imagine a whole generation or two after self-driving vehicles are generally available - how many people will bother learning to drive? I think it might become more of a job-specific skill than a general 'adult life' skill as it is now in most places.
I think you are absolutely right about some limited circumstances that make them the only legal option. But the analogy that I keep making is to classic cars. A lot of them don't have the safety features that we expect today. It isn't uncommon for their owners to say things like "I'm only safe on roads that existed in 1960". It is obviously an exaggeration, but the point is that even today there are plenty of cars that are legal to operate but probably wouldn't be anyone's preference on a busy 70 MPH interstate.
At some point, human driven cars become novelties, just like that. There is no reason to ban them, but as you suggest, maybe there will be some HOV-like lanes where they don't really have access. Or even some time constraints (not during rush hour on some key roads, not in lower manhattan, etc).
I think people underestimate the cultural impact that self-driving vehicles will have - imagine a whole generation or two after self-driving vehicles are generally available - how many people will bother learning to drive? I think it might become more of a job-specific skill than a general 'adult life' skill as it is now in most places.