Control theory assumes the system is linear. As soon as you get saturation the system stops being linear.

So in trad undergrad control theory instability implies "And then the system blows up" - numerically, literally, or sometimes both.

But depending on the system you can end up in regions of recursive instability which are better modelled by logistic/chaos theory:

https://en.wikipedia.org/wiki/Period-doubling_bifurcation