What matters is the climate equilibrium that we and the ecosystems we depend on evolved under. Most of Earth's history would be totally uninhabitable for humans ourselves, nevermind those ecosystems. Using the argument of the total time that Earth has existed or previous climate fluctuations totally misses the point.

It doesn't matter what happened millennia ago, what matters is how rapidly things are changing now.