But won't it add an additional hop hence additional latency to every single request ?

Nah, because the Cloudflare Workers run at closest edge location and are real fast.

The real solution with Cloudflare is to use their Load Balancing (https://developers.cloudflare.com/load-balancing) which is a paid feature.