Not necessarily. Once you enable and optimize your TLS stack you're also well on your way to deploying HTTP/2. Unlike HTTP/1.1, HTTP/2 requires only a single connection per origin, which means fewer sockets, memory buffers, TLS handshakes, and so on. As a result, it may well be the case that you will be able to handle more users with fewer resources.
That applies to modern hardware and modern infrastructue – not to people in third world countries on decade old cheap hardware with barely working networks.