Could you do HTTPS as default then offer to bump people to HTTP (and a slimmer site) if they want it? Do people with strong requirements for fast pages tend to use proxies that lighten the page (Opera used to have this built in) by reducing image sizes and such - presumably then it wouldn't matter if the original connection was HTTPS??

Would you mind sharing the site in question and the modal (ie most common) specs of your users devices?