Your servers do not offer SSL session resumption. This means that every response contains a server certificate (3807 bytes) and your server has to perform a 2048-bit RSA decryption. This occurs for every piece of data fetched over SSL, even the tiny button pictures that are smaller than the certificate itself.
You should really enable SSL session resumption. It will save a lot of money in server cost and bandwidth, and your users will be happier too.
[Edit: WordPress staff replied that this was a mistake in their configuration and now this is fixed.]