Speed measurement (so-called "benchmarking") is among the first actions that each developer performs when evaluating any communication library. However incomplete, inadequate, or inaccurate, benchmarking is one of the factors that lead to a library's selection. What steps should be followed and what questions should be answered to ensure that the measurement is correct and reflects a real situation?

Background

Before defining a benchmarking strategy, one needs to understand some SSL basics. SSL (Secure Socket Layer), is a layer on top of TCP in the OSI stack of network layers. It is generally possible to use SSL with any protocol that keeps packet order and does reliable delivery, but most implementations are socket based. The application or internet protocol developer can write a data exchange and put it on top of SSL, thus getting data security and authentication transparently. In most cases, it is possible to "insert" SSL between an application-level protocol and TCP.

SSL-secured communication consists of 3 steps:

1. SSL handshake:: During this step, parties exchange security information (certificates, session keys) and define whether the actual data exchange can take place. While security information is being exchanged, keys for asymmetric encryption algorithm are generated. The generation of keys is a slow operation.


2. Encrypted data exchange: During this step, parties encrypt the application-level protocol data into secured packets and send these packets to each other. Symmetric encryption algorithms, which are fast, are used to encrypt.


3. "Goodbye" step (optional): Most applications ignore this step and just close the internet connection. However, this step can be used in the future to perform a "session resume" operation, which saves time on subsequent connects (due to the reuse of pregenerated asymmetric keys).

SSL can use several symmetric and asymmetric algorithms. Some algorithms are faster than others. SSL always uses one symmetric encryption algorithm, one asymmetric encryption algorithm, and one hashing algorithm. The set of algorithms used is called the "cipher suite". The list of cipher suites to use is specified by the application. The SSL implementation then tries to select the right cipher suite from the list using the information received from the other party. Certain cipher suites can be used only when the certificate is specified for the party; other cipher suites are used for ad-hoc security (using the Diffie-Hellman algorithm to exchange the session key).

There exist several versions of SSL - SSL2, SSL3, and TLS1. These can use different cipher suites. For example, SSL2 can't use anonymous cipher suites - the server side of the communication must have a certificate.

Benchmarking Strategies

Consider the following when defining or evaluating a benchmarking strategy:

1. Don't compare apples to oranges. If you measure the speed of different libraries, always specify the same version of SSL and the same cipher suites for all libraries. When selecting the cipher suites, take into account the design of your system, e.g., whether client and/or server certificates are used. Remember, that SSL2 is outdated and a less secure version should be used only for fallback modes, when the unknown party doesn't support better versions.


2. Define what kind of data your system is going to transfer. If your system is supposed to transfer small data blocks (for example up to 8Kb), the speed of the handshake is more important than overall speed. To measure the speed of the handshake, you just need to establish an SSL connection and forcefully close the connection right after the SSL connection is established. Don't transfer any data at all. Remember to check all cipher suites that can be used in your system - different implementations of SSL have different speed on different cipher suites. On the other hand, if your system is about to transfer larger data blocks or even huge files, then the speed of the handshake shouldn't be considered a factor that should be taken into account.


3. If you perform certificate validation in code, turn it off. Your code can significantly influence the results.
4. Do not perform one-time measurements. Each measurement should be taken several times (the more the better) and averages should be calculated. Also, add random intervals between measurement cycles: Some SSL implementations might cache some information and random intervals will show "clear performance" without caching.


5. Check if the library supports the goodbye step and resuming sessions. Resuming sessions can save time in the handshake phase. This is especially important for multitier applications without persistent connections (connections that establish a separate connection for each request).


6. Check what platform the library is written for and how it is executed: implementations written in .NET managed code or Java byte-code will be slower than native-code implementations.

After you've completed your tests, throw them away -- in real life, all native SSL implementations have similar speeds that can vary in particular scenarios, but in general libraries are very close in speed. Consider speed in context with security and ease of use (which leads to better security): If your ultimate goal is to save a fraction of a second per connection, then you will lose time and money patching your code, improving security, and avoiding bugs in fast and insecure implementations. Remember, SSL is Secure Socket Layer and not Speedy Socket Layer.