OpenSSL for License Keys

In this article I will give an example of how one can generate and verify license keys (also known as serial numbers) using the tools included with Mac OS X, and in a way which should make it difficult for the cracker to generate his own fake license key(s).

Let us first consider what the license key actually is. Since it will be used to unlock an application, it needs to contain an identifier which the application can recognize. But giving all users the same license key (e.g. “42”) is not a good idea, since if one user leaks his license key, everybody will be able to use the program, and we won’t be able to tell which user actually leaked the license key.

To remedy this problem we can store a checksum of the users name in the license key, and let our application verify this checksum. So if a user wants to leak his license key, he also needs to include his name, which would probably discourage most.

To ensure that he cannot easily construct another name with the same checksum, we should use a cryptographic hash function to generate the checksum, and to ensure that he does not simply overwrite the checksum in his license key (with a checksum for a name he made up himself), we should give him the license key in an encrypted form, so that he would need to decrypt the license key, make the change, and then encrypt it, in order for him to spread a fake license key (i.e. with a modified user name checksum.)

Using symmetric encryption (like XOR, ROT13, or even DES/AES) the above won’t help much, since the user will only need to run our program through the debugger to see how it decrypts the license key, and from this, he can probably figure out how he should encrypt it (after making his changes to the checksum.)

But with asymmetric encryption this becomes much more difficult, because one cryptographic key is used for encryption, and another is used for decryption, and given one of them, you cannot easily find the other. You probably know this as public/private key encryption, which is currently supported by Mail.app in the form of S/MIME, where you can send a letter to your friend using his public key, and only he will be able to read it, using his private key to decrypt it.

So with these things in place, let’s try it out. We are using the openssl command which comes with OS X and supports a great number of functions related to cryptography, from their website:

The OpenSSL Project is a collaborative effort to develop a robust, commercial-grade, full-featured, and Open Source toolkit implementing the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) protocols as well as a full-strength general purpose cryptography library.

To get an idea of the functionality supported you can run man openssl from the command line.

The first thing we need is a checksum on the users name, we’ll choose SHA-1 as our cryptographic hash function, and since this is implemented by openssl, we can write the following in our shell:

% echo -n "Allan Odgaard" | openssl dgst -sha1
e9b6b54ba43bfd664cfbd2974272c0f98a3c2463

By default openssl outputs the checksum in hexadecimal notation, but you can add -binary to change that (but then you should send the output to a file instead of using the terminals standard output.)

The next step is the public/private key encryption. We will use RSA as our cryptographic algorithm. Again openssl lets us perform this, but first we need to decide the key size (in bits). The higher key size we use, the harder it is to attack the system (see notes at the end of this article for more on this). The key size dictates the minimum size of the encrypted data (i.e. our license key), so in this example we will use a key size of 248 bits (but for real world security this is not adequate, more on this later), because the checksum is 160 bits, and the default padding added in front of our data is 11 bytes (padding can be disabled using the -raw switch (when encrypting/decrypting), but the data size still needs to be a few bits less than the actual key size due to how RSA works.)

To generate a key we run the following command:

% openssl genrsa -out private.pem 248
Generating RSA private key, 248 bit long modulus
.....+++++++++++++++++++++++++++
.+++++++++++++++++++++++++++
e is 65537 (0x10001)

It generates the file private.pem which contains both the public and the private key. To encrypt something with this key, we run the following command:

% echo -n "Allan Odgaard" \
 | openssl dgst -sha1 -binary \
 | openssl rsautl -sign -inkey private.pem -hexdump
0000 - 2b 30 7e 47 c3 48 61 a2   +0~G.Ha.
0008 - 4a d5 27 9a f0 fa 2a f0   J.'...*.
0010 - c9 03 83 6e c8 f8 97 d4   ...n....
0018 - 92 7b 59 e9 04 00 4e      .{Y...N

This is a hex dump of an RSA encrypted SHA-1 message digest (checksum) of my name, using a randomly generated private key. In order to send this as a license key to the user, we could base64 encode it, which is also something openssl can help us with:

% echo -n "Allan Odgaard" \
 | openssl dgst -sha1 -binary \
 | openssl rsautl -sign -inkey private.pem \
 | openssl enc -base64
KzB+R8NIYaJK1Sea8Poq8MkDg27I+JfUkntZ6QQATg==

And now we have our license key, which can easily be transmitted using E-mail and entered (pasted) into a text field, though it doesn’t really look like a standard license key, to achieve that we could instead use base32, which consists only of A-Z and 2-7, and perhaps even insert some dashes in the string.

So now to verify this license key. As mentioned earlier, private.pem also contains the public key, so we need to extract this first:

% openssl rsa -in private.pem \
 -out public.pem -outform PEM -pubout
writing RSA key

This gives us a public key which we can embed in our program w/o the fear of anyone being able to calculate the private key from it (at least not w/o spending a lot of computer cycles doing so.)

So our application needs to base64 decode the license key, decrypt the result using our public key and then run SHA-1 on the users name and compare it with what we have decrypted.

From the shell we can base64 decode and decrypt it like this:

% echo KzB+R8NIYaJK1Sea8Poq8MkDg27I+JfUkntZ6QQATg== \
 | openssl enc -base64 -d \
 | openssl rsautl -verify -inkey public.pem -pubin \
 | xxd
0000000: e9b6 b54b a43b fd66  ...K.;.f
0000008: 4cfb d297 4272 c0f9  L...Br..
0000010: 8a3c 2463            .<$c

This is a hex dump of the decrypted license key (courtesy of xxd), and if you scroll back, you’ll see that it does in fact match the SHA-1 checksum for my name given above.

So the only thing missing is doing the above in code, since we do not want to call an external shell script to do our license key validation.

Luckily all the functionality of openssl is also available in C using libcrypto, which needs to be added to your Xcode project, or you can add -lcrypto to the linker options. Then to calculate a SHA-1 message digest on buffer with length buffer_size we do:

#include <openssl/sha.h>
⋮
uint8_t md[SHA_DIGEST_LENGTH];
SHA1(buffer, buffer_size, md);

And to “load” a public key (from memory) and decrypt a message with it (given as data with length data_size), we use the following:

#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/bio.h>

uint8_t pub_key[] = { ... };
if(BIO* bio = BIO_new_mem_buf(pub_key,
                               sizeof(pub_key)))
{
   RSA* rsa_key = 0;
   if(PEM_read_bio_RSA_PUBKEY(bio, &rsa_key,
                                    NULL, NULL))
   {
      assert(data_size == RSA_size(rsa_key));
      uint8_t* dst = new uint8_t[data_size];
      RSA_public_decrypt(data_size, data,
               dst, rsa_key, RSA_PKCS1_PADDING);
      ⋮
      RSA_free(rsa_key);
   }
   BIO_free(bio);
}

I have left out the part about base64 decoding in code, but apart from that, you should be set to experiment with this on your own!

Some final random notes:

• if you need more info about one of the functions used in the examples, simply run man «function_name» in Terminal.app, and you should get the description of the function,
• when the word key is not prefixed with license in the above, I refer to the public/private key (I hope it’s not too confusing),
• it may not seem intuitive that to encrypt we use -sign and to decrypt we use -verify. openssl does have -encrypt and -decrypt options, but -encrypt will always use the public key to encrypt the data, and -decrypt will always use the private key, since this is normally how public/private key cryptography works (i.e. you receive a message encrypted with your public key). But in our case we want the public key to be embedded in our program, so we want to encrypt with the private key, and that is basically what happens with cryptographic signatures, and that is why the options are named as they are.

On signing being the same as encrypting with a private key Carey Underwood makes the following point (May, 2009):

Because the public key is intended to be public, its creation is optimized for performance, not security. This makes it important to remember that the public key isn’t just called public, it is intrinsically and unavoidably public. Make sure you understand the implications of that before applying your understanding of RSA to other tasks.

When it comes to the choice of key size, Nicko van Someren writes:

I can factor 512-bit keys in about four weeks using a rack full of G5 X-servers, which is less resource than many university computer labs have on tap. At this time 576-bit keys have been factored and 768-bit is considered suspect for any long term security. Ultimately it boils down to how much effort you think your attacker will put in and for how long you want your software to be secure. At the moment we are still some way from anyone getting close to factoring 1024 bit RSA moduli or breaking 1024 bit discrete logs (used in DSA) so if you are willing to risk fake license keys turning up in a decade or so I’d go with 1024 bit keys.