Programming with Cryptographic Libraries: OpenSSL

1. Using OpenSSL's Toolkit.  In this section, you will learn a little bit more about what the OpenSSL toolkit can do by using its command line tool.  On UNIX machine, open a terminal or an X terminal (xterm) and on Windows XP machine, simply open a DOS terminal using command 'cmd'.  Navigate to where the OpenSSL.exe is stored and type 'openssl' at a prompt  to open the OpenSSL prompt.  Then type 'help' to see a list of common commands. 

This shows you what standard and cipher commands are available for openssl command line tool. The available commands are varied from the OpenSSL version.

Now you will encrypt a file using AES with 128-bit key in CBC mode, and decrypt the encrypted file using the openSSL tool.
First, you will need to create a file, in this tutorial the file will be called plaintext.txt.  Create a text file with the contents "Hello World."

To see how you use openSSL with encryption mode, type openssl enc help.

To encrypt the plaintext.txt file use the openssl enc -nosalt -e -aes-128-cbc -k key -in plaintext.txt -out ciphertext.txt command. 

The encryption command is broken down for a better understanding.  You will encrypt the plaintext.txt file using the –e option and –aes-128-cbc for AES with 128-bit key in CBC mode. You will also have to specify a key using the –k option, which has to be followed by a key. Then, specify the input and output file using –in <filename> and –out <filename> options. The –nosalt option is used for not including the salting function that is commonly used in password-based encryption.

Open the ciphertext.txt file and see if you can read it.  The context should be hard to identify due to the encryption.  Using a PC the ciphertext.txt file opened in Notepad will look similar to the following image.

Normally the ciphertext is encoded in a readable form, which is accomplished by using Base64 encoding.  This will encode the context of the file using only human readable characters, which consist of uppercase and lowercase letters, numbers, and some punctuation.  Now encrypt the plaintext.txt file using Base64 encoding.  You will use the same command with the addition of the basee64 option.  To encrypt your file use openssl enc –nosalt -e -aes-128-cbc -k key  -in plaintext.txt -out base64cipher.txt –base64

Open the base64cipher.txt file to view the affect the base64 encoding option has on the ciphertext.  Using a PC the ciphertext.txt file opened in Notepad will look similar to the following image. 

To decrypt the ciphertext you need to know the key, encryption mode, and use the Base64 encoding option.  Use the following command to decrypt the ciphertext.  openssl enc –nosalt -d -aes-128-cbc -k key -in base63cipher.txt -out decryptedtext.txt -base64

After you decrypted the ciphertext open the decryptedtext.tx file to see if the contents are "Hello World."  Before you continue with this tutorial here is a question to test your encryption knowledge.  If you noticed, your plaintext.txt file is 11 bytes (10 bytes for 10 letters for "Hello World" including the space and 1 byte for EOF character), and the encrypted file, base64cipher.txt file, is 25 bytes long. Should the ciphertext.bin be 11 bytes the same size as the plaintext file? Why is it 25 bytes long? Hint: it is about the encryption mode you used and AES. If you don’t know the answer, it is okay.

2. OpenSSL Programming.  This section will introduce you some OpenSSL APIs that you will use for programming your software. You will write your first encryption program, which will be an encrypted "Hello World?" program.  You will use the OpenSSL crypto library to compile your program.  When using the OpenSSL crypto library you will have to specify the openssl headers in your program.  In this example, you will use RC4 to encrypt a message, "Hello World"; therefore, you need to include the  <openssl/rc4.h> header in your program.

#include <openssl/rc4.h>

Now you need to know how to use RC4 functions to set up a key and to encrypt a message. If you are using SIS workstation, type in man rc4, and it will show you how to use it. However, this will not work for all ciphers. OpenSSL is known to lack good documentation.  It is recommended that you go to the source directory of RC4 crypto library and look at the rc4.h header file. In that directory, there are also some examples of how to use the library functions.  If you are using a SIS workstation use cd/opt/local/sources/packages/openssl-0.9.7e/crypto/rc4 where you will find the header file along with two test programs that use rc4 functions.  If you are using a PC to view the rc4 header file navigate to the OpenSSL directory where you will find an include folder, which is where you will find rc4.h.  The default install location is C:\OpenSSL\include\openssl.  The header file defines the rc4 key structure and two functions.  The two functions are briefly explained below.

  • void RC4_set_key(RC4_KEY *key, int len, const unsigned char *data);  which is the function for setting up an RC4 key. It requires a RC4_KEY pointer, the length of the key and the key data.
  • void RC4(RC4_KEY *key, unsigned long len, const unsigned char *indata, unsigned char *outdata); which is used for both encryption and decryption

The RC4 cipher is a stream cipher which encrypts a message byte-by-byte. Basically, RC4 generates a pseudo-random byte stream (rc4_bn) and use XOR to either encrypt or decrypt message bytes (msg_bn). Thus, RC4 encryption and decryption functions are similar.

Here is the program for the encrypted "Hello World?"

1 #include <stdlib.h>
2 #include <openssl/rc4.h>
4 int main(void)  {
6      char plaintext[12] = "Hello World?";
7      char ciphertext[12];
8      char mykey[16] = "this's my secret";
9      RC4_KEY rc4_key;
11     RC4_set_key(&rc4_key, 16, mykey); /* the key is 128 bits or 16 bytes long */
12     RC4(&rc4_key, 12, plaintext, ciphertext); /* plaintext is 12 bytes long */
14     printf("%s\n", plaintext);
15     printf("Encrypted: %s\n", ciphertext);
17     RC4_set_key(&rc4_key, 16, mykey);  /* reset the key stream */
18     RC4(&rc4_key, 12, ciphertext, plaintext); /* start decryption */
20     printf("Decrypted: %s\n", plaintext);
23     exit(1);
22 }

If you noticed pointers were not used at all, instead an array of characters were used.  Using pointers in secure programming creates the possibility an attacker can gain access to your machine by using a buffer overflow.  Additionally, you maty have a segmentation fault when you run your program if you do not handle the pointer safely.

The key handler is defined in line 9 RC4_KEY rc4_key;  the RC4 key will be set up using the RC4_set_key(&rc4_key, 16, mykey); function, which is done on line 11.  Now you can start the encryption using the RC4(&rc4_key, 12, plaintext, ciphertext); function, which is done on line 12.  The output, which is the encrypted text, will be placed in a file called ciphertext. 

To decrypt the ciphertext you need to set the key again because if you do not you will get bytes differently from the key stream for since the key stream will continuously produce the key bytes. Thus, to reset the key stream to ensure you will using the bytes that represent your key use the RC4_set_key() function, which is done on line 17.  To start the decryption process, call the RC4( ) function again, but at this time, your input is the ciphertext and the output is plaintext.

Remember when writing a program that uses OpenSSL be careful when using pointers.  If you use a character array, be sure to specify its size correctly. Also be sure that you understand how the encryption and decryption modes work especially when using CBC, ECB, CFB, and the streaming mode as you did in this example. There are several ways to programming with OpenSSL ranging in complexity. What you use here is very primitive, which gives you more flexibility for programming, yet mistakes can happens very easily. In the next section, you will learn more about using OpenSSL crypto function for different security services.

Now you must compile and run your program.  If you are using a SIS workstation use the gcc -g hello.c -o hello -I/opt/local/sources/packages/openssl-0.9.7e/include –lcrypto command.  You will have to include the path to the OpenSSL header file directory.  Do not forget to include the crypto library, which is -lcrypto. 

You can use an environment variable to set up this path so you will not have to type long pathname. If you are using bash shell you can use the export command to set an environment variable, which will be the path to OpenSSL.  Use export SSLINC=/opt/local/sources/packages/openssl-0.9.7e/include to set this environment variable.  After setting the environment variable you can see the ease to compile your program by using gcc -g hello.c -o hello –I $SSLINC –lcrypto

3. Encryption and Decryption using OpenSSL API.  In last section, you learned how to use OpenSSL crypto functions in a very primitive way. That is you used low-level functions in OpenSSL to encrypt and decrypt a message  To use this method you need to have some knowledge about how each encryption mode is used and how the cipher works.  There is also the possibility of creating run-time errors very easily.  However, OpenSSL provides a better APIs that requires programmers no knowledge of symmetric cryptography. You need to only specify what cipher you want to use and in what mode the cipher will be used. The OpenSSL API is called EVP API, which can be accessed by including the <openssl/evp.h> header file in your program.  By using the EVP, you do not need to load or prepare necessary objects. The EVP will automatically load them in the background. Below is a partial list of some ciphers and modes you can use with EVP.  The complete list of the EVP APIs can be found in evp.h file.
Cipher Mode Key Size (bits) Cipher EVP Call String for Cipher Lookup
ECB 128 AES EVP_aes_128_ecb() aes-128-ecb
CBC 128 AES EVP_aes_128_cbc() aes-128-cbc
CBC - Blowfish EVP_bf_cbc() bf-cbc
CFB - Blowfish EVP_bf_cfb() bf-cfb
CBC - CAST5 EVP_case_cbc() cast-cbc
CBC - DES EVP_des_cbc() des-cbc
CBC (3 keys) - 3DES EVP_des_ede3_cbc() des-ede3-cbc
CFB (2 keys) - 3DES EVP_des_ede_cfb() des-ede-cfb
- 40 RC4 EVP_rc4_40() rc40-4-
- 128 RC4 EVP_rc4() rc4
CBC 128 (12 rounds) RC5 EVP_rc5_32_16_12_cbc() rc5-cbc

Now you will use the same example of the encrypted "Hello World?" program, using the EVP APIs. You will also extend your program to use different ciphers and modes. Here is the new program.

1 #include <stdlib.h>
2 #include <openssl/evp.h>
4 int main(void) {
6       char plaintext[1024] = "Hello World?";
7       char ciphertext[1024];
8       char mykey[EVP_MAX_KEY_LENGTH] = "this's my key";
9       char iv[EVP_MAX_IV_LENGTH] = "my iv";
10     int in_len, out_len=0;
11     EVP_CIPHER_CTX ctx;
13     in_len = strlen(plaintext);
15     printf("No encrypt: %s\n", plaintext);
17     EVP_EncryptInit(&ctx, EVP_rc4(), mykey, iv);
18     EVP_EncryptUpdate(&ctx, ciphertext, &out_len, plaintext, in_len);
19     EVP_EncryptFinal(&ctx, &ciphertext[out_len], &out_len);
21     printf("Encrypted : %s\n", ciphertext);
23     EVP_DecryptInit(&ctx, EVP_rc4(), mykey, iv);
24     EVP_DecryptUpdate(&ctx, plaintext, &in_len, ciphertext, out_len);
25     EVP_DecryptFinal(&ctx, &plaintext[in_len], &in_len);
27     printf("Decrypted : %s\n", plaintext);
29     exit(1);
30 }

On line 1-2, the header files are specified where evp.h is used for the EVP API. On line 6-11, parameters are defined.  You need to add the IV (Initial Vector) as it is required by the EVP API.  The IV may not always be used for a default RC4 encryption requires no IV. The ctx parameter on Line 11 is used to define the context for this ciphering. It will be a cipher handler variable that you will use, for example, to set up a key size, to query the cipher mode, or to specify the encryption control.

Using the EVP API requires three phases, Initial, Update, and Final phase. The Initial phase you will set up the necessary parameters such as the cipher, the mode, the key and the IV. The Update phase is starts the encrypting or decrypting.  For example, a block cipher requires the data amount to be more then the block size before encryption can begin.  The Final phase will begin after the encryption has begun and when the size of the data left to encrypt is smaller than the block size.  For example, the block size for AES is 128 bits and for RC5 is 64 bits. When the encryption is at the last encryption cycle and the remaining message is shorter than the required block size, the Final phase will pad the remaining message up to the block size and perform the last encryption. The padding follows the standard padding called PKCS#5.  Lines 17-19 show the Initial, Update and Final function for encryption.

Lines 23-25 show the Initial, Update and Final phase for decryption. However, in this example, RC4 is a stream cipher and it can start encrypting or decrypting at any byte length. You are not required to use EVP_EncryptFinal and EVP_DecryptFinal for RC4 and you can remove it and see whether your program produce the same result.

Lines 27 - 28 prints out the decrypted text and exits the program.  For more information about using the EVP API, you should go to or get a book, Network Security with OpenSSL from O'Reilly.  However, you should view the source code since it is not up-to-date.

To compile the program use gcc -g hello2.c –o hello2 –I /opt/local/sources/packages/openssl-0.9.7e/include -lcrypto –lsocket.  The EVP API requires a socket library (libsocket.a) so when you compile, do not forget to add it in the command line.