Crypto song. Take a listen and enjoy! Harry Belafonte never sounded this good. ;-)

ESAPI 2.0 Symmetric Encryption User Guide

ESAPI.properties Properties Relevant to Symmetric Encryption

Those properties that are new since ESAPI 2.0-rc2 are shown in red. Values shown in blue are ones that you would replace.

Property Name

Default Value

Comment

ESAPI.Encryptor
org.owasp.esapi.reference.crypto.JavaEncryptor

The class implementing the Encryptor interface and returned by ESAPI.encryptor().

Encryptor.MasterKey

<initially unset>

The base64-encoded SecretKey. The key must be appropriate to the specified key size and cipher algorithm.

Set as per the instructions in the ESAPI Installation Guide.

Encryptor.MasterSalt

<initially unset>

A base64-encoded random “salt”. This should be at least 20-bytes. It is used to generate a random (but consistent) public/private key pair used in asymmetric encryption and digital signatures.

Set as per the instructions in the ESAPI Installation Guide.

Encryptor.EncryptionAlgorithm
AES

A deprecated property, superseded by Encryptor.CipherTransformation.

Encryptor.CipherTransformation
AES/CBC/PKCS5Padding

Specifies the cipher transformation to use for symmetric encryption. The format is cipherAlgorithm/cipherMode/paddingScheme.

Encryptor.EncryptionKeyLength
128

Key size, in bits. Required for cipher algorithms that support multiple key sizes.

Encryptor.ChooseIVMethod
random

Legal values are “random” or “fixed”. Random is recommended. Set to “fixed” only if required for compatibility with legacy or third party software. If set to “fixed”, then the property Encryptor.fixedIV must also be set to hex-encoded specific IV that you need to use.

CAUTION: While it is not required that the IV be kept secret, encryption relying on fixed IVs can lead to a known plaintext attack called a "Key Collision Attack". While this attack is probably not practical (for those with modest resources) for ciphers with 128-bit key size, this attack makes it possible to capture the ciphertexts from only 2(N/2) known plaintexts to discover the encryption key. Loughran and Dowling explain a Java implementation of Eli Biham's key collision attack on DES in their easy to understand paper, A Java Implemented Key Collision Attack on the Data Encryption Standard (DES). Since attacks only get better and the cost of storage is dropping rapidly, you are urged to avoid using "fixed" IVs except when required for backward compatibility. In particular, should never use fixed IVs just to avoid the storage cost of storing a random IV.

Encryptor.fixedIV
0x000102030405060708090a0b0c0d0e0f

A hex-encoded value to use as a fixed IV. Only used if the property Encryptor.fixedIV is set to “fixed”. Intended only for compatibility with legacy code. See the above related above caution for Encryptor.ChooseIVMethod.

Encryptor.CipherText.useMAC
true

Whether or not CipherText should use a message authentication code (MAC) with it. This prevents an adversary from altering the IV as well as allowing a more fool-proof way of determining the decryption failed because of an incorrect key being supplied. This refers to the "separate" MAC calculated and stored in CipherText, not part of any MAC that is calculated as a result of a "combined mode" cipher mode.

Note: If the cipher mode used is one specified in the comma-separated list of cipher modes given in the property Encryptor.cipher_modes.combined_modes, then a separate MAC is not calculated for CipherText regardless of the setting of this property. (Doing so would be superfluous.)

Encryptor.PreferredJCEProvider
<empty string>

Specifies the preferred JCE provider, that is the JCE provider that is first looked at for JCE algorithms. The Encryptor reference implementation, JavaEncryptor, attempts to load this JCE provider at position the first position when the JavaEncryptor class is first loaded.

The value may either be a provider name (e.g., “BC” for Bouncy Castle) or the fully qualified class name implementing java.security.Provider for the desired JCE provider. If left set to the empty string (the default) or unset, the effect is to not change the preferred JCE provider so that your application ends up using whatever your Java VM is already using, which is generally determined by the settings in your $JAVA_HOME/jre/lib/security/java.security file.

Encryptor.cipher_modes.additional_allowed
CBC

Additional cipher modes allowed for ESAPI 2.0 symmetric encryption. These cipher modes are in addition to those specified by the property Encryptor.cipher_modes.combined_modes.

Note: We will add support for streaming modes like CFB & OFB once we add support for 'specified' to the property Encryptor.ChooseIVMethod (probably in ESAPI 2.1).

Encryptor.cipher_modes.combined_modes
GCM,CCM,IAPM,EAX,OCB,CWC

Comma-separated list of cipher modes that provide both confidentiality and message authenticity. (NIST refers to such cipher modes as "combined modes" so that's what we shall call them.) If any of these cipher modes are used then no MAC is calculated and stored in the CipherText upon encryption. Likewise, if one of these cipher modes is used with decryption, no attempt will be made to validate the MAC contained in the CipherText object regardless of whether it contains one or not. Since the expectation is that these cipher modes support support message authenticity already, injecting a MAC in the CipherText object would be at best redundant.

Note that as of JDK 1.5, the SunJCE provider does not support any of these cipher modes. Of these listed, only GCM and CCM are currently NIST approved.

Encryptor.PlainText.overwrite
TRUE

Whether or not the plaintext bytes for the PlainText object may be overwritten with “*” characters and then marked eligible for garbage collection. If not set, this is still treated as 'true'. If this is set to 'true', you will not be able to use any PlainText object after you have used it with one of the Encryptor encrypt() methods.

Encryptor.KDF.PDF
HmacSHA256

This is the Pseudo Random Function (PRF) that ESAPI's Key Derivation Function (KDF) normally uses. NSA wanted us to support something stronger than HmacSHA1 (even though that is considered fine for now--unless perhaps you are guarding nuclear launch codes--and if so, your Java license probably prohibits that ;-).

(Note this is *only* the PRF used for ESAPI's KDF and *not* what is used for ESAPI's MAC. Currently, HMacSHA1 is always used for the MAC.)

Currently supported choices for JDK 1.5 and 1.6 are: HMacSHA1 (160 bits), HMacSHA256 (256 bits), HMacSHA384 (384 bits), and HMacSHA512 (512 bits).

Note that HMacMD5 is *not* supported for the PRF used by the KDF even though these JDKs support it. ESAPI 2.0 release candidates prior to 2.0_rc11 ALWAYS used HMacSHA1. It is somewhat faster than the SHA2 HMAC variations, but not significantly so.

The only legitimate reason to tweak this would be to change it to one of the new HMACs based on the future SHA-3 winner once it is announced by NIST and supported in ESAPI and most JDKs. Until then, don't meddle.

Encryptor.CharacterEncoding
UTF-8

The default encoding used for certain aspects such as signing and sealing.

How the Old (Deprecated) Methods Were Used

To encrypt / decrypt using the String-based, deprecated methods carried over from ESAPI 1.4, code similar to the following would be used.

    String myplaintext = "My plaintext";
    try {
        String ciphertext = ESAPI.encryptor().encrypt(myplaintext);
        String decrypted  = ESAPI.encryptor().decrypt(ciphertext);
        assert decrypted.equals(myplaintext);
    } catch(EncryptionException ex) {
        // Log error then return error designation however appropriate
    }

This code will still work, however if you are using the standard (default) reference for ESAPI.Encryptor, which is org.owasp.esapi.reference.crypto.JavaEncryptor, the cipher transformation used with be that specified by the property Encryptor.CipherTransformation with a key size (when the algorithm supports a variable key size) of that specified by Encryptor.EncryptionKeyLength and the IV type specified by Encryptor.ChooseIVMethod. What is not provided by these methods (and why they are deprecated) is that they provide no mechanism to ensure message authenticity unless they are used with a so-called “combined” cipher mode such as CCM or GCM. (Note that as of JDK 1.6, the default JCE provider, “SunJCE”, does not support any combined cipher modes.)

Requirements for using ESAPI Symmetric Encryption

The parties participating in using ESAPI's symmetric encryption capabilities must agree on the following:

  1. All parties must share the same encryption key(s). If multiple keys are used, each party must know which key to use for encryption / decryption. Parties must ensure that exchanging, storing, and all management of these encryption keys is done securely. How this is done is presently outside the scope of ESAPI encryptions, but developers may find the advice in OWASP's Cryptographic Storage Cheat Sheet helpful in this regard.
  2. If parties are only exchanging raw ciphertext (plus IV, when appropriate), then they must also agree on the cipher transformation to use.
  3. If the involved parties participating in encryption are not using a "combined" cipher mode that provides both confidentiality and authenticity (for example, something like GCM or CCM), then they must whether or not there will be an additional MAC sent allow with the raw ciphertext and other cipher spec information in order to provide evidence of authenticity / data integrity. (See the property Encryptor.CipherText.useMAC for further details.) Failure to agree on this may allow an adversary to carry out certain chosen ciphertext attacks against their encrypted data resulting in (possibly complete) leakage of the corresponding plaintext.

Encrypting / Decrypting with the New Methods -- The Simple Usage

Using the new encryption / decryption methods is somewhat more complicated, but this is in part because they are more flexible and that flexibility means that more information needs to be communicated as to the details of the encryption.

A code snippet using the new methods that use the master encryption key would look something like this:

    String myplaintext = "My plaintext";
    try {
        CipherText ciphertext =
            ESAPI.encryptor().encrypt( new PlainText(myplaintext) );
        PlainText recoveredPlaintext = ESAPI.encryptor().decrypt(ciphertext) );
        assert myplaintext.equals( recoveredPlaintext.toString() );
    } catch(EncryptionException ex) {
        // Log error then return error designation however appropriate.
    }

Yes, this is a bit more complicated, but it will 1) work across different hardware platforms and operating systems whereas the older methods may not, and 2) it provides for authenticity and confidentiality of the ciphertext regardless of which cipher mode is chosen.

Also, these new methods allow a general byte array to be encrypted, not just a Java String. If one needed to encrypt a byte array with the old deprecated method, one would first have to use

    byte[] plaintextByteArray = { /* byte array to be encrypted */ };
    String plaintext = new String(plaintextByteArray, "UTF-8");

all the while catching the required UnsupportedEncodingException. For example, to handle this in ESAPI 1.4, one would have to write something like:

    try {
        byte[] plaintextByteArray = { /* byte array to be encrypted */ };
        String myplaintext = new String(plaintextByteArray, "UTF-8");
        String ciphertext = ESAPI.encryptor().encrypt(myplaintext);
        String decrypted  = ESAPI.encryptor().decrypt(ciphertext);
        byte[] recoveredBytes = decrypted.getBytes(“UFT-8”);
        assert java.util.Arrays.equals( plaintextByteArray, recoveredBytes );
    } catch( UnsupportedEncodingException ex) {
        // Should not happen but need to catch and deal with it anyhow.
        // Log error then return error designation however appropriate.
    } catch(EncryptionException ex) {
        // Log error then return error designation however appropriate.
    }

However, dealing with this in ESAPI 2.0 is not any more cumbersome than dealing with Strings:

    try {
        byte[] plaintextByteArray = { /* byte array to be encrypted */ };
        CipherText ciphertext =
            ESAPI.encryptor().encrypt( new PlainText(plaintextByteArray) );
        PlainText recoveredPlaintext = ESAPI.encryptor().decrypt(ciphertext) );

        assert java.util.Arrays.equals( plaintextByteArray,
                                        recoveredPlaintext.asBytes() );
    } catch(EncryptionException ex) {
        // Log error then return error designation however appropriate.
    }

Ideally when you are encrypting sensitive data you do not want the plaintext sensitive data to be left lying around after it is encrypted. Instead, you should overwrite them after their value as been used. However, when you are using immutable Strings, this is not possible using native Java methods. But if you are able to pass in byte arrays that are passed directly to PlainText objects (as shown above), the default is to overwrite this after they are encrypted. (Note: Verify!) If the default for Encryptor.PlainText.overwrite of true had been used, then the array plaintextByteArray would have been overwritten with ASCII “*” characters.

Encrypting / Decrypting with the New Methods – Storing Encrypted Data

If you use one of the new Encryptor encrypt() / decrypt() methods, how do you persist the CipherText object returned by the encrypt() methods and how do you restore it to pass to the decrypt() method?

The following example code snippet will illustrate this. In the following example we will simply write out the serialized CipherText object to a local file, but obviously you could hex- or base64-encode the serialized byte array and store it in a database or sent it in a SOAP XML message to a web service, etc.

    public class PersistedEncryptedData
    {
        public static int persistEncryptedData(PlainText plaintext,
                                                String filename)
            throws EncryptionException, IOException
        {
            File serializedFile = new File(filename);
            serializedFile.delete(); // Delete any old serialized file.

            CipherText ct = ESAPI.encryptor().encrypt(plaintext);
            byte[] serializedCiphertext = ct.asPortableSerializedByteArray();

            FileOutputStream fos = new FileOutputStream(serializedFile);
            fos.write(serializedCiphertext);
            fos.close();
            return serializedCiphertext.length;
        }

        public static PlainText restorePlaintext(String encryptedDataFilename)
            throws EncryptionException, IOException
        {
            File serializedFile = new File(encryptedDataFilename);
            FileInputStream fis = new FileInputStream(serializedFile);
            int avail = fis.available();
            byte[] bytes = new byte[avail];
            fis.read(bytes, 0, avail);

            CipherText restoredCipherText =
                                CipherText.fromPortableSerializedBytes(bytes);
            fis.close();
            PlainText plaintext = ESAPI.encryptor().decrypt(restoredCipherText);
            return plaintext;
        }
    }
  

Advanced Usage

Encrypting / Decrypting with the New Methods

ESAPI 1.4 and earlier only allowed you to use the master key (MasterPassword in ESAPI 1.4; Encryptor.MasterKey in ESAPI 2.0) to encrypt and decrypt with. But encryption with a single key seldom is sufficient. For instance, lets say that your application has a need to encrypt both bank account numbers and credit card numbers. The encrypted bank account numbers are to be sent to one recipient and the encrypted credit card numbers are to be sent to a different recipient. Obviously in such cases, you do not want to share the same key for both recipients.

In ESAPI 1.4 there was not much you can do, but in ESAPI 2.0 and later, there are new encryption / decryption methods that allow you to specify a specific SecretKey. There is also a static helper method in CryptoHelper to allow you to generate a SecretKey of a specific type. (Distribution of this key is out of scope for this particular example, but for the moment, we will assume that secret keys are first generated, and then distributed to the recipients out-of-band. On you could distribute them dynamically via asymmetric encryption assuming that you've previously exchanged public keys with the recipients.)

The following illustrates how these new methods might be used.

First, we would generate some appropriate secret keys and distribute them securely (e.g., perhaps over SSL/TLS) or exchange them earlier out-of-band to the intended recipients. (E.g., one could put them on two separate thumb drives and use a trusted courier to distribute them to the recipients or one could use PGP-mail or S/MIME to securely email them, etc.)

    // Generate two random, 128-bit AES keys to be distributed out-of-band.
    import javax.crypto.SecretKey;
    import org.owasp.esapi.crypto.CryptoHelper;
    import org.owasp.esapi.codecs.Hex;

    public class MySecretKeys {
        public void main(String[] args) {
          try {
            SecretKey bankAcctKey = CryptoHelper.generateSecretKey("AES", 128);
            SecretKey credCardKey = CryptoHelper.generateSecretKey("AES", 128);

            System.out.println("Bank account key: " +
                Hex.encode( bankAcctKey.getEncoding(), true ) );
            System.out.println("Credit card key: " +
                Hex.encode( credCardKey.getEncoding(), true ) );
          } catch(Exception ex) {
            ex.printStackTrace(System.err);
            System.exit(1);
          }
          System.exit(0);
        }
    }

Second, these keys would be printed out and stored somewhere secure by our application, perhaps using something like ESAPI's EncryptedProperties class, where they could later be retrieved and used.

In the following code, we assume that the SecretKey values have already been initialized elsewhere.

    SecretKey bankAcctKey = ...;        // These might be read from EncryptedProperties
    SecretKey credCardKey = ...;        // or from a restricted database, etc.
    ...
    String bankAccountNumber = ...;     // Assume obtained elsewhere
    String creditCardNumber = ...;      // Ditto
    ...
    try {
        // Encrypt each with their appropriate secret key
        CipherText encryptedBankAcct =
            ESAPI.encryptor().encrypt( bankAcctKey, new PlainText(bankAccountNumber) );
        CipherText encryptedCreditCard =
            ESAPI.encryptor().encrypt( credCardKey, new PlainText(creditCardNumber) );
        ...
        // Decrypt using appropriate secret key
        PlainText recoveredBankAcct = ESAPI.encryptor().decrypt( bankAcctKey, encryptedBankAcct ) );
        assert bankAccountNumber.equals( recoveredBankAcct );
        ... etc. ...
    } catch(EncryptionException ex) {
        // Log error then return error designation however appropriate.
    }

Using ESAPI with Multiple Cipher Transformations

For the most part, the architecture of ESAPI assumes that you will stick to using the defaults in the ESAPI.properties configuration file or implment your own classes--possibly by extending the ESAPI reference classes--and use these classes instead of the reference classes.

For most things, this works well. Most applications likely can standardize on a single cipher transformation such as AES/CBC/PKCS5Padding with a 128-bit AES key and use that 100% of the time. However, on occassion, an application may need to use two separate cipher transformations (or even two different cipher algorithms) to handle legacy applications or deal with multiple partners.

This section discusses how to do this without implementing your own classes or extending the ESAPI reference class, JavaEncryptor. Note that it is recognized that this approach is somewhat of a kludge. A simpler approach is planned for ESAPI 2.1, but the approach shown here is workable even though it's not pretty.

If you find yourself in need of encrypting with a different cipher transformation, the first thing that you should count on is not using the same encryption key for each. While in some cases this likely would work (e.g., you are only using a different cipher mode or you have 256-bit AES key for "Encryptor.MasterKey" but also have a need to do encryption with a 128-bit AES key), it is not guaranteed to do so. Instead, you should count on generating a separate encryption key and using the encrypt / decrypt methods taking an additional SecretKey parameter as show in the previous section.

Rather than repeating all the details of how to do this in this user guide, we encourage you to investigate how this was done in the Junit testing for the JavaEncryptor class. Please look at the source code for the private method runNewEncryptDecryptTestCase(String, int, byte[]) in the EncryptorTest JUnit test in the source code "src/test/java/org/owasp/esapi/reference/crypto/EncryptorTest.java". This code calls:

		ESAPI.securityConfiguration().setCipherTransformation(cipherXform);
which sets ESAPI to use the specified cipher transformation, cipherXform. As a convenience (for later restoral), it returns the previous cipher transformation.

There are also a few non-obvious key size adjustments that are also going on with DES and DESede (aka, triple DES) keys that are made there as well. This has to do with the fact that for DES keys (which includes DESede), the "true" key size differs from the "effective" key size. (E.g., in DES, the "true" key size--from the DES algorithms's perspective is 64-bits, however the effective key size for DES is only 56-bits because of the 8-bits of parity "imposed" by the NSA in the early 1970s.) This inconsistency manifests itself in the JCE by the fact that KeyGenerator.init() wants the effective key size to be specified (so 56-bits for DES, 112- or 168-bits for DESede), but SecretKey.getEncoding().length stores the "true" key size (e.g., 64-bits or 192-bits for DES and DESede respectively). Other cipher algorithms do not have this discrepancy between true and effective key sizes. Since a SecretKey is returned by the KeyGenerator, this is only all too confusing. The reference JavaEncryptor and its JUnit test, EncryptorTest attempt to deal with this discrepency. See the adjustments made to key size in EncryptorTest.runNewEncryptDecryptTestCase() for further details.

Using ESAPI Symmetric Encryption with FIPS 140-2 Cryptographic Modules

If you wish to use ESAPI with a FIPS 140-2 validated, JCE-compliant cryptographic module such as the IBM Java JCE FIPS 140-2 Cryptographic Module (hereafter referred to as IBMJCEFIPS) or the RSA Security BSafe Crypto-J (hereafter referred to as Crypto-J), or other similar products, you must follow these instructions. Failure to do so will mean that you will fail to meet FISMA compliance. (Note: The mention of these two specific vendor products does not constitute an endorsement from either OWASP or this author.)
  1. Follow the vendor's regular instructions to configure your application to use the vendor product in a FIPS 140-2 compliant manner. Generally this will include some special configuration and/or initialization requirements that must be followed to restrict the vendor software to use FIPS 140-2 approved algorithms.

    If the vendor's instructions do not already include a recommendation to edit the $JAVA_HOME/jre/lib/security/java.security file, the OWASP team recommends that you do so to prevent you from accidentally using any other cryptographic module (such as SunJCE). To do this, you will need to change the property "security.provider.1" (which represents the default) provider so that it refers to the fully-qualified classname of your vendor's JCE Provider class. (By default, this is set to "sun.security.provider.Sun".) In fact, you may wish to go as far as to either comment out or remove the other providers to reduce the possibility that they will accidentally be used by your application.
  2. Edit the ESAPI.properties that your application is going to be using and set the property "Encryptor.PreferredJCEProvider" to the fully qualified classname of your vendor's FIPS 140-2 compliant JCE provider class. (If you are using a Java SecurityManager you may also need to grant org.owasp.esapi.reference.crypto.JavaEncryptor permissions to change the JCE provider.)
  3. Edit the ESAPI.properties that your application is going to be using and set the property Encryptor.CipherText.useMAC to false. This is critical as having this property set to true causes ESAPI to use derived keys for the actual encryption and MAC calculation, which is against FIPS 140-2 compliance. (Note: This does not mean that OWASP believes that this key derivation is weak--its design has been suggested some cryptographers--but it would be creating and using a related key for encryption in a manner not reviewed by NIST and thus it is not acceptable to FIPS 140-2 approval.)
  4. Edit the ESAPI.properties that your application is going to be using and set the property "Encryptor.CipherTransformation" to a cipher transformation that is FIPS 140-2 certified for you vendor's software. OWASP recommends using a NIST "combined" cipher mode, that is one that provides for both message confidentiality and message authenticity if such a cipher mode is available from your vendor and FIPS 140-2 certified. (Cipher modes "GCM" and "CCM" are the only FIPS 140-2 approved as of this writing.) If such a "combined" cipher mode is not available, then use a cipher transformation like "AES/CBC/PKCS5Padding" with a random IV. This is almost always available as one of the FIPS 140-2 certified cipher transformations, however it will not provide you with message authenticity because to remain FIPS 140-2 compliant you will have had to disable the MAC calculation (via setting Encryptor.CipherText.useMAC to false) causing ESAPI to skip calculating an explicit MAC for you and hence providing no assurance of data integrity to the party attempting to decrypt your data. Without authenticity however, your encryption may still be vulnerable to the "padded oracle" chose ciphertext attack. Consult with your local cryptographic expert in such cases as this depends greatly on the circumstances of how you are using encryption.
  5. If your vendor softare requires you to explicitly to initialize their software for FIPS-mode (e.g., to cause the required FIPS 140-2 "power-on self tests" to run) by calling some software method, then this must be done by your application code before calling any of the ESAPI crypto-related code. Specifically, it must be called before your application calls ESAPI.encryptor().

We believe that following these steps will still allow for your application to be FIPS 140-2 / FISMA compliant, however, as always, you should check with your FIPS auditor before using ESAPI in this manner. Should your auditors ask, feel free to point them to the ESAPI source code, which we believe will convince them that ESAPI is not a cryptographic module. (Rather, it only provides a wrapper to call other cryptographic modules through the standard JCE APIs.)

Acknowledgments

The OWASP ESAPI development team would like to acknowledge the contributions of cryptographers David A. Wagner and Ian Grigg. David provided the outline of how to securely compute derived keys for confidentiality and authenticity from a single master key and Ian convinced us to take a packetizing approach to laying out the portable, serialized CipherText object.

In addition, Kevin Kenan has agreed to review all the crypto code from ESAPI 2.0-rc5 or 2.0-rc6 candidate release. Others are invited to participate as well, especially those with a background in cryptography. (See "Request to review ESAPI 2.0 crypto" for details.)

I would also like to thank Jessica Fitzgerald-McKay and Andy Sampson of the Systems and Network Analysis Center of the NSA for their excellent feedback in response to the review request made in Google Issue #81. Working with bureaucracy is not something that I do particularly well, but Jessica and Andy made it as painless as it possibly could be. Their feedback, which at this point, unfortunately we are unable to quote, was generally favorable, but in the places where they had recommendations, these were very precise and helpful.

Lastly, I would like to thank Jeffrey Walton of Software Integrity, LLC. Jeff provided an extremely thorough analysis of ESAPI 2.0's (as of 2.0_rc10) Key Derivation Function (KDF). Jeff's analysis convinced me to bring ESAPI's KDF more in line with NIST's recommendations for KDFs as described in NIST Special Publication 800-108 (and specifically section 5.1). You can read about Jeff's review at Analysis of ESAPI 2.0's Key Derivation Function