How does the App GW protocol handle encryption?


 
From Chapter 8 of the App GW protocol
 
8. Encryption
Any host may be configured to use encrypted messages. When this is selected, the a private key must be selected and configured in the router, and at the host. When the connection is made, the router will generate a random, 8 byte session key, encrypt the session key using DES and the private key, and send the encrypted session key. To allow for checking, the session key is sent doubled, as a 16 byte string. For example, if the key was ABCDEFGH, ABCDEFGHABCDEFGH would be sent as a 16 byte encrypted string to the host. The host should decrypt this using the session key, verify that the first 8 bytes are the same as the last 8 bytes. If this check fails, the connection should be rejected.
All tagged fields in subsequent QUERY_REQ and QUERY_RESP messages will be encrypted and decrypted using this 8 byte session key. Each tagged field will be padded with nulls until it is a multiple of 8 bytes long, and encrypted. The tagged field sent will always be a multiple of 8 bytes. After decryption, the trailing nulls must be counted and removed in order to compute the true length of the string, if necessary.
Any tagged field that does not decrypt to a valid ASCII string will be treated as an error, and ignored.
 


Some additional information not included in the guide that may be  helpful:
 
- the private key string is
stored in the registry where the Application Gateway process
is running, more precisely: HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems,
Inc.\ICM\{ICM Instance
name}\Router{A,B}\Router\CurrentVersion\Configuration\ApplicationGateway,
the key name is SessionKey;
- seems the documentation
interchanges the terms "SessionKey" and "Private Key" freely
- look above, in the registry it is named SessionKey, in fact, that is the
Private Key;
- the host is supposed to expect the
encrypted session key in binary form, not in base64.
- only the payload is encrypted for
the regular fields (for instance var1..10), their type and payload length is
actually not encrypted, however, this does not apply to the ECC variables and
ECC array variables, where their name and payload length is encrypted as well,
- the DES algorithm uses the
Electronic Codebook
(EC cipher format (for a nice
example take a look at the lovely penguin on Wikipedia:
http://en.wikipedia.org/wiki/Electronic_code_book#Electronic_codebook_.28ECB.29).
 
And here is a code sample in Java:
 
      public final
static byte[] getEncrypted(byte[] key, byte[] messageToEncrypt ) throws
DataLengthException, IllegalStateException, InvalidCipherTextException,
NoSuchAlgorithmException, NoSuchProviderException, NoSuchPaddingException,
InvalidKeyException, ShortBufferException, IllegalBlockSizeException,
BadPaddingException {
            //first,
pad it up to *8 with nulls:
            int
remainder = 0;
            remainder
= messageToEncrypt.length % 8;
            int
oldLength = messageToEncrypt.length;
            int
newLength = oldLength;
            if
(remainder > 0) {
                  newLength
+= (8 - remainder);
            }
            byte[]
paddedMessageToEncrypt = new byte;
            System.arraycopy(messageToEncrypt,
0, paddedMessageToEncrypt, 0, oldLength); // copy the original bytearray over
            if
(remainder > 0) {
                  java.util.Arrays.fill(paddedMessageToEncrypt,oldLength,paddedMessageToEncrypt.length-1,(byte)
0);
            }
            Security.addProvider(new
BouncyCastleProvider());
            SecretKeySpec
keySpec = new SecretKeySpec( key, "DES");
            Cipher
cipher = Cipher.getInstance("DES/ECB/NoPadding", "BC");
            cipher.init(Cipher.ENCRYPT_MODE,
keySpec);
            //get
the output length
            byte[]
cipherText = new
byte[cipher.getOutputSize(paddedMessageToEncrypt.length)];
            int
ciperTextLength = cipher.update(paddedMessageToEncrypt, 0,
paddedMessageToEncrypt.length, cipherText, 0);
            ciperTextLength
+= cipher.doFinal(cipherText, ciperTextLength);
            return
cipherText;
      } // method
getEncrypted

Here is the decryption code snippet in Java provided by a customer. No support is provided for it.



Security.addProvider(new BouncyCastleProvider());

SecretKeySpec keySpec = new SecretKeySpec( key, "DES");

Cipher cipher = Cipher.getInstance("DES/ECB/NoPadding", "BC");

cipher.init(Cipher.DECRYPT_MODE,keySpec);

byte[] decryptedText = new

byte[cipher.getOutputSize(encryptedMessage.length)];

int ctLength = cipher.update(encryptedMessage, 0, encryptedMessage.length,

decryptedText, 0);

ctLength += cipher.doFinal(decryptedText, ctLength);

byte[] result = decryptedText;





where key is a byte array containing the 8 bytes of the session key (which

is quite readable since it's all ASCII) and the encyptedMessage variable is

obviously what you get from the ICM router. One must import the

org.bouncycastle libraries as well.

And as the docs say, the result byte array is padded with nulls.

I do not have any C# code you will need to translate the Java yourself.