Encryption

A few random notes I've been making about security libraries I've been using this past year or so.

How Random is Random?

SecureRandom is the gold standard. However, "depending on the implementation, the generateSeed and nextBytes methods may block as entropy is being gathered, for example, if they need to read from /dev/random on various Unix-like operating systems." [JavaDocs] This hasn't been a problem for me so far as I create one million 64-bit random numbers in my unit tests and the whole process takes about a second or two.

On Linux, you can see the temperature of the CPU, fan speeds etc by installing the tools mentioned here (AskUbuntu). This is one way to generate randomness.

There's an interesting addition to the Java API called ThreadLocalRandom that is more efficient than java.util.Random but still not appropriate for secure random number generators.

PGP or GPG?

"OpenPGP is the IETF-approved standard that defines encryption technology that uses processes that are interoperable with PGP. pgp is Symantec's proprietary encryption solution. pgp adheres to the OpenPGP standard and provides an interface that allows users to easily encrypt their files." [NetworkWorld]

"gpg is the OpenPGP part of the GNU Privacy Guard (GnuPG). It is a tool to provide digital encryption and signing services using the OpenPGP standard. gpg features complete key management and all the bells and whistles you would expect from a full OpenPGP implementation." [gpg man pages].

You can have the public key embedded in the file which can identify the recipient.  Why this is useful? "As far as I know, the recipient's public key IDs, key Validity dates, name, and email address are embedded in the GPG ASCII Armor file (GnuPG Manual ). So using pub key file / Key ID / Name / Email to identify which public key to use should all be equivalent." [StackExchange]

Importing a private key

If you haven't got the key, you can't decrypt a file. But if you have, you don't need to specify it. For instance, if I try to decrypt a file for which I don't have a key, I see:

$ gpg --output file.zip -d file.zip.pgp 
gpg: encrypted with RSA key, ID EAC258F9825D4C9C
gpg: decryption failed: No secret key

However, I can import it:

$ gpg --import ~/Temp/key.txt
gpg: key EAC258F9825D4C9C: public key "XXXX-TEST " imported
gpg: key EAC258F9825D4C9C: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

and now decrypt it:

$ gpg --output file.zip -d file.zip.pgp 
gpg: encrypted with 4096-bit RSA key, ID EAC258F9825D4C9C, created 2020-02-27
      "XXXX-TEST "

Bouncy Castle

Bouncy Castle is the defacto library to allow the JVM to access OpenPGP files. 

One gotcha I found when using Bouncy Castle in an über JAR that was called in a Docker container was:

Caused by: java.util.jar.JarException: file:/home/henryp/main-1.0-SNAPSHOT-jar-with-dependencies.jar has unsigned entries ...

There doesn't seem to be a huge amount you can do about this if you insist on using über JARs as "You can't bundle a cryptographic library. They have to be signed for the JVM to load them, and the signature is destroyed when merged into the shadow jar." [GitHub]

This seems to be something specific to Oracle's JDK because if my Docker config file starts with:

FROM openjdk:11-jdk-slim

I don't have this problem. 


Encrypted ZIPs

I was hoping to stream a zip file that was encrypted, decrypting and unzipping as I went but was worried about the ZIP format. Note that a "directory is placed at the end of a ZIP file. This identifies what files are in the ZIP and identifies where in the ZIP that file is located. This allows ZIP readers to load the list of files without reading the entire ZIP archive. ZIP archives can also include extra data that is not related to the ZIP archive." [Wikipedia] 

So, could I really decrypt and unzip a stream?

Changing to GZIP wouldn't help either because "Both zip and gzip use the same compressing format internally, the main difference is in the metadata: zip has it at the end of the file, gzip at the beginning (and gzip only supports one enclosed file easily)." [StackOverflow]

But decrypting the stream and forking a process that unzips it using PipedInputStream and PipedOutputStream seems to work even on files of a about 1gb.

Encrypted Parquet

Parquet Modular Encryption allows certain columns to be encrypted.

OAuth

“Designed specifically to work with … (HTTP), OAuth essentially allows access tokens to be issued to third-party clients by an authorization server, with the approval of the resource owner” [Wikipedia] "A trust store is used to authenticate peers. A key store is used to authenticate yourself." [StackOverflow]

You can get tokens in Google DataFlow with this:

    val credentials = ComputeEngineCredentials.create()
    val accessToken = credentials.refreshAccessToken()
    logger.info(s"accessToken = $accessToken")             // OAuth token
    logger.info(s"getAccount = ${credentials.getAccount}") // service account name

Although almost ubiquitous, OAuth has its drawbacks:

Ross A. Baker @rossabaker Jun 08 04:27

I think the specification is far too complex for what it accomplishes.

I had a lengthy argument implementing it when I worked at a security company as a replacement for a request signing algorithm.

In request signing, you can't just steal a token the way you can in OAuth2.

And the argument is, "Well, it's sent over TLS, what does it matter?"

And as we were having that argument, those tokens were appearing in clear text in our logs.

Was it a shitty implementation? Absolutely. But all it takes is one mistake like that.

It's neither as convenient as basic auth, nor as secure as something like an HMAC-signed request. I feel like it operates in a middle ground that suits no purpose very well.

Gavin Bisesi @Daenyth Jun 08 14:48

another pain point is that IIUC the oauth spec is very full of "MAY" options and relatively few "MUST" options, so every actual implementation does stuff differently and nothing is compatible with anything else eg you often need specific logic to support X vs Y backends

An alternative to OAuth is Request Signing. Basically, the server has all private keys, clients only have their own and messges are encrypted and signed - see Andrew Hoang's blog.

Gotchas

When storing pass phrases etc in files, be careful that your editor does not add a newline. For instance, open a file in vi such:

$ vi /tmp/5Chars.txt

Type the string 12345, save and close it.

$ ls -l /tmp/5Chars.txt

-rw-r--r-- 1 henryp kismet 6 Nov 24 10:23 /tmp/5Chars.txt

What? it's 6 bytes, not 5! One solution is this:

$ echo -n 12345 > /tmp/5Chars.txt 

$ ls -l /tmp/5Chars.txt

-rw-r--r-- 1 henryp kismet 5 Nov 24 10:25 /tmp/5Chars.txt

That's better.