KeyCloak Security

KeyCloak/OAuth

The nature of OAuth in web usage includes a frontend client like a web browser and a backend client which is the application (for example, Apache Polaris). 

Realms are isolated namespaces of security. They can be federated but are not by default.

The JWT (JSON Web Token) is associated with a realm that is passed to the backend client. It contains an ISSued field that indicates who issued it. The KeyCloak issuing it signs it with its private key.

The ISS is a URL that the embeds both the realm and the domain of the ISS. The backend client can use this URL to find a public key of KeyCloak with which it can verify the provenance of the JWT. 

This endpoint is called the Realm JWKS (JSON Web Key Set).

You can see the contents of the token with a simple Linux line like this:

echo $POLARIS_TOKEN | cut -d. -f2 | tr '_-' '/+' | base64 -d | jq

If a claim in the JWT is azp (authorized party), it identifies to whom the token was issued - typically the front end app. If it's audience, it identifies for whom the token was issued - typically the backend.

So, an example would be that the user logs in and their web browser is the azp. They ask KeyCloak for a token on behalf of the back end, the aud. KeyCloak signs both so the backend knows the token is genuinely for it and who requested it. If the azp is not in its whitelist, the backend can reject it. 

KeyCloak Configuration

To relax this constraint (say, for integration tests), Polaris can set the environment variable quarkus.oidc.token.issuer=any.

Interestingly, OAuth2 servers have a handy endpoint at .well-known/openid-configuration listing salient details. So, for KeyCloak, run:

curl -s https://YOUR_DOMAIN/realms/YOUR_REALM/.well-known/openid-configuration | jq

If your backend is in the same private network as KeyCloak, you can set KC_HOSTNAME_STRICT_BACKCHANNEL=false to make KeyCloak dynamically generate backchannel URLs based on the request's Host header - the mandatory HTTP header that defines which virtual host is the endpoint.

Note that an attacker could use Host Header Injection to trick KeyCloak into certifying a malicious domain. To stop this, the load balancer must be correctly configured to ignore inappropriate Host values plus KeyCloak should have KC_HOSTNAME_STRICT=true and KC_HOSTNAME set so even if the hacker gets past the load balancer, KeyCloak says "nice try but I know what public facing name I have". Again, you might like to relax this for testing.

Talking of the load balancer, traffic between it and KeyCloak is unencrypted so we need to tell KeyCloak communication is plain HTTP after TLS termination with KC_HTTP_ENABLED=true and set a KC_PROXY_HEADERS strategy to tell it what the load balancer is doing with the traffic. We use xforwarded so it knows our AWS ALB injects headers in a standard way.

Permissions and Lakes

OpenID (authorisation) is built on top of OAuth (authentication). 

"It allows clients to verify the identity of the end user based on the authentication performed by an authorization server, as well as to obtain basic profile information about the end user in an interoperable and RESTlike manner." Zero Trust Networks (O'Reilly)

The (Java) KeyCloak is a common choice for an open source solution. KeyCloak is an Identity Provider (IdP).

Polaris can vend credentials but it (rather than the cloud IAM system) controls who gets what. It acts as an ACL for ACLs, if you like. This code shows how a request is associated with a realm and a realm with a credential.

If you're going to use Polaris in production, you'll probably need a certificate from a recognised Certificate Authority like Let's Encrypt. The reason is that all HTTPS clients need to make a call to a list of hard coded authorities who will sign off the certificate as genuine.

AWS users can use AWS Certificate Manager (ACM) to certify endpoints - rather than configuring Polaris to use SSL. You can have AWS manage the whole thing; or you can "install an externally signed private CA certificate on your subordinate CA. This CA certificate must be signed by a parent CA. Installing the certificate completes the creation and activation of the CA."

Either way, the idea is that the Elastic Load Balancer provides an HTTPS endpoint, does all the de/encryption gubbins and then forwards plain HTTP on to Polaris that sits securely in your Virtual Private Cloud.

To this end, it seems you must deploy the AWS Load Balancer Controller in your Kubernetes cluster much like the vpc-cni EksAddon.

Note that it can take a minute or two for a mapping from a domain name to an endpoint to be registered. Run:

dig A YOUR_DOMAIN_NAME +short

to see if your DNS is updated.

Fine Grained Access

OpenFGA adds Fine Grained Access control. This implementation is written in Go.

Kubernetes

Azure and GCP do things differently. It uses a sidecar that leverages Let's Encrypt.

"Uploading and managing TLS secrets can be difficult. In addition, certificates can often come at a significant cost. To help solve this problem, there is a nonprofit called “Let’s Encrypt” running a free Certificate Authority that is API-driven. Since it is API-driven, it is possible to set up a Kubernetes cluster that automatically fetches and installs TLS certificates for you. It can be tricky to set up, but when working, it’s very simple to use. The missing piece is an open source project called cert-manager created by Jetstack, a UK startup, onboarded to the CNCF." - Kubernetes Up & Running 3rd Ed., O'Reilly

Certificates and Challenges

In the context of Kubernetes, the certificate contains the secret name where the final certificate will be stored and a reference to an issuer. The result is a Kubernetes secret containing the actual public and private key for HTTPS.

The challenge proves to the CA that you own the domain. You can either use a HTTP-01 challenge where you host a token on a URL that uses the domain. 

Or you use DNS-01 where you ask your DNS provider to host a record containing the token (see below).

Domain Ownership Validation

AWS has Route53 that nicely integrates management of domain names with Kubernetes. That is, you can seemlessly have an EKS ingress assigned an AWS managed domain.

Google, however, have recently sold their domain name arm so you need to persuade them that the domain you own is really yours before they'll point it at your Kubernetes ingress. To do this, run:

gcloud certificate-manager dns-authorizations create ARBITRARY_STRING --domain="YOUR_DOMAIN" --project YOUR_PROJECT

Secrets

So much for security outside the cloud. Here is how you deal with it inside.

You'll need to install secrets-store-csi-driver-provider-aws which is a CRD that runs in Kubernetes and talks to AWS. It allows you to mount the secrets in your container as if they were any other filesystem.

I followed the instructions in the AWS CSI driver (above) but I could just not get Option 1 to work despite a lot of time checking that everything was OK. It's still a mystery but Option 2 worked first time. 

Restart a deployment with something like:

kubectl rollout restart deployment polaris

This is especially useful if you've updated the secrets.

Encryption

A few random notes I've been making about security libraries I've been using this past year or so.

How Random is Random?

SecureRandom is the gold standard. However, "depending on the implementation, the generateSeed and nextBytes methods may block as entropy is being gathered, for example, if they need to read from /dev/random on various Unix-like operating systems." [JavaDocs] This hasn't been a problem for me so far as I create one million 64-bit random numbers in my unit tests and the whole process takes about a second or two.

On Linux, you can see the temperature of the CPU, fan speeds etc by installing the tools mentioned here (AskUbuntu). This is one way to generate randomness.

There's an interesting addition to the Java API called ThreadLocalRandom that is more efficient than java.util.Random but still not appropriate for secure random number generators.

PGP or GPG?

"OpenPGP is the IETF-approved standard that defines encryption technology that uses processes that are interoperable with PGP. pgp is Symantec's proprietary encryption solution. pgp adheres to the OpenPGP standard and provides an interface that allows users to easily encrypt their files." [NetworkWorld]

"gpg is the OpenPGP part of the GNU Privacy Guard (GnuPG). It is a tool to provide digital encryption and signing services using the OpenPGP standard. gpg features complete key management and all the bells and whistles you would expect from a full OpenPGP implementation." [gpg man pages].

You can have the public key embedded in the file which can identify the recipient.  Why this is useful? "As far as I know, the recipient's public key IDs, key Validity dates, name, and email address are embedded in the GPG ASCII Armor file (GnuPG Manual ). So using pub key file / Key ID / Name / Email to identify which public key to use should all be equivalent." [StackExchange]

Importing a private key

If you haven't got the key, you can't decrypt a file. But if you have, you don't need to specify it. For instance, if I try to decrypt a file for which I don't have a key, I see:

$ gpg --output file.zip -d file.zip.pgp 
gpg: encrypted with RSA key, ID EAC258F9825D4C9C
gpg: decryption failed: No secret key

However, I can import it:

$ gpg --import ~/Temp/key.txt
gpg: key EAC258F9825D4C9C: public key "XXXX-TEST " imported
gpg: key EAC258F9825D4C9C: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

and now decrypt it:

$ gpg --output file.zip -d file.zip.pgp 
gpg: encrypted with 4096-bit RSA key, ID EAC258F9825D4C9C, created 2020-02-27
      "XXXX-TEST "

Bouncy Castle

Bouncy Castle is the defacto library to allow the JVM to access OpenPGP files. 

One gotcha I found when using Bouncy Castle in an über JAR that was called in a Docker container was:

Caused by: java.util.jar.JarException: file:/home/henryp/main-1.0-SNAPSHOT-jar-with-dependencies.jar has unsigned entries ...

There doesn't seem to be a huge amount you can do about this if you insist on using über JARs as "You can't bundle a cryptographic library. They have to be signed for the JVM to load them, and the signature is destroyed when merged into the shadow jar." [GitHub]

This seems to be something specific to Oracle's JDK because if my Docker config file starts with:

FROM openjdk:11-jdk-slim

I don't have this problem. 


Encrypted ZIPs

I was hoping to stream a zip file that was encrypted, decrypting and unzipping as I went but was worried about the ZIP format. Note that a "directory is placed at the end of a ZIP file. This identifies what files are in the ZIP and identifies where in the ZIP that file is located. This allows ZIP readers to load the list of files without reading the entire ZIP archive. ZIP archives can also include extra data that is not related to the ZIP archive." [Wikipedia] 

So, could I really decrypt and unzip a stream?

Changing to GZIP wouldn't help either because "Both zip and gzip use the same compressing format internally, the main difference is in the metadata: zip has it at the end of the file, gzip at the beginning (and gzip only supports one enclosed file easily)." [StackOverflow]

But decrypting the stream and forking a process that unzips it using PipedInputStream and PipedOutputStream seems to work even on files of a about 1gb.

Encrypted Parquet

Parquet Modular Encryption allows certain columns to be encrypted.

OAuth

“Designed specifically to work with … (HTTP), OAuth essentially allows access tokens to be issued to third-party clients by an authorization server, with the approval of the resource owner” [Wikipedia] "A trust store is used to authenticate peers. A key store is used to authenticate yourself." [StackOverflow]

You can get tokens in Google DataFlow with this:

    val credentials = ComputeEngineCredentials.create()
    val accessToken = credentials.refreshAccessToken()
    logger.info(s"accessToken = $accessToken")             // OAuth token
    logger.info(s"getAccount = ${credentials.getAccount}") // service account name

Although almost ubiquitous, OAuth has its drawbacks:

Ross A. Baker @rossabaker Jun 08 04:27

I think the specification is far too complex for what it accomplishes.

I had a lengthy argument implementing it when I worked at a security company as a replacement for a request signing algorithm.

In request signing, you can't just steal a token the way you can in OAuth2.

And the argument is, "Well, it's sent over TLS, what does it matter?"

And as we were having that argument, those tokens were appearing in clear text in our logs.

Was it a shitty implementation? Absolutely. But all it takes is one mistake like that.

It's neither as convenient as basic auth, nor as secure as something like an HMAC-signed request. I feel like it operates in a middle ground that suits no purpose very well.

Gavin Bisesi @Daenyth Jun 08 14:48

another pain point is that IIUC the oauth spec is very full of "MAY" options and relatively few "MUST" options, so every actual implementation does stuff differently and nothing is compatible with anything else eg you often need specific logic to support X vs Y backends

An alternative to OAuth is Request Signing. Basically, the server has all private keys, clients only have their own and messges are encrypted and signed - see Andrew Hoang's blog.

Gotchas

When storing pass phrases etc in files, be careful that your editor does not add a newline. For instance, open a file in vi such:

$ vi /tmp/5Chars.txt

Type the string 12345, save and close it.

$ ls -l /tmp/5Chars.txt

-rw-r--r-- 1 henryp kismet 6 Nov 24 10:23 /tmp/5Chars.txt

What? it's 6 bytes, not 5! One solution is this:

$ echo -n 12345 > /tmp/5Chars.txt 

$ ls -l /tmp/5Chars.txt

-rw-r--r-- 1 henryp kismet 5 Nov 24 10:25 /tmp/5Chars.txt

That's better.

Spark, HBase and security

Getting Spark and HBase to talk to each other is clearly non-trivial if Jiras are anything to go by (see here and here).

My situation is this: I'm building TF-IDF statistics from a corpus of text (almost a terabyte of data). The dictionary is far too large to pass around as a broadcast variable that is typically done in most examples (eg, chapter 6 of Advanced Analytics with Spark). So, an external data store is needed. Simply because of various constraints, it has to be HBase version 0.98.18 and Spark 1.3.1.

The data is highly confidential and must be protected by Kerberos. The system is set up by simply using su to start a new shell thus giving a fresh ticket. The application runs under YARN and the Driver connects to HBase and initialises the tables before outsourcing the work to the Executors. And it's there were we see "Failed to find any Kerberos tgt" [Ticket Giving Ticket] when the Executors try to access these tables.

What is happening looks like this:

From "Hadoop: The Definitive Guide" by Tom White

The exact stack trace I was seeing is described at StackOverflow here but unfortunately their solution didn't work for me.

After a lot of head-scratching, I looked at the code in the new(ish) hbase-spark codebase. After some judicious thievery from the HBaseContext code, I wrote something like:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.mapreduce.{Job, TableMapReduceUtil}
import org.apache.hadoop.security.{Credentials, UserGroupInformation}
import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod
import org.apache.spark.{SparkContext, SerializableWritable}
import org.apache.spark.broadcast.Broadcast

object HBaseKerberosSecurity {

  def callThisFromTheDriverCode(config: Configuration, sc: SparkContext): Broadcast[SerializableWritable[Credentials]] = {
    val job = Job.getInstance(config)
    TableMapReduceUtil.initCredentials(job)
    sc.broadcast(new SerializableWritable(job.getCredentials))
  }

  def callThisFromYourExecutorCode(credentialConf: Broadcast[SerializableWritable[Credentials]]): Unit = {
    val ugi = UserGroupInformation.getCurrentUser
    val credentials = ugi.getCredentials // see YarnSparkHadoopUtils
    if (credentials != null) {
      ugi.addCredentials(credentials)
      ugi.setAuthenticationMethod(AuthenticationMethod.PROXY)
      ugi.addCredentials(credentialConf.value.value)
  }

}

and success! The executors too were allowed to read and write from HBase.

Note: make sure that this is called before you ever need a connection. If you call it once but your executor dies, a new one will come up and carry on from where the last one finished. If this code is not called again, you won't be authenticated.

SSH: Performance and Behaviour

The problem

We had set up a Jetty server to use SSL much as in a previous post. This was fine until we started pooling connections (pooling connections didn't greatly improve performance in London-to-London communication but improved Hong Kong-to-London performance by about 30% since the ping time for a packet was a huge 220ms). But when we pooled connections and introduced SSL, all communication froze after some happy-path results.

The problem did not appear to be with encryption itself as the first few requests succeeded. But after 30 hits, all the Jetty threads were blocked like this (from running jstack):

"qtp401625763-13" #13 prio=5 os_prio=0 tid=0x00007f3e2021b800 nid=0x1ecb runnable [0x00007f3e0d2e7000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
- locked <0x00000000d90404a8> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000d906e358> (a sun.security.ssl.AppInputStream)
at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:391)

(where we're using Jetty 7.6.15)

At first, we thought something was wrong with our certificates etc but a handful of requests at first went thorugh without issue. But look for this in the server-side logs when you have given the JVM argument -Djavax.net.debug=all

*** ServerHelloDone

on both the client and server side, then you know the server completed the handshake OK. Look for:

main, READ: TLSv1.2 Change Cipher Spec, length = 1

or, in Jetty: 

qtp1418621776-17, WRITE: TLSv1.2 Change Cipher Spec, length = 1

and you'll know the whole handshake pretty much completed OK.

What happens in SSL?

Asymmetric encryption used briefly at start-up to establish a more efficient cipher. The private keys in this exchange are ephemeral and generated on the server side as soon as it hears the client say "hello" and on the client side as soon as it receives the server's choice of key.

Syn

In kickstarting the SSL handshake, the client advertises all of its cipher suite codes, elliptic curve details etc to the server (see sun.security.ssl.HandshakeMessage$ClientHello.send(..) ). The client thread then blocks waiting for the server to respond.

Syn/Ack

Upon receiving the client's message, (see sun.security.ssl.ServerHandshaker.clientHello(..) ) the server chooses an algorithm that both client and server support. When I was stepping through the code, this appeared to be DSA. The server-side must have a public and private key that corresponds to this algorithm in its keystore (see ServerHandshaker.setupPrivateKeyAndChain(..)).

Here, a sun.security.ssl.DHCrypt is instantiated. From the JavaDocs:

"This class implements the Diffie-Hellman key exchange algorithm.  D-H means combining your private key with your partners public key to generate a number. The peer does the same with its private key and our public key. Through the magic of Diffie-Hellman we both come up with the same number. This number is secret (discounting MITM attacks) and hence called the shared secret."

Along with the server certificates, all of this is sent to the client and the server thread blocks.

Ack

The client then unblocks and deserializes the ServerHello created on the server side (via a bespoke deserialization process). It uses the cipher suite the server told it to use and checks the server's certificates and stores the server's public key. It then sends its own public key (via DHClientKeyExchange) to the server.

Symmetric cipher keys are then generated (Handshaker.calculateConnectionKeys), the client tells the server that it is ready to talk then blocks.

A little more server Ack

Given what the client and server have exchanged, the server now sets its own agreed secret key using the same method in Handshaker (that is the superclass to both ServerHandshaker and ClientHandshaker) and sends a Finished object back to the client. The server is now finished and it notifies an listeners in a separate thread (this listener can be found as an inner class in Jetty's SslConnectorEndPoint.run). The thread in SslConnectorEndPoint then awaits incoming data.

Introducing MAT

There's a very nice tool from the ladies and gentlemen of Eclipse called MAT. Very quickly, you can dump the memory of a JVM and query its contents with a SQL-like language. For instance, I found all the client-side sockets that were listening to my server port of 8192 by executing:

select * from java.net.SocksSocketImpl where port = 8192

Interestingly, all their incoming references originated in the connection pool. So that is what is keeping them open and correspondingly keeping the server threads listening for a request that never comes! All client threads doing something useful are blocked by Jetty not having any server threads to service them; and all those Jetty threads are listening on sockets whose other end is held open by the idle sockets in the client's connection pool.

Conclusion

By replacing Jetty's SslSocketConnector with its SslSelectChannelConnector, the server behaves asynchronously and connection pooling doesn't make it grind to a halt.

We rolled-our own nonce encryption and it performed poorly. When we got rid of it, we found that Jetty using SSL had roughly the same performance as plain, clear-text HTTP.


Further reading

1. A very good (maths) blog on elliptic curves.

Java Security Crib Sheet

Certificates

"A certificate is a statement, issued by one person, that the public key of another person has a certain value. Essentially, a certificate is a signed public key. Marian creates the certificate by placing some information about her, some information about Will, and Will's public key value into a file. She then signs the file with her own private key. Robin Hood (or anyone else) can download this certificate and verify it using Marian's public key. Robin Hood trusts Marian, so he also now has a trustworthy copy of Will's public key, which he can use to verify files signed by Will.

"To verify a certificate, you need a public key.To verify a public key, you need a certificate. Essentially, one certificate can be verified by another, which is verified by another and so forth. This is called certificate chaining. The chain can't be infinite, so where does it start? The certificate chain starts with a certificate whose issuer and subject are the same. Usually such a certificate is issued by a Certificate Authority."

KeyStore

"A KeyStore is a handy box that holds keys and certificates. One KeyStore contains all the information a single person (or application, or identity) needs for authentication. Usually, you have two distinct uses for authentication

 - You need to prove to others who you are
 - You need to make sure that other people are legitimate

"In the first case you can use a private key to sign data. A certificate that contains the matching public key can be used to prove your identity... The private key is used to sign data; the certificates can be presented as credentials backing up the signature.

"In the second case, you can use other people's certificates to prove to yourself that they are who they say they are."

Message Digest

"A message digest is a special number calculated from a set of input data. If you are familiar with hash functions, it will help you to know that a message digest is a lot like a hash value, except longer. Message digests are sometimes called secure hash functions or cryptographic has functions."

MACs

"A Message Authentication Code (MAC), for example, is basically a message digest with an associated key. It produces a short value based on both its input data and the key. In theory, only someone with the same key can produce the same MAC from the same input data."

"MACs differ from digital signatures as MAC values are both generated and verified using the same secret key." [1]

Signing Files

"Another approach to authentication comes from the combination of a message digest and an asymmetric cipher. If Marian encrypts the message digest with her private key, Robin Hood can download the encrypted message digest, decrypt it using Marian's public key, and compare the message digest to one that he computes from the downloaded file. If they match, then he can be sure that the file is correct."

"The encrypted digest is called a signature; Marian has signed the file."

All quotes taken from Java Cryptography, Jonathan Knudsen, except:

[1] Wikipedia