See all posts

Debugging Keycloak and suomi.fi

by Topi Pohjosaho | 06 Jun 2024
Keycloak ● SAML ● suomi.fi ● AWS

TL;DR Since version 24, Keycloak refuses to decrypt SAML assertions using a signing certificate. We worked around this by adding the same certificate to Keycloak as an encryption certificate. This can be done using the Web UI or kcadm.sh with a JSON payload.

Background

Like many others, our AWS based system uses Keycloak to implement suomi.fi Single Sign-On. Read more about this setup from our cloud blog.

Problem

It was supposed to be a routine upgrade from Keycloak 23 to 24. I had not worked with Keycloak before, but we have good documentation for the process.

However, after updating our test Keycloak to 24.0.4, suomi.fi login stopped working.

Keycloak internal error

In the Keycloak log we could see the error PL00092: Null Value:Key for EncryptedData not found.

2024-05-16 10:23:54,030 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1) Uncaught server error: org.keycloak.broker.provider.IdentityBrokerException: Could not process response from SAML identity provider.
  at org.keycloak.broker.saml.SAMLEndpoint$Binding.handleLoginResponse(SAMLEndpoint.java:598)
  at org.keycloak.broker.saml.SAMLEndpoint$Binding.handleSamlResponse(SAMLEndpoint.java:681)
  at org.keycloak.broker.saml.SAMLEndpoint$Binding.execute(SAMLEndpoint.java:287)
  at org.keycloak.broker.saml.SAMLEndpoint.postBinding(SAMLEndpoint.java:192)
  at org.keycloak.broker.saml.SAMLEndpoint$quarkusrestinvoker$postBinding_e2ae3e4e98121b36952f2279cd4bb60100612099.invoke(Unknown Source)
  at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
  at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
  at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
  at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
  at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
  at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
  at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: PL00092: Null Value:Key for EncryptedData not found.
  at org.keycloak.saml.common.DefaultPicketLinkLogger.nullValueError(DefaultPicketLinkLogger.java:195)
  at org.keycloak.saml.processing.core.util.XMLEncryptionUtil.decryptElementInDocument(XMLEncryptionUtil.java:287)
  at org.keycloak.saml.processing.core.saml.v2.util.AssertionUtil.decryptAssertion(AssertionUtil.java:612)
  at org.keycloak.broker.saml.SAMLEndpoint$Binding.handleLoginResponse(SAMLEndpoint.java:458)
  ... 14 more

A lot of digging ensued. Eventually, my colleague found this in version 21 release notes:

SAML SP metadata changes

In this version, Keycloak will refuse to decrypt assertions encrypted using a realm key generated for signing purpose. This change means all encrypted communication from IDP to SP (where Keycloak acts as the SP) will stop working.

There are two ways to make this work:

  • either update the IDP configuration with the metadata generated by a newer version of Keycloak,
  • or run Keycloak in backward compatibility mode that will make Keycloak work with the metadata generated by older Keycloak versions. This mode can be enabled using -Dkeycloak.saml.deprecated.encryption=true flag. Note this backward compatibility mode is planned to be removed in Keycloak 24.

The backwards compatibility flag -Dkeycloak.saml.deprecated.encryption=true has been removed in version 24, but it was not mentioned in version 24 release notes.

This is the removed code in SAMLEndpoint.java.

Keycloak now refuses to use a key with use SIG (signature) for decryption in SAMLDecryptionKeysLocator.java:

Stream<KeyWrapper> keysStream = session.keys().getKeysStream(realm)
        .filter(key -> key.getStatus().isEnabled() && KeyUse.ENC.equals(key.getUse()));

Solution

We solved this by simply adding the same certificate with use ENC (encryption). This can be done using the admin console, or programmatically. The result is that we have the same certificate twice under Realm settings -> Keys, both with use SIG and ENC. This way we can upgrade Keycloak without changing suomi.fi configuration.

Note that Keycloak key use (SIG/ENC) has nothing to do with x.509 key usage extension - Keycloak does not seem to care about that.

Adding the encryption certificate programmatically

We have a bash script that is run after Keycloak starts. The script adds our signing certificate from a Java keystore, if not already added.

We added this to the script:

# Add the same certificate for encryption use
# In version 24, Keycloak will refuse to decrypt assertions encrypted using a realm key 
# generated for signing purpose.
ENCRYPTION_ALGORITHM="RSA-OAEP"
provider_id=$(kcadm.sh get keys -r suomifi | \
  jq -r ".keys[] select(.algorithm==\"${ENCRYPTION_ALGORITHM}\") | .providerId")
if [ -z "$provider_id" ]
then
  echo "PROVISION: Add encryption certificate"

  # Escape newlines
  PRIVATE_KEY=$(awk '{printf "%s\\n", $0}' $PRIVATE_KEY_FILE)
  PUBLIC_KEY=$(awk '{printf "%s\\n", $0}' $PUBLIC_KEY_FILE)

  JSON_STRING="{
    \"name\" : \"rsa-enc\",
    \"providerId\" : \"rsa-enc\",
    \"providerType\" : \"org.keycloak.keys.KeyProvider\",
    \"parentId\" : \"suomifi\",
    \"config\" : {
      \"privateKey\": [\"${PRIVATE_KEY}\"],
      \"certificate\" : [\"${PUBLIC_KEY}\"],
      \"active\" : [ \"true\" ],
      \"priority\" : [ \"103\" ],
      \"enabled\" : [ \"true\" ],
      \"algorithm\" : [ \"${ENCRYPTION_ALGORITHM}\" ]
    }
  }"

  # Add certificate from JSON payload (RSA-OAEP not supported by JavaKeystoreKeyProvider)
  echo "$JSON_STRING" | kcadm.sh create components \
    --file - \
    --target-realm suomifi \
    --no-config \
    --server http://localhost:8080 \
    --realm master \
    --user "$KEYCLOAK_ADMIN" \
    --password "$KEYCLOAK_ADMIN_PASSWORD"
else
  echo "PROVISION: Found certificate with algorithm ${ENCRYPTION_ALGORITHM}, no need to add."
fi

The official documentation does not mention using JSON payloads, but this is the only way we got this to work. Keycloak is not able to add a key with the algorithm RSA-OAEP from a Java keystore.

We found the JSON payload solution from this blog post: Managing custom realm keys in Keycloak programmatically.

Hopefully this post helps someone else.