Page History

...

Unpack to some location, like /opt/kafka, which will be referred to as <kafkaInstallationDir>.

Kafka has historically leveraged the Zookeeper software for its controller layer, but as Use Kafka Raft ("KRaft") for Kafka's controller layer. As of Kafka 3.3.1 (Oct 2022) it’s recommended to use the native Kafka Raft (“KRaft”) which improves the native KRaft is recommended over the Zookeeper controller software. KRaft improves upon performance and consolidates the server configuration files and process to 1 each (per node). NOTE: These instructions where tested with Kafka version 3.3.1.

Configure server.properties

In KRaft mode, the relevant configuration file is located at <kafkaInstallationDir>/config/kraft/server.properties.

A Use a minimum server cluster size of 3 is recommended for in SHRINE production environments, in which each server node functions as both a broker and controller. This is enabled by setting the process.roles to broker,controller, as noted in the Server Basics section:

...

Create Server Keystores and Truststores

TODO: bit about creating a keystore unique to each server node, and a common truststore. Catalyst has its own CA with self-signed wildcard certs, enabling 1 shared server keystore and 1 shared truststore. Production clusters must get certs signed by a real CA with no wildcard CNs. This is yet-untested on internal systems so we have no proven documentation for it.

In order to secure traffic through the internet with TLS/SSL, Kafka requires client/server authentication via public key infrastructure (PKI). Each Kafka server needs a keystore for identifying itself, while each client and server needs a truststore to authenticate servers. As the Kafka administrator you may choose to have all server certificates signed by a true Certificate Authority (CA), or to manage a private CA within your organization and use it for signing.

In either case, each Kafka server node's keystore must store a unique private key (always kept secure) and certificate (which gets signed). Each server node's truststore must store a list of all signed server certificates, or alternatively a CA's own cert, in order to let server nodes behave as logical clients during inter-broker communication. The servers' truststore contents can be identical to that of clients. One benefit of managing a private CA is enabling all client and server truststores to identically contain only the CA's cert, in effect telling all systems in the network to trust the CA and every cert signed by it. Additionally this enables server nodes to join or leave the cluster without truststores needing cert addition or revocation. See here for comprehensive documentation on PKI architecture and keystore management in Kafka:

https://kafka.apache.org/33/documentation.html#security_ssl

https://docs.confluent.io/platform/current/security/security_tutorial.html

Create keystores and truststores in PKCS12 format. When creating, you will be prompted for passwords. Add the file Add the keystore and truststore locations and passwords to the end of server.properties:

# ssl.key.password= needed if using real CA?<thisServersKeystorePassword>
ssl.keystore.location=<path/to/this/servers/kafka_server_keystore.pkcs12>
ssl.keystore.password=<thisServersKeystorePassword>
ssl.truststore.location=<path/to/shared/kafka_server/_truststore.pkcs12>
ssl.truststore.password=<sharedServerTruststorePassword>

...

shrine {
...
  kafka {
    sasl.jaas.username = "yourShrineHubUser"
    ssl.truststore.location = "path/to/your/kafka_client/_truststore.pkcs12"
  }
...
}//shrine

...

shrine.kafka.sasl.jaas.username = "kafka-admin"
shrine.kafka.ssl.truststore.location = "path/to/your/kafka_client/_truststore.pkcs12"

shrine.kafka.sasl.jaas.password = "yourKafkaAdminPassword" 
shrine.kafka.ssl.truststore.password = "clientTruststorePassword"

...