BranchesTags

Repository files navigation

ci

kafkaproxy

kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.

Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This can be a cumbersome restriction in several different situations, such as:

  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes

This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints where the the proxy instances can be reached.

Run kafkaproxy in Docker

kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.

Command line

The following configuration parameters are mandatory:

  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.

For example:

docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.

Demonstration setup with docker-compose

If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the example directory. So clone the kafkaproxy repo and run the following commands:

Step 1: Start kafka, zookeeper and kafkaproxy.

docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.

Step 2: Create my-test-topic.

docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.

echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.

docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.

docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is only available from within the docker network as kafka1:9092. The kafkaproxy is configured to proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.

Configuration

kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following section describe the configuration options in detail.

General configuration

kafkaproxy requires some general information to start.

NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.

Client SSL configuration

The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances. Configuration can be provided using the following environment variables:

NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.

Kafka SSL configuration

The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances. Configuration can be provided using the following environment variables:

NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.

Features

  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via
    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA

Further Reading

About

No description, website, or topics provided.

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
ci
kafkaproxy
kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.
Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during
client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This
can be a cumbersome restriction in several different situations, such as:
  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes
This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting
the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints
where the the proxy instances can be reached.
Run kafkaproxy in Docker
kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.
Command line
The following configuration parameters are mandatory:
  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.
For example:
docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.
Demonstration setup with docker-compose
If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the
example directory. So clone the kafkaproxy repo and run the following commands:
Step 1: Start kafka, zookeeper and kafkaproxy.
docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.
Step 2: Create my-test-topic.
docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.
echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.
docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.
docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is
only available from within the docker network as kafka1:9092. The kafkaproxy is configured to
proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.
Configuration
kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following
section describe the configuration options in detail.
General configuration
kafkaproxy requires some general information to start.
NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.
Client SSL configuration
The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.
Kafka SSL configuration
The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.
Features
  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via

    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA
Further Reading
About
No description, website, or topics provided.
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
1 star
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ci
kafkaproxy
kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.
Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during
client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This
can be a cumbersome restriction in several different situations, such as:
  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes
This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting
the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints
where the the proxy instances can be reached.
Run kafkaproxy in Docker
kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.
Command line
The following configuration parameters are mandatory:
  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.
For example:
docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.
Demonstration setup with docker-compose
If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the
example directory. So clone the kafkaproxy repo and run the following commands:
Step 1: Start kafka, zookeeper and kafkaproxy.
docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.
Step 2: Create my-test-topic.
docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.
echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.
docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.
docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is
only available from within the docker network as kafka1:9092. The kafkaproxy is configured to
proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.
Configuration
kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following
section describe the configuration options in detail.
General configuration
kafkaproxy requires some general information to start.
NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.
Client SSL configuration
The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.
Kafka SSL configuration
The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.
Features
  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via

    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA
Further Reading
About
No description, website, or topics provided.
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
1 star
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ci
kafkaproxy
kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.
Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during
client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This
can be a cumbersome restriction in several different situations, such as:
  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes
This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting
the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints
where the the proxy instances can be reached.
Run kafkaproxy in Docker
kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.
Command line
The following configuration parameters are mandatory:
  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.
For example:
docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.
Demonstration setup with docker-compose
If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the
example directory. So clone the kafkaproxy repo and run the following commands:
Step 1: Start kafka, zookeeper and kafkaproxy.
docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.
Step 2: Create my-test-topic.
docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.
echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.
docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.
docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is
only available from within the docker network as kafka1:9092. The kafkaproxy is configured to
proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.
Configuration
kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following
section describe the configuration options in detail.
General configuration
kafkaproxy requires some general information to start.
NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.
Client SSL configuration
The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.
Kafka SSL configuration
The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.
Features
  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via

    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA
Further Reading
About
No description, website, or topics provided.
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
1 star
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
ci
kafkaproxy
kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.
Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during
client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This
can be a cumbersome restriction in several different situations, such as:
  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes
This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting
the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints
where the the proxy instances can be reached.
Run kafkaproxy in Docker
kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.
Command line
The following configuration parameters are mandatory:
  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.
For example:
docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.
Demonstration setup with docker-compose
If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the
example directory. So clone the kafkaproxy repo and run the following commands:
Step 1: Start kafka, zookeeper and kafkaproxy.
docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.
Step 2: Create my-test-topic.
docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.
echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.
docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.
docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is
only available from within the docker network as kafka1:9092. The kafkaproxy is configured to
proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.
Configuration
kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following
section describe the configuration options in detail.
General configuration
kafkaproxy requires some general information to start.
NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.
Client SSL configuration
The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.
Kafka SSL configuration
The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.
Features
  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via

    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA
Further Reading
About
No description, website, or topics provided.
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
1 star
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ci
kafkaproxy
kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.
Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during
client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This
can be a cumbersome restriction in several different situations, such as:
  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes
This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting
the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints
where the the proxy instances can be reached.
Run kafkaproxy in Docker
kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.
Command line
The following configuration parameters are mandatory:
  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.
For example:
docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.
Demonstration setup with docker-compose
If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the
example directory. So clone the kafkaproxy repo and run the following commands:
Step 1: Start kafka, zookeeper and kafkaproxy.
docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.
Step 2: Create my-test-topic.
docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.
echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.
docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.
docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is
only available from within the docker network as kafka1:9092. The kafkaproxy is configured to
proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.
Configuration
kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following
section describe the configuration options in detail.
General configuration
kafkaproxy requires some general information to start.
NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.
Client SSL configuration
The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.
Kafka SSL configuration
The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.
Features
  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via

    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA
Further Reading
About
No description, website, or topics provided.
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
1 star
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ci
kafkaproxy
kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.
Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during
client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This
can be a cumbersome restriction in several different situations, such as:
  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes
This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting
the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints
where the the proxy instances can be reached.
Run kafkaproxy in Docker
kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.
Command line
The following configuration parameters are mandatory:
  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.
For example:
docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.
Demonstration setup with docker-compose
If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the
example directory. So clone the kafkaproxy repo and run the following commands:
Step 1: Start kafka, zookeeper and kafkaproxy.
docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.
Step 2: Create my-test-topic.
docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.
echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.
docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.
docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is
only available from within the docker network as kafka1:9092. The kafkaproxy is configured to
proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.
Configuration
kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following
section describe the configuration options in detail.
General configuration
kafkaproxy requires some general information to start.
NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.
Client SSL configuration
The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.
Kafka SSL configuration
The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.
Features
  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via

    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA
Further Reading
About
No description, website, or topics provided.
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
1 star
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
ci
kafkaproxy
kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.
Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during
client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This
can be a cumbersome restriction in several different situations, such as:
  • Network topologies preventing direct access to the broker nodes
  • Multiple networks from which broker nodes should be reachable
  • DNS resolution restrictions when accessing TLS secured broker nodes
  • Using the sidecar pattern for TLS termination in Kubernetes
This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting
the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints
where the the proxy instances can be reached.
Run kafkaproxy in Docker
kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.
Command line
The following configuration parameters are mandatory:
  • KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
  • KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
  • KAFKAPROXY_BOOTSTRAP_SERVERS: the bootstrap server via which the kafka cluster can be contacted. This is usually a load balancer in front of the kafka cluster.
For example:
docker run \
--net host \
-e KAFKAPROXY_HOSTNAME=localhost \
-e KAFKAPROXY_BASE_PORT=4000 \
-e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
-d dajudge/kafkaproxy:0.0.3

Note: You will have to make the proxy ports defined in your broker map available from outside the container with -p PORT:PORT if you're not using --net host.
Demonstration setup with docker-compose
If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the
example directory. So clone the kafkaproxy repo and run the following commands:
Step 1: Start kafka, zookeeper and kafkaproxy.
docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.
Step 2: Create my-test-topic.
docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.
echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.
docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.
docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is
only available from within the docker network as kafka1:9092. The kafkaproxy is configured to
proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.
Configuration
kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following
section describe the configuration options in detail.
General configuration
kafkaproxy requires some general information to start.
NameDefault valueDestription
KAFKAPROXY_HOSTNAMEThe hostname of the proxy as seen by the clients.
KAFKAPROXY_BASE_PORTThe base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.
KAFKAPROXY_BOOTSTRAP_SERVERSThe comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.
KAFKAPROXY_LOG_LEVELINFOThe log level of the root logger. This must be a valid log level for logback.
Client SSL configuration
The client SSL configuration determines how the Kafka clients have to connect to the kafkaproxy instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_CLIENT_SSL_ENABLEDfalseEnables SSL encrypted communication between clients and kafkaproxy.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.
KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORDThe password to access the proxy's server key store. Provide no value if the key store is not password protected.
KAFKAPROXY_CLIENT_SSL_KEY_PASSWORDThe password to access the proxy's server key. Provide no value if the key is not password protected.
KAFKAPROXY_CLIENT_SSL_AUTH_REQUIREDfalseRequire a valid client certificate from clients connecting to the proxy.
Kafka SSL configuration
The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.
Configuration can be provided using the following environment variables:
NameDefault valueDestription
KAFKAPROXY_KAFKA_SSL_ENABLEDfalseEnables SSL encrypted communication kafkaproxy and the Kafka brokers.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATIONThe filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.
KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORDThe password to access the trust store. Provide no value if the trust store is not password protected.
KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAMEtrueIndicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.
KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGYNONE
  • NONE: don't use a client certificate for broker connections.
  • KEYSTORE: use the client certificate provided by KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION
KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATIONThe filesystem location of the proxy's client key store. Required only when KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY is set to KEYSTORE.
KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORDThe password to access the proxy's client key store. Provide no value if the key store is not password protected.
KAFKAPROXY_KAFKA_SSL_KEY_PASSWORDThe password to access the proxy's client key. Provide no value if the key is not password protected.
Features
  • SSL support from client to proxy
  • SSL support from proxy to broker
  • TODO: Forwarding 2-way client SSL authentication information via on-the-fly generation of impostor certificates via

    • local KeyPair generation / signing
    • CFSSL
    • AWS Certificate Manager Private CA
Further Reading
About
No description, website, or topics provided.
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
1 star
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages