1, introduction

This topic provides instructions for installing the Confluent platform configuration that can be used for production in a multi node environment using the replicated ZooKeeper ensemble.

With this installation method, you can manually connect to each node, download the archive file, and run the conflict platfrom installation command.

Note: you must complete these steps for each node in the cluster.

Prerequisite
Before installing the fluent platform, your environment must have the following Software and hardware requirements.

2. Get software

1. Go to the download page, select your archive, or use curl to download directly

Confluent Platform
ZIP

curl -O http://packages.confluent.io/archive/5.4/confluent-5.4.0-2.12.zip

TAR

curl -O http://packages.confluent.io/archive/5.4/confluent-5.4.0-2.12.tar.gz

Fluent platform only uses fluent community components
ZIP

curl -O http://packages.confluent.io/archive/5.4/confluent-community-5.4.0-2.12.zip

TAR

curl -O http://packages.confluent.io/archive/5.4/confluent-community-5.4.0-2.12.tar.gz

Extract the contents of the archive. For ZIP files, run this command on the terminal:

unzip confluent-5.4.0-2.12.zip

For tar files:

tar xzf confluent-5.4.0-2.12.tar.gz

3. Configuration file for Confluent Platform

Configure the fluent platform using the component property files. By default, they are in < path to configure > / etc /. You must configure at least the following components:

(1)Zookeeper

These instructions assume that you are running ZooKeeper in copy mode. Replication mode requires at least three servers, and failover must have an odd number of servers. For more information, see ZooKeeper document.

1. Navigate to the ZooKeeper properties file (/ etc / Kafka / ZooKeeper. Properties) and modify it as follows.

tickTime=2000
dataDir=/var/lib/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24

This configuration is used for three node integration. The configuration file should be the same on all nodes in the integration. Ticketime, dataDir, and clientPort are all set to typical single server values. initLimit and syncLimit control how long it takes for the following zookeeper server to initialize the current leader and how long it takes for the leader to be out of sync. In this configuration, a following can take 10000 milliseconds to initialize, and it can be out of sync within 4000 milliseconds according to the ticketime set to 2000 Ms.

The server. * property is a member variable in the set. The form is:
server.<myid>=<hostname>:<leaderport>:<electionport>

• Myid is the server identification number. There are three servers, each with a different myid of 1, 2, and 3. Set myid by creating a file named myid in dataDir that contains an ASCII encoded integer. This value must match a myid value in the configuration file. If another possible member has started with a conflicting myid value, you will see an error.
• The leader port is used by the follower to connect to the existing leader. This port should be open between all Zookeeper collection members
• electionport is used to show that leader s are selected among all members.

autopurge.snapRetainCount and autopurge.purgeInterval are set to purge all snapshots except three snapshots every 24 hours.

2. Navigate to the ZooKeeper log directory (for example, / var/lib/zookeeper /) and create a file named myid. The myid file consists of a line containing the machine ID in the format < machine ID >. When the ZooKeeper server starts, it knows which server it is by referencing the myid file. For example, server 1 has a myid value of 1.

(2)Kafka

In a production environment, multiple brokers are required. During startup, broker registers itself in ZooKeeper and becomes a member of the cluster.

Navigate to the Apache Kafka ® properties file (/ etc/ Kafka /server.properties), and customize the following:

• Connect to the same zookeeper sensitive by setting zookeeper.connect in all nodes to the same value. Replace all instances of localhost with hostname or FQDN (fully qualified domain name of the node). For example, if your hostname is zookeeper:
  zookeeper.connect=zookeeper:2181

• Use one of these methods to configure the broker id for each node in the cluster.
  Generate broker id dynamically: add broker.id.generation.enable=true and comment out broker.id. for example

  ############################# Server Basics #############################
  
  # The ID of the broker. This must be set to a unique integer for each broker.
  #broker.id=0
  broker.id.generation.enable=true
  

  Set broker id manually: set a unique value for broker.id on each node. Configure how other brokers and clients use listeners and (optionally) advertised.listeners to communicate with brokers.
  ○ listeners: the comma separated list of the uri and the listener name to listen to.
  ○ advertised.listeners: the comma separated list of uri and the listener name to be used by other broker s and clients. The advertised.listeners parameter ensures that the agent declares an address that can be accessed from both local and external hosts.

  For more information, see Production Configuration Options

• Configure security for your environment.
  For general safety instructions, see Security
  For role-based access control (RBAC), see Configure Metadata Service (MDS)
  For SSL encryption, SASL authentication, and authorization, see Security Tutorial

(3)Control Center

1. Navigate to the Control Center properties file (/ etc / configure Control Center / Control Center - production. Properties) and customize the following:

# host/port pairs to use for establishing the initial connection to the Kafka cluster
bootstrap.servers=<hostname1:port1,hostname2:port2,hostname3:port3,...>

# location for Control Center data
confluent.controlcenter.data.dir=/var/lib/confluent/control-center

# the Confluent license
confluent.license=<your-confluent-license>

# ZooKeeper connection string with host and port of a ZooKeeper servers
zookeeper.connect=<hostname1:port1,hostname2:port2,hostname3:port3,...>

This configuration is for a three node, multi node cluster. For more information, Control Center configuration details . for information about Confluent platform licenses, see Managing Confluent Platform Licenses.

2. Navigate to the Kafka server configuration file (/ etc/ Kafka /server.properties) and enable the fluent metrics reporter.

##################### Confluent Metrics Reporter #######################
# Confluent Control Center and Confluent Auto Data Balancer integration
#
# Uncomment the following lines to publish monitoring data for
# Confluent Control Center and Confluent Auto Data Balancer
# If you are using a dedicated metrics cluster, also adjust the settings
# to point to your metrics Kafka cluster.
metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=localhost:9092

#
# Uncomment the following line if the metrics cluster has a single broker
confluent.metrics.reporter.topic.replicas=1

3. Add these lines to the Kafka connection properties file (/ etc / Kafka / connect - distribution. Properties) to add support for interceptors.

# Interceptor setup
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

(4)Schema Registry

Navigate to the schema registry properties file (/ etc / schema - registry / schema - Registry. Properties) and specify the following properties

# Specify the address the socket server listens on, e.g. listeners = PLAINTEXT://your.host.name:9092
listeners=http://0.0.0.0:8081

# The host name advertised in ZooKeeper. This must be specified if your running Schema Registry
# with multiple nodes.
host.name=192.168.50.1

# List of Kafka brokers to connect to, e.g. PLAINTEXT://hostname:9092,SSL://hostname2:9092
kafkastore.bootstrap.servers=PLAINTEXT://hostname:9092,SSL://hostname2:9092

This configuration is for a three node, multi node cluster. For more information, see Running Schema Registry in Production.

4. Open Confluetn Platform

Use the Kafka CLI command to install the fluent platform.

Note: ZooKeeper, Kafka, and Schema Registry must be started in this particular order and before any other components.

1. Turn on Zookeeper and run this command on its own terminal

<path-to-confluent>/bin/zookeeper-server-start <path-to-confluent>/etc/kafka/zookeeper.properties

2. Turn on Kafka and run this command on its own terminal

<path-to-confluent>/bin/kafka-server-start <path-to-confluent>/etc/kafka/server.properties

3. Open Schema Registry and run the command on its own terminal

<path-to-confluent>/bin/schema-registry-start <path-to-confluent>/etc/schema-registry/schema-registry.properties

4. Start other fluent platform components according to your own needs.
Control Center

<path-to-confluent>/bin/control-center-start <path-to-confluent>/etc/confluent-control-center/control-center.properties

Kafka Connect

<path-to-confluent>/bin/control-center-start <path-to-confluent>/etc/confluent-control-center/control-center.properties

Confluetn REST Proxy

<path-to-confluent>/bin/kafka-rest-start <path-to-confluent>/etc/kafka-rest/kafka-rest.properties

KSQL

<path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksql/ksql-server.properties

5. Uninstalling the fluent platform

1. Delete the conflict directory. For example, if you already have Confluent Platform 5.4.0 installed

rm -rf confluent-5.4.0

2. Delete data file of Confluent Platfrom

rm -rf /var/lib/<confluent-platform-data-files>

Reference documents: Manual Install using ZIP and TAR Archives