Kafka producer client source parsing
To analyze the Kafka producer source code, let's first take a look at an example of how producer sends a message, trace the source code through the example, and study what the producer does during the entire sending process, as well as the architecture design.
Send message example
Construct Property Object
Property object sets the necessary Kafka producer side configuration information
//Create Property Object
Properties properties = new Properties();
//Setting the broker's address allows you to support multiple brokers, so you can use other brokers when some brokers hang up. Must be specified
properties.put("bootstrap.servers", "172.23.16.84:9092");
//Any message format sent to the broker side must be a byte array, so it needs to be serialized, specifying the serializer for the key, except using the one provided by default
//Serializer, or you can create a custom serializer by implementing the Serializer interface
properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
//Specify the serializer for the value part
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
//Specify the amount of feedback that Producer needs to collect before the request is sent, 0 will not wait for feedback from any server to think that the send was successful
//; 1 If you wait for the leader node to write to the local log, you think the send is successful, you don't care about the synchronization of the follower node, the leader node hangs up, but the follwer node does
//Failure to synchronize will result in message loss; All or -1 leader nodes will wait for all replicas to confirm that the message was sent successfully
properties.put("acks", "-1");
//Set the number of send message retries
properties.put("retries", 3);
//Batch size
properties.put("batch.size", 323840);
//delay time
properties.put("linger.ms", 10);
//Buffer size
properties.put("buffer.memory", 33554432);
//Maximum blocking time
properties.put("max.block.ms", 3000);
//Set compression algorithm, no compression by default
properties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
Construct Producer
Create a producer using the attribute object constructed in the previous step
//Build producers Producer producer = new KafkaProducer(properties);
Send messages synchronously
Call producer's send method to complete message sending
// Send messages synchronously
try {
producer.send(new ProducerRecord("my-topic",
Integer.toBinaryString(1), Integer.toString(1))).get();
} catch (InterruptedException e) {
throw new RuntimeException(e);
} catch (ExecutionException e) {
throw new RuntimeException(e);
}
Close Producer
After the message is sent, if you do not need to continue using the producer, you need to shut down and free up resources
//Close Producer producer.close();
Send Message Source Resolution
A typical producer send message flow is shown above. For the above process, let's examine the Kafka producer side send message source. There is nothing to study about the first step of constructing a configuration attribute object, just setting the attribute and its corresponding value. Let's start with the second step
Constructing Producers Using Attribute Objects
Initialize the producer by calling the KafkaProducer(properties) constructor, which internally calls the overloaded constructor KafkaProducer(Properties properties, Serializer keySerializer, Serializer valueSerializer) to complete the initialization, where keySerializer, valueSerializer specify the key/value serialization class, which overrides the key/value serialization class set in the property object
public KafkaProducer(Properties properties) {
this(properties, null, null);
}
public KafkaProducer(Properties properties, Serializer<K> keySerializer, Serializer<V> valueSerializer) {
this(Utils.propsToMap(properties), keySerializer, valueSerializer);
}
public KafkaProducer(Map<String, Object> configs, Serializer<K> keySerializer, Serializer<V> valueSerializer) {
this(new ProducerConfig(ProducerConfig.appendSerializerToConfig(configs, keySerializer, valueSerializer)),
keySerializer, valueSerializer, null, null, null, Time.SYSTEM);
}
Then proceed to call the overloaded constructor KafkaProducer (Map < String, Object > configs, Serializer keySerializer, Serializer valueSerializer), complete the construction producer, call propsToMap (Properties) before calling this constructor to convert the Properties property configuration to Map type and do the necessary checks, and the Properties class is actually java. Inherit the Map structure of Hashtable<Object, Object>under the util package. Traverse through the properties object in the propsToMap method and check if the key is of type String or throw a ConfigException exception if not
public static Map<String, Object> propsToMap(Properties properties) {
Map<String, Object> map = new HashMap<>(properties.size());
for (Map.Entry<Object, Object> entry : properties.entrySet()) {
if (entry.getKey() instanceof String) {
String k = (String) entry.getKey();
map.put(k, properties.get(k));
} else {
throw new ConfigException(entry.getKey().toString(), entry.getValue(), "Key must be a string.");
}
}
return map;
}
Then continue to call the overloaded constructor KafkaProducer (ProducerConfig config, Serializer keySerializer, Serializer valueSerializer, ProducerMetadata metadata, KafkaClient kafkaClient, ProducerInterceptors<K, V> interceptors, Time) to complete the producer creation. Before calling this constructor, configs, keySerializer, and valueSerializer are treated as follows
-
First call ProducerConfig. AppndSerializerToConfig (Map<String, Object> configs, Serializer<?> keySerializer, Serializer<?> valueSerializer) method sets keySerializer, valueSerializer override to configs, provided both are not empty
-
The ProducerConfig (Map <String, Object > props) constructor is then called to complete the ProducerConfig creation using the configs after the last step
ProducerConfig construction process
In the second step above, the ProducerConfig (Map<String, Object> props) constructor is called to complete the ProducerConfig construction, which calls the parent constructor AbstractConfig (ConfigDef definition, Map<?,?> originals) internally and passes in CONFIG when the parent constructor is called, which is modified by final and initialized in the static code block.
static {
CONFIG = new ConfigDef().define(BOOTSTRAP_SERVERS_CONFIG, Type.LIST, Collections.emptyList(), new ConfigDef.NonNullValidator(), Importance.HIGH, CommonClientConfigs.BOOTSTRAP_SERVERS_DOC)
.define(CLIENT_DNS_LOOKUP_CONFIG,
Type.STRING,
ClientDnsLookup.USE_ALL_DNS_IPS.toString(),
in(ClientDnsLookup.DEFAULT.toString(),
ClientDnsLookup.USE_ALL_DNS_IPS.toString(),
ClientDnsLookup.RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY.toString()),
Importance.MEDIUM,
CommonClientConfigs.CLIENT_DNS_LOOKUP_DOC)
.define(BUFFER_MEMORY_CONFIG, Type.LONG, 32 * 1024 * 1024L, atLeast(0L), Importance.HIGH, BUFFER_MEMORY_DOC)
.define(RETRIES_CONFIG, Type.INT, Integer.MAX_VALUE, between(0, Integer.MAX_VALUE), Importance.HIGH, RETRIES_DOC)
.define(ACKS_CONFIG,
Type.STRING,
"1",
in("all", "-1", "0", "1"),
Importance.HIGH,
ACKS_DOC)
.define(COMPRESSION_TYPE_CONFIG, Type.STRING, "none", Importance.HIGH, COMPRESSION_TYPE_DOC)
.define(BATCH_SIZE_CONFIG, Type.INT, 16384, atLeast(0), Importance.MEDIUM, BATCH_SIZE_DOC)
.define(LINGER_MS_CONFIG, Type.LONG, 0, atLeast(0), Importance.MEDIUM, LINGER_MS_DOC)
.define(DELIVERY_TIMEOUT_MS_CONFIG, Type.INT, 120 * 1000, atLeast(0), Importance.MEDIUM, DELIVERY_TIMEOUT_MS_DOC)
.define(CLIENT_ID_CONFIG, Type.STRING, "", Importance.MEDIUM, CommonClientConfigs.CLIENT_ID_DOC)
.define(SEND_BUFFER_CONFIG, Type.INT, 128 * 1024, atLeast(CommonClientConfigs.SEND_BUFFER_LOWER_BOUND), Importance.MEDIUM, CommonClientConfigs.SEND_BUFFER_DOC)
.define(RECEIVE_BUFFER_CONFIG, Type.INT, 32 * 1024, atLeast(CommonClientConfigs.RECEIVE_BUFFER_LOWER_BOUND), Importance.MEDIUM, CommonClientConfigs.RECEIVE_BUFFER_DOC)
.define(MAX_REQUEST_SIZE_CONFIG,
Type.INT,
1024 * 1024,
atLeast(0),
Importance.MEDIUM,
MAX_REQUEST_SIZE_DOC)
.define(RECONNECT_BACKOFF_MS_CONFIG, Type.LONG, 50L, atLeast(0L), Importance.LOW, CommonClientConfigs.RECONNECT_BACKOFF_MS_DOC)
.define(RECONNECT_BACKOFF_MAX_MS_CONFIG, Type.LONG, 1000L, atLeast(0L), Importance.LOW, CommonClientConfigs.RECONNECT_BACKOFF_MAX_MS_DOC)
.define(RETRY_BACKOFF_MS_CONFIG, Type.LONG, 100L, atLeast(0L), Importance.LOW, CommonClientConfigs.RETRY_BACKOFF_MS_DOC)
.define(MAX_BLOCK_MS_CONFIG,
Type.LONG,
60 * 1000,
atLeast(0),
Importance.MEDIUM,
MAX_BLOCK_MS_DOC)
.define(REQUEST_TIMEOUT_MS_CONFIG,
Type.INT,
30 * 1000,
atLeast(0),
Importance.MEDIUM,
REQUEST_TIMEOUT_MS_DOC)
.define(METADATA_MAX_AGE_CONFIG, Type.LONG, 5 * 60 * 1000, atLeast(0), Importance.LOW, METADATA_MAX_AGE_DOC)
.define(METADATA_MAX_IDLE_CONFIG,
Type.LONG,
5 * 60 * 1000,
atLeast(5000),
Importance.LOW,
METADATA_MAX_IDLE_DOC)
.define(METRICS_SAMPLE_WINDOW_MS_CONFIG,
Type.LONG,
30000,
atLeast(0),
Importance.LOW,
CommonClientConfigs.METRICS_SAMPLE_WINDOW_MS_DOC)
.define(METRICS_NUM_SAMPLES_CONFIG, Type.INT, 2, atLeast(1), Importance.LOW, CommonClientConfigs.METRICS_NUM_SAMPLES_DOC)
.define(METRICS_RECORDING_LEVEL_CONFIG,
Type.STRING,
Sensor.RecordingLevel.INFO.toString(),
in(Sensor.RecordingLevel.INFO.toString(), Sensor.RecordingLevel.DEBUG.toString(), Sensor.RecordingLevel.TRACE.toString()),
Importance.LOW,
CommonClientConfigs.METRICS_RECORDING_LEVEL_DOC)
.define(METRIC_REPORTER_CLASSES_CONFIG,
Type.LIST,
Collections.emptyList(),
new ConfigDef.NonNullValidator(),
Importance.LOW,
CommonClientConfigs.METRIC_REPORTER_CLASSES_DOC)
.define(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION,
Type.INT,
5,
atLeast(1),
Importance.LOW,
MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_DOC)
.define(KEY_SERIALIZER_CLASS_CONFIG,
Type.CLASS,
Importance.HIGH,
KEY_SERIALIZER_CLASS_DOC)
.define(VALUE_SERIALIZER_CLASS_CONFIG,
Type.CLASS,
Importance.HIGH,
VALUE_SERIALIZER_CLASS_DOC)
.define(SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG,
Type.LONG,
CommonClientConfigs.DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT_MS,
Importance.MEDIUM,
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_DOC)
.define(SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_CONFIG,
Type.LONG,
CommonClientConfigs.DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS,
Importance.MEDIUM,
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_DOC)
/* default is set to be a bit lower than the server default (10 min), to avoid both client and server closing connection at same time */
.define(CONNECTIONS_MAX_IDLE_MS_CONFIG,
Type.LONG,
9 * 60 * 1000,
Importance.MEDIUM,
CommonClientConfigs.CONNECTIONS_MAX_IDLE_MS_DOC)
.define(PARTITIONER_CLASS_CONFIG,
Type.CLASS,
DefaultPartitioner.class,
Importance.MEDIUM, PARTITIONER_CLASS_DOC)
.define(INTERCEPTOR_CLASSES_CONFIG,
Type.LIST,
Collections.emptyList(),
new ConfigDef.NonNullValidator(),
Importance.LOW,
INTERCEPTOR_CLASSES_DOC)
.define(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG,
Type.STRING,
CommonClientConfigs.DEFAULT_SECURITY_PROTOCOL,
Importance.MEDIUM,
CommonClientConfigs.SECURITY_PROTOCOL_DOC)
.define(SECURITY_PROVIDERS_CONFIG,
Type.STRING,
null,
Importance.LOW,
SECURITY_PROVIDERS_DOC)
.withClientSslSupport()
.withClientSaslSupport()
.define(ENABLE_IDEMPOTENCE_CONFIG,
Type.BOOLEAN,
false,
Importance.LOW,
ENABLE_IDEMPOTENCE_DOC)
.define(TRANSACTION_TIMEOUT_CONFIG,
Type.INT,
60000,
Importance.LOW,
TRANSACTION_TIMEOUT_DOC)
.define(TRANSACTIONAL_ID_CONFIG,
Type.STRING,
null,
new ConfigDef.NonEmptyString(),
Importance.LOW,
TRANSACTIONAL_ID_DOC)
.defineInternal(AUTO_DOWNGRADE_TXN_COMMIT,
Type.BOOLEAN,
false,
Importance.LOW);
}
It seems that CONFIG initialization is very complex, but its initialization process is to continuously define the default values, types, etc. of various configurations, in preparation for user settings of configuration properties, such as type conversion and default settings. Continue to call its overloaded constructor AbstractConfig (ConfigDef definition, Map<?,?> originals) in AbstractConfig (ConfigDef definition, Map<?,?> originals, Map<String,?> configProviderProps, Boolean doLog), which has the following source code
public AbstractConfig(ConfigDef definition, Map<?, ?> originals, Map<String, ?> configProviderProps, boolean doLog) {
/* check that all the keys are really strings */
for (Map.Entry<?, ?> entry : originals.entrySet())
if (!(entry.getKey() instanceof String))
throw new ConfigException(entry.getKey().toString(), entry.getValue(), "Key must be a string.");
this.originals = resolveConfigVariables(configProviderProps, (Map<String, Object>) originals);
this.values = definition.parse(this.originals);
this.used = Collections.synchronizedSet(new HashSet<>());
Map<String, Object> configUpdates = postProcessParsedConfig(Collections.unmodifiableMap(this.values));
for (Map.Entry<String, Object> update : configUpdates.entrySet()) {
this.values.put(update.getKey(), update.getValue());
}
definition.parse(this.values);
this.definition = definition;
if (doLog)
logAll();
}
It mainly does the following:
-
Determines whether the user-set parameter key is of string type or throws a ConfigException exception if it is not
-
Call resolveConfigVariables method to complete ConfigProvider function support, implement calling provider set by user to get configuration and update user configuration parameter value, assign originals property
-
Then parse the configuration parameter originals after the last step, call the parse method of ConfigDef to pass in originals, define the legal parameters and their types, etc. Let's take a look at the parse method
public Map<String, Object> parse(Map<?, ?> props) { // Check all configurations are defined List<String> undefinedConfigKeys = undefinedDependentConfigs(); if (!undefinedConfigKeys.isEmpty()) { String joined = Utils.join(undefinedConfigKeys, ","); throw new ConfigException("Some configurations in are referred in the dependents, but not defined: " + joined); } // parse all known keys Map<String, Object> values = new HashMap<>(); for (ConfigKey key : configKeys.values()) values.put(key.name, parseValue(key, props.get(key.name), props.containsKey(key.name))); return values; }-
Getting an undefined configuration parameter is simply to see that the key in configKeys does not contain the value of the dependents property of its corresponding value. ConfigKeys are initialized in a static block of code
-
Throw an exception if such data exists
-
Loop the value of configKeys, get the value of the corresponding parameter from props, and put it into values. We can see that values are a Map structure, where the key is the name in the ConfigKey object, which is the supported parameter. Value is obtained by the parseValue method. The first parameter is the ConfigKey object key, the second parameter is the value of the supported parameter obtained from the props user parameter, and the third parameter is true if the parameter is configured in the user parameter. The logic to execute in the parseValue method is to call the parseType method if the user sets the parameter and parse the configuration parameter value to convert it to the specified type. If the user does not set this parameter, it determines if there is a default value, gets the default value if there is one, throws an exception if there is no default value, and invokes the check method check if the parameter is required for Kafka
-
-
Initialize a thread-safe Set to assign to the user property
-
The postProcessParsedConfig method, which handles the parameter defaults. For example, the user did not set reconnect. Backoff. The max.ms parameter value, but reconnect is set. Backoff. MS parameter value, reconnect will be the default. Backoff. The max.ms parameter value is set to reconnect.backoff.ms parameter value; If the user has set transactional.id, but enable is not set. Idempotence parameter value, enable is set by default. Idempotence is true
-
Loop the default settings from the previous step into the configuration parameter Map
-
Call the parse method again
-
Assign processed definitions to the attribute definition of the current object
-
Print parameters to console
Summary: The above steps complete the creation of ProducerConfig
The resolveConfigVariables method source code is as follows
private Map<String, ?> resolveConfigVariables(Map<String, ?> configProviderProps, Map<String, Object> originals) {
Map<String, String> providerConfigString;
Map<String, ?> configProperties;
Map<String, Object> resolvedOriginals = new HashMap<>();
Map<String, String> indirectVariables = extractPotentialVariables(originals);
resolvedOriginals.putAll(originals);
if (configProviderProps == null || configProviderProps.isEmpty()) {
providerConfigString = indirectVariables;
configProperties = originals;
} else {
providerConfigString = extractPotentialVariables(configProviderProps);
configProperties = configProviderProps;
}
Map<String, ConfigProvider> providers = instantiateConfigProviders(providerConfigString, configProperties);
if (!providers.isEmpty()) {
ConfigTransformer configTransformer = new ConfigTransformer(providers);
ConfigTransformerResult result = configTransformer.transform(indirectVariables);
if (!result.data().isEmpty()) {
resolvedOriginals.putAll(result.data());
}
}
providers.values().forEach(x -> Utils.closeQuietly(x, "config provider"));
return new ResolvingMap<>(resolvedOriginals, originals);
}
Do the following
-
Resolve parameters with String type in user-set configuration parameters by extractPotentialVariables method and assign them to indirectVariables
-
resolvedOriginals local variable copy user parameters
-
CongProviderProps is empty, executes assigning indirectVariables to providerConfigString local variable, and assigns user parameters to configProperties
-
Invoke instantiateConfigProviders to instantiate the configuration provider, which is mainly used to obtain keys, such as configuration but key information from environment variables. Docker and K8s usually specify key information through environment variables
-
Using the previous attempt to get the provider, determine if the provider is not empty, first initialize a ConfigTransformer, then call ConfigTransformer's transform ation method, get the configuration parameters through the provider, and replace the variable part
-
Update resolvedOriginals with the replaced parameters
-
Circular shutdown provider
-
Using resolvedOriginals, originals to build the Return Object ResolvingMap
The instantiateConfigProviders source code is as follows
private Map<String, ConfigProvider> instantiateConfigProviders(Map<String, String> indirectConfigs, Map<String, ?> providerConfigProperties) {
final String configProviders = indirectConfigs.get(CONFIG_PROVIDERS_CONFIG);
if (configProviders == null || configProviders.isEmpty()) {
return Collections.emptyMap();
}
Map<String, String> providerMap = new HashMap<>();
for (String provider: configProviders.split(",")) {
String providerClass = CONFIG_PROVIDERS_CONFIG + "." + provider + ".class";
if (indirectConfigs.containsKey(providerClass))
providerMap.put(provider, indirectConfigs.get(providerClass));
}
// Instantiate Config Providers
Map<String, ConfigProvider> configProviderInstances = new HashMap<>();
for (Map.Entry<String, String> entry : providerMap.entrySet()) {
try {
String prefix = CONFIG_PROVIDERS_CONFIG + "." + entry.getKey() + CONFIG_PROVIDERS_PARAM;
Map<String, ?> configProperties = configProviderProperties(prefix, providerConfigProperties);
ConfigProvider provider = Utils.newInstance(entry.getValue(), ConfigProvider.class);
provider.configure(configProperties);
configProviderInstances.put(entry.getKey(), provider);
} catch (ClassNotFoundException e) {
log.error("ClassNotFoundException exception occurred: " + entry.getValue());
throw new ConfigException("Invalid config:" + entry.getValue() + " ClassNotFoundException exception occurred", e);
}
}
return configProviderInstances;
}
Do the following
-
Gets the config set by the user from the parameter passed in with a value of type string. Providers parameter value
-
Returns an empty collection if this parameter is not set
-
If this parameter is set, then the parameter values are separated by commas, each provider name is looped, and providerClass format config is constructed. Providers. < Provider name >. Class and determine if the indirectConfigs parameter contains a parameter whose key is providerClass. If it does, place the provider name, the corresponding true provider full path class name, in the providerMap collection
-
Loop providerMap, construct prefix, in config format. Providers. < Provider name >. Param. Calling the configProviderProperties(prefix, providerConfigProperties) method resolves the prefix key in the user configuration parameter, intercepts the latter part, and puts the intercepted key and value into the new map to assign to configProperties
-
Call Utils.newInstance(entry.getValue(), ConfigProvider.class) method to instantiate provider based on provider's full path name
-
Set provider's config configProperties
-
Put provider name and corresponding instance in map to return
ConfigTransformer's transform ation method source code is as follows
public ConfigTransformerResult transform(Map<String, String> configs) {
Map<String, Map<String, Set<String>>> keysByProvider = new HashMap<>();
Map<String, Map<String, Map<String, String>>> lookupsByProvider = new HashMap<>();
// Collect the variables from the given configs that need transformation
for (Map.Entry<String, String> config : configs.entrySet()) {
if (config.getValue() != null) {
List<ConfigVariable> vars = getVars(config.getValue(), DEFAULT_PATTERN);
for (ConfigVariable var : vars) {
Map<String, Set<String>> keysByPath = keysByProvider.computeIfAbsent(var.providerName, k -> new HashMap<>());
Set<String> keys = keysByPath.computeIfAbsent(var.path, k -> new HashSet<>());
keys.add(var.variable);
}
}
}
// Retrieve requested variables from the ConfigProviders
Map<String, Long> ttls = new HashMap<>();
for (Map.Entry<String, Map<String, Set<String>>> entry : keysByProvider.entrySet()) {
String providerName = entry.getKey();
ConfigProvider provider = configProviders.get(providerName);
Map<String, Set<String>> keysByPath = entry.getValue();
if (provider != null && keysByPath != null) {
for (Map.Entry<String, Set<String>> pathWithKeys : keysByPath.entrySet()) {
String path = pathWithKeys.getKey();
Set<String> keys = new HashSet<>(pathWithKeys.getValue());
ConfigData configData = provider.get(path, keys);
Map<String, String> data = configData.data();
Long ttl = configData.ttl();
if (ttl != null && ttl >= 0) {
ttls.put(path, ttl);
}
Map<String, Map<String, String>> keyValuesByPath =
lookupsByProvider.computeIfAbsent(providerName, k -> new HashMap<>());
keyValuesByPath.put(path, data);
}
}
}
// Perform the transformations by performing variable replacements
Map<String, String> data = new HashMap<>(configs);
for (Map.Entry<String, String> config : configs.entrySet()) {
data.put(config.getKey(), replace(lookupsByProvider, config.getValue(), DEFAULT_PATTERN));
}
return new ConfigTransformerResult(data, ttls);
}
It does the following
-
Loop in the user configuration Map config, which is a configuration parameter of type Value String. If the value of the configuration parameter is not empty, try to get a matching value with the matching pattern ${([^}]*?): (([^}]*?):)? ([^}]*?)\} That is to match the value in the format ${mycustom:/path/pass/to/get/method:password} and parse out the first part mycustom as providerName; The second part/path/pass/to/get/method is the path, and the path may also be empty. The third part password is used as a variable to construct ConfigVariable
-
Loop through the ConfigVariable set that you parsed up to parse into the keysByProvider Map structure, which is a set of Map <String, Map <String, Set>, key providerName for the outer Map, key path for the inner Map, and variable values for the inner Map
-
Loop through the keysByProvider parsed in the previous step, get the key of the outermost Map, which is the provider name, and get the provider instance corresponding to the provider name from the configProviders. Gets the value value of the outer Map, that is, the path and the key pair of the variables collection assign keysByPath. If neither is empty, loops keysByPath further, gets the path, and assigns keys to the corresponding variables collection, calls the get method of the provider instance to pass in path, keys, and gets the parsed configData object. Provider instances refer to instantiated objects of classes that inherit the ConfigProvider interface, and we can implement the ConfigProvider interface to achieve our own way of getting configuration parameters, such as through environment variables. Here I will explain the implementation of the get(String path, Set keys) method with the official sample FileConfigProvider
public ConfigData get(String path, Set<String> keys) { Map<String, String> data = new HashMap<>(); if (path == null || path.isEmpty()) { return new ConfigData(data); } try (Reader reader = reader(path)) { Properties properties = new Properties(); properties.load(reader); for (String key : keys) { String value = properties.getProperty(key); if (value != null) { data.put(key, value); } } return new ConfigData(data); } catch (IOException e) { throw new ConfigException("Could not read properties from file " + path); } }-
First, determine if the path is empty, and if it is empty, return the ConfigData instance directly with an empty Map for its data attribute. Because path is not required, if we want to get the configuration from the environment variable, matching paths does not have any effect, but here FileConfigProvider loads the configuration from the path path path, so it does not return empty directly
-
After the path is not empty, logic continues, Reader is constructed through path file path, reader content is loaded directly through properties, then keys are looped, keys are obtained from properties and the corresponding configuration values are put into data, data is a Map structure, key is a loop key, and value is the corresponding configuration value for the configured key. ConfigData is then constructed from the acquired data and returned
-
-
Gets the data property and ttl property of configData, and if ttl is not empty and greater than 0, associates ttl with path and stores it in a Map structure, where ttl represents the lifetime of the data in seconds. Initialize lookupsByProvider, associate provider name with path and corresponding parameter list into Map structure of lookupsByProvider
-
Next replace the regular expression \$\{([^}]*?) with the data parsed above: (([^}]*?):)? ([^}]*?)\} Matched parts. Replace with replace (Map<String, Map<String, Map<String, String>> lookupsByProvider, String value, Pattern pattern)
private static String replace(Map<String, Map<String, Map<String, String>>> lookupsByProvider, String value, Pattern pattern) { if (value == null) { return null; } Matcher matcher = pattern.matcher(value); StringBuilder builder = new StringBuilder(); int i = 0; while (matcher.find()) { ConfigVariable configVar = new ConfigVariable(matcher); Map<String, Map<String, String>> lookupsByPath = lookupsByProvider.get(configVar.providerName); if (lookupsByPath != null) { Map<String, String> keyValues = lookupsByPath.get(configVar.path); String replacement = keyValues.get(configVar.variable); builder.append(value, i, matcher.start()); if (replacement == null) { // No replacements will be performed; just return the original value builder.append(matcher.group(0)); } else { builder.append(replacement); } i = matcher.end(); } } builder.append(value, i, value.length()); return builder.toString(); }- First parse out the part that satisfies the regular expression test, then replace the part that matches the regular expression according to the configuration parameters parsed by the previous provider, and then get the replaced string, overwriting the old value previously included
-
Use the replaced user configuration parameters and the ttls parsed above to construct the return object ConfigTransformerResult
Complete Producer Creation
Formally enter the process of creating producers after the creation of ProducerConfig is completed in the above steps
KafkaProducer(ProducerConfig config,
Serializer<K> keySerializer,
Serializer<V> valueSerializer,
ProducerMetadata metadata,
KafkaClient kafkaClient,
ProducerInterceptors<K, V> interceptors,
Time time) {
try {
this.producerConfig = config;
this.time = time;
String transactionalId = config.getString(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
this.clientId = config.getString(ProducerConfig.CLIENT_ID_CONFIG);
LogContext logContext;
if (transactionalId == null)
logContext = new LogContext(String.format("[Producer clientId=%s] ", clientId));
else
logContext = new LogContext(String.format("[Producer clientId=%s, transactionalId=%s] ", clientId, transactionalId));
log = logContext.logger(KafkaProducer.class);
log.trace("Starting the Kafka producer");
Map<String, String> metricTags = Collections.singletonMap("client-id", clientId);
MetricConfig metricConfig = new MetricConfig().samples(config.getInt(ProducerConfig.METRICS_NUM_SAMPLES_CONFIG))
.timeWindow(config.getLong(ProducerConfig.METRICS_SAMPLE_WINDOW_MS_CONFIG), TimeUnit.MILLISECONDS)
.recordLevel(Sensor.RecordingLevel.forName(config.getString(ProducerConfig.METRICS_RECORDING_LEVEL_CONFIG)))
.tags(metricTags);
List<MetricsReporter> reporters = config.getConfiguredInstances(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG,
MetricsReporter.class,
Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
JmxReporter jmxReporter = new JmxReporter();
jmxReporter.configure(config.originals(Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId)));
reporters.add(jmxReporter);
MetricsContext metricsContext = new KafkaMetricsContext(JMX_PREFIX,
config.originalsWithPrefix(CommonClientConfigs.METRICS_CONTEXT_PREFIX));
this.metrics = new Metrics(metricConfig, reporters, time, metricsContext);
this.partitioner = config.getConfiguredInstance(
ProducerConfig.PARTITIONER_CLASS_CONFIG,
Partitioner.class,
Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
long retryBackoffMs = config.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG);
if (keySerializer == null) {
this.keySerializer = config.getConfiguredInstance(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
Serializer.class);
this.keySerializer.configure(config.originals(Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId)), true);
} else {
config.ignore(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
this.keySerializer = keySerializer;
}
if (valueSerializer == null) {
this.valueSerializer = config.getConfiguredInstance(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
Serializer.class);
this.valueSerializer.configure(config.originals(Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId)), false);
} else {
config.ignore(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
this.valueSerializer = valueSerializer;
}
List<ProducerInterceptor<K, V>> interceptorList = (List) config.getConfiguredInstances(
ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
ProducerInterceptor.class,
Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
if (interceptors != null)
this.interceptors = interceptors;
else
this.interceptors = new ProducerInterceptors<>(interceptorList);
ClusterResourceListeners clusterResourceListeners = configureClusterResourceListeners(keySerializer,
valueSerializer, interceptorList, reporters);
this.maxRequestSize = config.getInt(ProducerConfig.MAX_REQUEST_SIZE_CONFIG);
this.totalMemorySize = config.getLong(ProducerConfig.BUFFER_MEMORY_CONFIG);
this.compressionType = CompressionType.forName(config.getString(ProducerConfig.COMPRESSION_TYPE_CONFIG));
this.maxBlockTimeMs = config.getLong(ProducerConfig.MAX_BLOCK_MS_CONFIG);
int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
this.apiVersions = new ApiVersions();
this.transactionManager = configureTransactionState(config, logContext);
this.accumulator = new RecordAccumulator(logContext,
config.getInt(ProducerConfig.BATCH_SIZE_CONFIG),
this.compressionType,
lingerMs(config),
retryBackoffMs,
deliveryTimeoutMs,
metrics,
PRODUCER_METRIC_GROUP_NAME,
time,
apiVersions,
transactionManager,
new BufferPool(this.totalMemorySize, config.getInt(ProducerConfig.BATCH_SIZE_CONFIG), metrics, time, PRODUCER_METRIC_GROUP_NAME));
List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(
config.getList(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG),
config.getString(ProducerConfig.CLIENT_DNS_LOOKUP_CONFIG));
if (metadata != null) {
this.metadata = metadata;
} else {
this.metadata = new ProducerMetadata(retryBackoffMs,
config.getLong(ProducerConfig.METADATA_MAX_AGE_CONFIG),
config.getLong(ProducerConfig.METADATA_MAX_IDLE_CONFIG),
logContext,
clusterResourceListeners,
Time.SYSTEM);
this.metadata.bootstrap(addresses);
}
this.errors = this.metrics.sensor("errors");
this.sender = newSender(logContext, kafkaClient, this.metadata);
String ioThreadName = NETWORK_THREAD_PREFIX + " | " + clientId;
this.ioThread = new KafkaThread(ioThreadName, this.sender, true);
this.ioThread.start();
config.logUnused();
AppInfoParser.registerAppInfo(JMX_PREFIX, clientId, metrics, time.milliseconds());
log.debug("Kafka producer started");
} catch (Throwable t) {
// call close methods if internal objects are already constructed this is to prevent resource leak. see KAFKA-2121
close(Duration.ofMillis(0), true);
// now propagate the exception
throw new KafkaException("Failed to construct kafka producer", t);
}
}
-
Set the property value for the created KafkaProducer producer, set the incoming config to the producerConfig property, set the current system time to the time property, and set the client.id parameter value set to clientId property
-
Create MetricConfig, set corresponding properties, including setting number of samples, window time, record level, tag, where tag is the client id, create JmxReporter, MetricsContext, use previously created object related to indicator monitoring to create Metrics object and assign it to metrics property
-
Get partitioner and initialize to partitioner property
-
If the keySerializer parameter is empty, the configured key is obtained from the parameter. Serializer and initialize to the keySerializer property. Then call the configure method of keySerializer to complete encoding resolution settings; If not null, assign the parameter keySerializer directly to the property keySerializer property
-
Set the value of the valueSerializer property as in the previous step
-
If the incoming interceptors parameter is not empty, the interceptors property is initialized using the incoming interceptors parameter. If empty, get the configured interceptor from the config parameter and use it to create the ProducerInterceptors producer interceptor
-
The configureClusterResourceListeners method, which constructs the cluster resource listener object, is simply to construct the ClusterResourceListeners object and set its property value. The user can implement the ClusterResourceListener interface, which will call back the user-implemented method each time the cluster metadata is updated
-
Setting various parameter properties
-
The configureTransactionState method completes the creation of the transaction manager
private TransactionManager configureTransactionState(ProducerConfig config, LogContext logContext) { TransactionManager transactionManager = null; final boolean userConfiguredIdempotence = config.originals().containsKey(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG); final boolean userConfiguredTransactions = config.originals().containsKey(ProducerConfig.TRANSACTIONAL_ID_CONFIG); if (userConfiguredTransactions && !userConfiguredIdempotence) log.info("Overriding the default {} to true since {} is specified.", ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, ProducerConfig.TRANSACTIONAL_ID_CONFIG); if (config.idempotenceEnabled()) { final String transactionalId = config.getString(ProducerConfig.TRANSACTIONAL_ID_CONFIG); final int transactionTimeoutMs = config.getInt(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG); final long retryBackoffMs = config.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG); final boolean autoDowngradeTxnCommit = config.getBoolean(ProducerConfig.AUTO_DOWNGRADE_TXN_COMMIT); transactionManager = new TransactionManager( logContext, transactionalId, transactionTimeoutMs, retryBackoffMs, apiVersions, autoDowngradeTxnCommit); if (transactionManager.isTransactional()) log.info("Instantiated a transactional producer."); else log.info("Instantiated an idempotent producer."); } return transactionManager; }-
Gets from the parameter whether to turn on the secret equality setting, whether to set the Id of the thing, and prints a prompt if the Id of the thing is set but the idempotency setting is not turned on
-
If the idempotency setting is turned on and the Id of the thing is set, TransactionManager creation occurs, and the creation of TransactionManager takes configuration values from the configuration for initialization
-
-
Create Record Accumulator
-
ClientUtils. The parseAndValidateAddresses static method parses and validates the address, which requires the bootstrap of the settings to be passed in. Server server server server address, and client.dns.lookup parameter value, which defaults to use_all_dns_ips
public static List<InetSocketAddress> parseAndValidateAddresses(List<String> urls, ClientDnsLookup clientDnsLookup) { List<InetSocketAddress> addresses = new ArrayList<>(); for (String url : urls) { if (url != null && !url.isEmpty()) { try { String host = getHost(url); Integer port = getPort(url); if (host == null || port == null) throw new ConfigException("Invalid url in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG + ": " + url); if (clientDnsLookup == ClientDnsLookup.RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY) { InetAddress[] inetAddresses = InetAddress.getAllByName(host); for (InetAddress inetAddress : inetAddresses) { String resolvedCanonicalName = inetAddress.getCanonicalHostName(); InetSocketAddress address = new InetSocketAddress(resolvedCanonicalName, port); if (address.isUnresolved()) { log.warn("Couldn't resolve server {} from {} as DNS resolution of the canonical hostname {} failed for {}", url, CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, resolvedCanonicalName, host); } else { addresses.add(address); } } } else { InetSocketAddress address = new InetSocketAddress(host, port); if (address.isUnresolved()) { log.warn("Couldn't resolve server {} from {} as DNS resolution failed for {}", url, CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, host); } else { addresses.add(address); } } } catch (IllegalArgumentException e) { throw new ConfigException("Invalid port in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG + ": " + url); } catch (UnknownHostException e) { throw new ConfigException("Unknown host in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG + ": " + url); } } } if (addresses.isEmpty()) throw new ConfigException("No resolvable bootstrap urls given in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG); return addresses; }-
Loop parameter bootstrap. List of service addresses for servers, resolving host s and ports for each address
-
View client. Dns. Set the value of the lookup parameter if it is resolve_canonical_bootstrap_servers_only, get the corresponding IP list inetAddresses based on the host from the previous step, and only verify if the host itself is an IP address. Loop inetAddresses, get the host name of the specification, and use the corresponding port to create the InetSocketAddress object; If not resolve_canonical_bootstrap_servers_onl, the InetSocketAddress is created directly using host and port. Returns the parsed InetSocketAddress list object
-
-
Create production-side metadata objects and initialize them with user-set configurations of related metadata
-
newSender method, create transmitter
Sender newSender(LogContext logContext, KafkaClient kafkaClient, ProducerMetadata metadata) { int maxInflightRequests = configureInflightRequests(producerConfig); int requestTimeoutMs = producerConfig.getInt(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG); ChannelBuilder channelBuilder = ClientUtils.createChannelBuilder(producerConfig, time, logContext); ProducerMetrics metricsRegistry = new ProducerMetrics(this.metrics); Sensor throttleTimeSensor = Sender.throttleTimeSensor(metricsRegistry.senderMetrics); KafkaClient client = kafkaClient != null ? kafkaClient : new NetworkClient( new Selector(producerConfig.getLong(ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG), this.metrics, time, "producer", channelBuilder, logContext), metadata, clientId, maxInflightRequests, producerConfig.getLong(ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG), producerConfig.getLong(ProducerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG), producerConfig.getInt(ProducerConfig.SEND_BUFFER_CONFIG), producerConfig.getInt(ProducerConfig.RECEIVE_BUFFER_CONFIG), requestTimeoutMs, producerConfig.getLong(ProducerConfig.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG), producerConfig.getLong(ProducerConfig.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_CONFIG), ClientDnsLookup.forConfig(producerConfig.getString(ProducerConfig.CLIENT_DNS_LOOKUP_CONFIG)), time, true, apiVersions, throttleTimeSensor, logContext); short acks = configureAcks(producerConfig, log); return new Sender(logContext, client, metadata, this.accumulator, maxInflightRequests == 1, producerConfig.getInt(ProducerConfig.MAX_REQUEST_SIZE_CONFIG), acks, producerConfig.getInt(ProducerConfig.RETRIES_CONFIG), metricsRegistry.senderMetrics, time, requestTimeoutMs, producerConfig.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG), this.transactionManager, apiVersions); }-
configureInflightRequests method, get the max.In of the configuration. Flight. Requests. Per. Connection parameter value, which is not allowed to be greater than 5 if idempotency is turned on. This parameter value controls the maximum number of undetermined requests allowed to be sent on a single connection before blocking. Setting greater than 1 risks disorder
-
Get Request Timeout
-
ClientUtils.createChannelBuilder static method that completes channel constructor creation based on user-configured parameters
-
Create Producer Indicator Parameters
-
Create a produceThrottleTime sensor
-
Create a NetworkClient that contains various connection-related parameters such as buffer size, backoff time, connection timeout, and so on
-
configureAcks method, get and verify acks settings, for enabled things you must set acks to -1
-
Create Sender object, put various parsing settings into Sender property, Sender class implements Runnable interface
-
-
Creates a KafkaThread io thread, which inherits Thread and initializes the thread using the sender object
public KafkaThread(final String name, Runnable runnable, boolean daemon) { super(runnable, name); configureThread(name, daemon); } private void configureThread(final String name, boolean daemon) { setDaemon(daemon); setUncaughtExceptionHandler((t, e) -> log.error("Uncaught exception in thread '{}':", name, e)); }-
Its constructor calls the parent Thread's constructor and uses sender to create threads
-
Set this thread as a daemon thread and set handling of uncaught exceptions
-
-
Start io thread
-
AppInfoParser.registerAppInfo method, logging related information
-
The KafkaProducer producer has been created since then
ClientUtils.createChannelBuilder Create Channel Constructor Resolution
This static method creates a new channel constructor based on the provided configuration information
public static ChannelBuilder createChannelBuilder(AbstractConfig config, Time time, LogContext logContext) {
SecurityProtocol securityProtocol = SecurityProtocol.forName(config.getString(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG));
String clientSaslMechanism = config.getString(SaslConfigs.SASL_MECHANISM);
return ChannelBuilders.clientChannelBuilder(securityProtocol, JaasContext.Type.CLIENT, config, null,
clientSaslMechanism, time, true, logContext);
}
-
First get the security protocol from the configuration parameters, default PLAINTEXT
-
Gets the SASL mechanism for client connections, defaulting to GSSAPI
-
Call the static method clientChannelBuilder of ChannelBuilders to create a channel constructor
public static ChannelBuilder clientChannelBuilder( SecurityProtocol securityProtocol, JaasContext.Type contextType, AbstractConfig config, ListenerName listenerName, String clientSaslMechanism, Time time, boolean saslHandshakeRequestEnable, LogContext logContext) { if (securityProtocol == SecurityProtocol.SASL_PLAINTEXT || securityProtocol == SecurityProtocol.SASL_SSL) { if (contextType == null) throw new IllegalArgumentException("`contextType` must be non-null if `securityProtocol` is `" + securityProtocol + "`"); if (clientSaslMechanism == null) throw new IllegalArgumentException("`clientSaslMechanism` must be non-null in client mode if `securityProtocol` is `" + securityProtocol + "`"); } return create(securityProtocol, Mode.CLIENT, contextType, config, listenerName, false, clientSaslMechanism, saslHandshakeRequestEnable, null, null, time, logContext, null); }-
First check the parameter values if the security protocol is SASL_PLAINTEXT or SASL_SSL, then the contextType, clientSaslMechanism parameter must not be empty
-
Call the create method to complete the creation
-
create Create ChannelBuilder
Complete ChannelBuilder creation
private static ChannelBuilder create(SecurityProtocol securityProtocol,
Mode mode,
JaasContext.Type contextType,
AbstractConfig config,
ListenerName listenerName,
boolean isInterBrokerListener,
String clientSaslMechanism,
boolean saslHandshakeRequestEnable,
CredentialCache credentialCache,
DelegationTokenCache tokenCache,
Time time,
LogContext logContext,
Supplier<ApiVersionsResponse> apiVersionSupplier) {
Map<String, Object> configs = channelBuilderConfigs(config, listenerName);
ChannelBuilder channelBuilder;
switch (securityProtocol) {
case SSL:
requireNonNullMode(mode, securityProtocol);
channelBuilder = new SslChannelBuilder(mode, listenerName, isInterBrokerListener, logContext);
break;
case SASL_SSL:
case SASL_PLAINTEXT:
requireNonNullMode(mode, securityProtocol);
Map<String, JaasContext> jaasContexts;
String sslClientAuthOverride = null;
if (mode == Mode.SERVER) {
@SuppressWarnings("unchecked")
List<String> enabledMechanisms = (List<String>) configs.get(BrokerSecurityConfigs.SASL_ENABLED_MECHANISMS_CONFIG);
jaasContexts = new HashMap<>(enabledMechanisms.size());
for (String mechanism : enabledMechanisms)
jaasContexts.put(mechanism, JaasContext.loadServerContext(listenerName, mechanism, configs));
// SSL client authentication is enabled in brokers for SASL_SSL only if listener-prefixed config is specified.
if (listenerName != null && securityProtocol == SecurityProtocol.SASL_SSL) {
String configuredClientAuth = (String) configs.get(BrokerSecurityConfigs.SSL_CLIENT_AUTH_CONFIG);
String listenerClientAuth = (String) config.originalsWithPrefix(listenerName.configPrefix(), true)
.get(BrokerSecurityConfigs.SSL_CLIENT_AUTH_CONFIG);
// If `ssl.client.auth` is configured at the listener-level, we don't set an override and SslFactory
// uses the value from `configs`. If not, we propagate `sslClientAuthOverride=NONE` to SslFactory and
// it applies the override to the latest configs when it is configured or reconfigured. `Note that
// ssl.client.auth` cannot be dynamically altered.
if (listenerClientAuth == null) {
sslClientAuthOverride = SslClientAuth.NONE.name().toLowerCase(Locale.ROOT);
if (configuredClientAuth != null && !configuredClientAuth.equalsIgnoreCase(SslClientAuth.NONE.name())) {
log.warn("Broker configuration '{}' is applied only to SSL listeners. Listener-prefixed configuration can be used" +
" to enable SSL client authentication for SASL_SSL listeners. In future releases, broker-wide option without" +
" listener prefix may be applied to SASL_SSL listeners as well. All configuration options intended for specific" +
" listeners should be listener-prefixed.", BrokerSecurityConfigs.SSL_CLIENT_AUTH_CONFIG);
}
}
}
} else {
// Use server context for inter-broker client connections and client context for other clients
JaasContext jaasContext = contextType == JaasContext.Type.CLIENT ? JaasContext.loadClientContext(configs) :
JaasContext.loadServerContext(listenerName, clientSaslMechanism, configs);
jaasContexts = Collections.singletonMap(clientSaslMechanism, jaasContext);
}
channelBuilder = new SaslChannelBuilder(mode,
jaasContexts,
securityProtocol,
listenerName,
isInterBrokerListener,
clientSaslMechanism,
saslHandshakeRequestEnable,
credentialCache,
tokenCache,
sslClientAuthOverride,
time,
logContext,
apiVersionSupplier);
break;
case PLAINTEXT:
channelBuilder = new PlaintextChannelBuilder(listenerName);
break;
default:
throw new IllegalArgumentException("Unexpected securityProtocol " + securityProtocol);
}
channelBuilder.configure(configs);
return channelBuilder;
}
-
channelBuilderConfigs method, mainly completes parameter processing
-
Create a channel builder for the corresponding security policy based on the security policy. For example, a PlaintextChannelBuilder channel constructor is created by default
-
Set the parameters returned by the first step to the channel builder
send message
After the sender is created, the sender can be used to send messages, but what is sent also needs to be constructed into what the sender needs
Construct the object type ProducerRecord needed to send the message
Before sending a message, we need to construct the message to be sent into a ProducerRecord object, which can be created directly by the ProducerRecord construction method. In addition to setting the Key and Value of the message to be sent, we need to set the topic to be sent, we can also set the partition to be sent, and so on.
Complete message sending via producer send method
The send method of KafkaProducer is used to complete the sending of messages, which are asynchronous and return Future objects. Sending messages may or may not specify an error callback method. Let's take a look at the message sending source code that does not specify an error callback method
Send message method with no error callback specified
public Future<RecordMetadata> send(ProducerRecord<K, V> record) {
return send(record, null);
}
public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
// intercept the record, which can be potentially modified; this method does not throw exceptions
ProducerRecord<K, V> interceptedRecord = this.interceptors.onSend(record);
return doSend(interceptedRecord, callback);
}
-
Call the onSend method of ProducerInterceptors, loop through all interceptors, and call the onSend method of the user registered interceptor, which is executed before serialization. And the output of the previous interceptor is the input of the next interceptor
-
Perform doSend to actually complete the message sending operation
The doSend method completes the real send
private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
TopicPartition tp = null;
try {
throwIfProducerClosed();
// first make sure the metadata for the topic is available
long nowMs = time.milliseconds();
ClusterAndWaitTime clusterAndWaitTime;
try {
clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), nowMs, maxBlockTimeMs);
} catch (KafkaException e) {
if (metadata.isClosed())
throw new KafkaException("Producer closed while send in progress", e);
throw e;
}
nowMs += clusterAndWaitTime.waitedOnMetadataMs;
long remainingWaitMs = Math.max(0, maxBlockTimeMs - clusterAndWaitTime.waitedOnMetadataMs);
Cluster cluster = clusterAndWaitTime.cluster;
byte[] serializedKey;
try {
serializedKey = keySerializer.serialize(record.topic(), record.headers(), record.key());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG).getName() +
" specified in key.serializer", cce);
}
byte[] serializedValue;
try {
serializedValue = valueSerializer.serialize(record.topic(), record.headers(), record.value());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG).getName() +
" specified in value.serializer", cce);
}
int partition = partition(record, serializedKey, serializedValue, cluster);
tp = new TopicPartition(record.topic(), partition);
setReadOnly(record.headers());
Header[] headers = record.headers().toArray();
int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(apiVersions.maxUsableProduceMagic(),
compressionType, serializedKey, serializedValue, headers);
ensureValidRecordSize(serializedSize);
long timestamp = record.timestamp() == null ? nowMs : record.timestamp();
if (log.isTraceEnabled()) {
log.trace("Attempting to append record {} with callback {} to topic {} partition {}", record, callback, record.topic(), partition);
}
// producer callback will make sure to call both 'callback' and interceptor callback
Callback interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
if (transactionManager != null && transactionManager.isTransactional()) {
transactionManager.failIfNotReadyForSend();
}
RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey,
serializedValue, headers, interceptCallback, remainingWaitMs, true, nowMs);
if (result.abortForNewBatch) {
int prevPartition = partition;
partitioner.onNewBatch(record.topic(), cluster, prevPartition);
partition = partition(record, serializedKey, serializedValue, cluster);
tp = new TopicPartition(record.topic(), partition);
if (log.isTraceEnabled()) {
log.trace("Retrying append due to new batch creation for topic {} partition {}. The old partition was {}", record.topic(), partition, prevPartition);
}
// producer callback will make sure to call both 'callback' and interceptor callback
interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
result = accumulator.append(tp, timestamp, serializedKey,
serializedValue, headers, interceptCallback, remainingWaitMs, false, nowMs);
}
if (transactionManager != null && transactionManager.isTransactional())
transactionManager.maybeAddPartitionToTransaction(tp);
if (result.batchIsFull || result.newBatchCreated) {
log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
this.sender.wakeup();
}
return result.future;
// handling exceptions and record the errors;
// for API exceptions return them in the future,
// for other exceptions throw directly
} catch (ApiException e) {
log.debug("Exception occurred during message send:", e);
if (callback != null)
callback.onCompletion(null, e);
this.errors.record();
this.interceptors.onSendError(record, tp, e);
return new FutureFailure(e);
} catch (InterruptedException e) {
this.errors.record();
this.interceptors.onSendError(record, tp, e);
throw new InterruptException(e);
} catch (KafkaException e) {
this.errors.record();
this.interceptors.onSendError(record, tp, e);
throw e;
} catch (Exception e) {
// we notify interceptor about all exceptions, since onSend is called before anything else in this method
this.interceptors.onSendError(record, tp, e);
throw e;
}
}
-
throwIfProducerClosed method that checks if the sender property is empty or closed and throws a "Cannot perform operation after producer has been closed" exception
-
waitOnMetadata method, getting cluster metadata information
-
The number of times before the waitOnMetadata method is executed + the time after the waitOnMetadata method is executed as the value of the nowMs local variable
-
Maximum blocking time - waitOnMetadata method gets metadata time as remainingWaitMs remaining wait time value
-
Call keySerializer.serialize method completes serialization of key
-
Call valueSerializer.serialize method completes value serialization
-
Invoke the partition method to calculate the target partition for sending the message
-
setReadOnly method that sets the read-only identity isReadOnly to true if the sending header is a RecordHeaders type object
-
AbstractRecords. The estimateSizeInBytesUpperBound method, which gets the maximum estimated value of the message to be sent without considering compression
-
The ensureValidRecordSize method, which determines whether the size of the sent message is greater than the online size, is based primarily on comparison with maxRequestSize, totalMemorySize
-
If the timestamp for sending data is empty, use the nowMs above as the timestamp
-
New InterceptorCallback<>Create callback
-
The append method of RecordAccumulator appends the message record to be sent to the memory cache pool where the message was sent
-
this.sender.wakeup() calls the sending thread to send the message
WatOnMetadata Gets Cluster Metadata Information
To be added
Partition method calculates the destination partition for sending messages
To be added
Appnd method of RecordAccumulator
To be added
Calls the sending thread to send messages
To be added