2021SC@SDUSC
Bolt actuator in Trident
2021SC@SDUSC
Similar to the coordination bolt in transaction Topology, Trident uses the tridentboltexecution to execute the SubTopologyBolt in Trident.
ITridentBatchBolt.java
public interface ITridentBatchBolt extends IComponent {
void prepare(Map<String, Object> conf, TopologyContext context, BatchOutputCollector collector);
void execute(BatchInfo batchInfo, Tuple tuple);
void finishBatch(BatchInfo batchInfo);
Object initBatchState(String batchGroup, Object batchId);
void cleanup();
}
Compared with the IBatchBolt interface, this interface has more initBatchState methods and cleanup methods. Trident avoids the need to deserialize a new Bolt object every time. After receiving the first message of a transaction in the node group, the initBatchState method will be called
Transaction isolation is achieved by starting the transaction and calling the finishBatch method at the end.
TrackedBatch class
Messages from multiple transactions may be processed simultaneously in the SubTopologyBolt node. The class trackdbatch is used to track the transactions being processed in the Bolt
public static class TrackedBatch {
int attemptId;
BatchInfo info;
CoordCondition condition;
int reportedTasks = 0;
int expectedTupleCount = 0;
int receivedTuples = 0;
Map<Integer, Integer> taskEmittedTuples = new HashMap<>();
boolean failed = false;
boolean receivedCommit;
Tuple delayedAck = null;
public TrackedBatch(BatchInfo info, CoordCondition condition, int attemptId) {
this.info = info;
this.condition = condition;
this.attemptId = attemptId;
receivedCommit = condition.commitStream == null;
}
}
TridentBoltExecutor
Data member and prepare method
Map<GlobalStreamId, String> batchGroupIds;
Map<String, CoordSpec> coordSpecs;
Map<String, CoordCondition> coordConditions;
ITridentBatchBolt bolt;
long messageTimeoutMs;
long lastRotate;
RotatingMap<Object, TrackedBatch> batches;
OutputCollector collector;
CoordinatedOutputCollector coordCollector;
BatchOutputCollector coordOutputCollector;
TopologyContext context;
@Override
public void prepare(Map<String, Object> conf, TopologyContext context, OutputCollector collector) {
messageTimeoutMs = context.maxTopologyMessageTimeout() * 1000L;
lastRotate = System.currentTimeMillis();
batches = new RotatingMap<>(2);
this.context = context;
this.collector = collector;
coordCollector = new CoordinatedOutputCollector(collector);
coordOutputCollector = new BatchOutputCollectorImpl(new OutputCollector(coordCollector));
coordConditions = (Map) context.getExecutorData("__coordConditions");
if (coordConditions == null) {
coordConditions = new HashMap<>();
for (String batchGroup : coordSpecs.keySet()) {
CoordSpec spec = coordSpecs.get(batchGroup);
CoordCondition cond = new CoordCondition();
cond.commitStream = spec.commitStream;
cond.expectedTaskReports = 0;
for (String comp : spec.coords.keySet()) {
CoordType ct = spec.coords.get(comp);
if (ct.equals(CoordType.single())) {
cond.expectedTaskReports += 1;
} else {
cond.expectedTaskReports += context.getComponentTasks(comp).size();
}
}
cond.targetTasks = new HashSet<>();
for (String component : Utils.get(context.getThisTargets(),
coordStream(batchGroup),
new HashMap<String, Grouping>()).keySet()) {
cond.targetTasks.addAll(context.getComponentTasks(component));
}
coordConditions.put(batchGroup, cond);
}
context.setExecutorData("coordConditions", coordConditions);
}
bolt.prepare(conf, context, coordOutputCollector);
}
batchGroupsIds:
It is used to store the mapping relationship from global flow to node group sequence number.
coordSpecs;
It is used to store the receiving relationship of coordination messages among nodes in each node group.
coordConditions:
Used to store coordination conditions within each node group.
batches:
Used to track the transactions being processed by the node.
bolt:
The Bolt node represented by tridentboltexecution is currently of type SubTopologyBolt.
messageTimeoutMs,astRotate:
Set the message timeout, which is the same as the message timeout setting of the system.
prepare method:
It is mainly used to complete the initialization of member variables. Task s in the same Executor will belong to the same group
The coordination message settings of these tasks are the same for the same component. Trident stores this setting in
In the shared data of the Executor. Since tasks in the Executor are initialized in order, other tasks in the Executor will directly obtain these settings and do not need to be recalculated.
Method analysis of key members
execute
@Override
public void execute(Tuple tuple) {
if (TupleUtils.isTick(tuple)) {
long now = System.currentTimeMillis();
if (now - lastRotate > messageTimeoutMs) {
batches.rotate();
lastRotate = now;
}
return;
}
String batchGroup = batchGroupIds.get(tuple.getSourceGlobalStreamId());
if (batchGroup == null) {
coordCollector.setCurrBatch(null);
bolt.execute(null, tuple);
collector.ack(tuple);
return;
}
IBatchID id = (IBatchID) tuple.getValue(0);
TrackedBatch tracked = (TrackedBatch) batches.get(id.getId());
if (tracked != null) {
if (id.getAttemptId() > tracked.attemptId) {
batches.remove(id.getId());
tracked = null;
} else if (id.getAttemptId() < tracked.attemptId) {
// no reason to try to execute a previous attempt than we've already seen
return;
}
}
if (tracked == null) {
tracked =
new TrackedBatch(new BatchInfo(batchGroup, id, bolt.initBatchState(batchGroup, id)), coordConditions.get(batchGroup),
id.getAttemptId());
batches.put(id.getId(), tracked);
}
coordCollector.setCurrBatch(tracked);
//System.out.println("TRACKED: " + tracked + " " + tuple);
TupleType t = getTupleType(tuple, tracked);
if (t == TupleType.COMMIT) {
tracked.receivedCommit = true;
checkFinish(tracked, tuple, t);
} else if (t == TupleType.COORD) {
int count = tuple.getInteger(1);
tracked.reportedTasks++;
tracked.expectedTupleCount += count;
checkFinish(tracked, tuple, t);
} else {
tracked.receivedTuples++;
boolean success = true;
try {
bolt.execute(tracked.info, tuple);
if (tracked.condition.expectedTaskReports == 0) {
success = finishBatch(tracked, tuple);
}
} catch (FailedException e) {
failBatch(tracked, e);
}
if (success) {
collector.ack(tuple);
} else {
collector.fail(tuple);
}
}
coordCollector.setCurrBatch(null);
}
if (TupleUtils.isTick(tuple)):
Bolt needs to track all the transactions being processed. However, due to the failure of some transactions, the tracked transactions may not be cleaned up in time, which will lead to memory leakage in the long run. This section deals with this situation, which is very similar to the message timeout technology in the sput node, that is, when a message from the Tick stream is received, the system will timeout the older data in the batch.
if (batchGroup == null):
If the source stream of a message does not belong to any node group, the message is not tracked, but directly
Call the execute method of the proxy class and Ack the input message.
if(t == TupleType.COMMIT)
If the received message type is a transaction submission message, set receivedcommit to true, which is one of the conditions for the end of transaction processing.
After receiving the control message, the checkFinish method is called to check whether the transaction has ended
private void checkFinish(TrackedBatch tracked, Tuple tuple, TupleType type) {
if (tracked.failed) {
failBatch(tracked);
collector.fail(tuple);
return;
}
CoordCondition cond = tracked.condition;
boolean delayed = tracked.delayedAck == null
&& (cond.commitStream != null && type == TupleType.COMMIT
|| cond.commitStream == null);
if (delayed) {
tracked.delayedAck = tuple;
}
boolean failed = false;
if (tracked.receivedCommit && tracked.reportedTasks == cond.expectedTaskReports) {
if (tracked.receivedTuples == tracked.expectedTupleCount) {
finishBatch(tracked, tuple);
} else {
//TODO: add logging that not all tuples were received
failBatch(tracked);
collector.fail(tuple);
failed = true;
}
}
if (!delayed && !failed) {
collector.ack(tuple);
}
}