1. General
Reprint and supplement: http://www.qishunwang.net/news_show_82511.aspx
2.SlotPool
2.1. introduce
slot pool is the pool used by JobMaster to manage slots Is an interface class that defines the management operations of related slots
The main methods are as follows
2.1.1. Life cycle related interfaces
Interface | meaning |
---|---|
start | start-up |
suspend | Hang |
close | close |
2.1.2 resource manager connection
Interface | meaning |
---|---|
connectToResourceManager | Establish a connection with ResourceManager |
disconnectResourceManager | Close ResourceManager connection |
registerTaskManager | Register a TaskExecutor with the given ResourceId |
releaseTaskManager | Release TaskExecutor |
2.1.3 relevant to slot operation
Interface | meaning |
---|---|
offerSlots | Release slot |
failAllocation | Identify the slot as failed according to the given allocation id |
getAvailableSlotsInformation | Get the currently available slots information |
getAllocatedSlotsInformation | Get all slot information |
allocateAvailableSlot | Allocate available slot s with the given allocation id under the given request id. If no slot with the given allocation id is available, this method returns {@ code null}. |
requestNewAllocatedSlot | Request the allocation of a new slot from the resource manager. This method does not return a slot from a slot that is already available in the pool, but adds a new slot to the pool, which will be allocated and returned immediately. |
requestNewAllocatedBatchSlot | Request allocation of a new batch slot from the resource manager Unlike ordinary slots, batch slots will time out only when the slot pool does not contain an appropriate slot. In addition, it does not respond to fault signals from the resource manager. |
disableBatchSlotRequestTimeoutCheck | Disable batch slot request timeout check. Called when someone else wants to take over the timeout check responsibility. |
createAllocatedSlotReport | Creates a report on assigned slot s belonging to the specified task manager. |
3.SlotPoolImpl implementation class
SlotPoolImpl is the implementation class of the SlotPool interface
The slot pool serves slot requests issued by {@ link ExecutionGraph}.
When it cannot provide a slot request, it will try to get a new slot from the resource manager.
If no ResourceManager is currently available, or if the ResourceManager rejects it, or if the request times out, it will fail the slot request.
The slot pool also saves all the slots provided to it and accepted, so registered free slots can be provided even if the resource manager is closed.
Slots are released only when they are useless, for example, when the job is fully running, but we still have some available slot s.
All allocations or slot offers will be identified by their own generated AllocationID, which we will use to disambiguate.
3.1. attribute
/** The interval (in milliseconds) in which the SlotPool writes its slot distribution on debug level. * SlotPool The time interval (in milliseconds) at the debug level to write its slot distribution. * */ private static final long STATUS_LOG_INTERVAL_MS = 60_000; private final JobID jobId; /** All registered TaskManagers, slots will be accepted and used only if the resource is registered. * All registered taskmanagers and slot s will be accepted and used only when the resource is registered. * */ private final HashSet<ResourceID> registeredTaskManagers; /** The book-keeping of all allocated slots. * //All slots assigned to the current JobManager * */ private final AllocatedSlots allocatedSlots; /** The book-keeping of all available slots. * All available slots (already assigned to the JobManager, but the payload has not been loaded) * */ private final AvailableSlots availableSlots; /** All pending requests waiting for slots. * All slot request s in the waiting state (requests have been sent to the resource manager) wait for all pending requests from slots. * */ private final DualKeyLinkedMap<SlotRequestId, AllocationID, PendingRequest> pendingRequests; /** The requests that are waiting for the resource manager to be connected. * slot request in waiting status (the request has not been sent to the ResourceManager, and no connection has been established with the ResourceManager at this time) * Wait for a request to connect to the resource manager. * */ private final LinkedHashMap<SlotRequestId, PendingRequest> waitingForResourceManager; /** Timeout for external request calls (e.g. to the ResourceManager or the TaskExecutor). * External request invocation timed out (for example, to ResourceManager or TaskExecutor). * */ private final Time rpcTimeout; /** Timeout for releasing idle slots. * Free slots timeout * */ private final Time idleSlotTimeout; /** Timeout for batch slot requests. * Batch slot request timed out * */ private final Time batchSlotTimeout; private final Clock clock; /** the fencing token of the job manager. */ private JobMasterId jobMasterId; /** The gateway to communicate with resource manager. */ private ResourceManagerGateway resourceManagerGateway; private String jobManagerAddress; // Component main thread executor private ComponentMainThreadExecutor componentMainThreadExecutor;
3.2 life cycle related interfaces
Interface | meaning |
---|---|
start | start-up |
suspend | Hang |
close | close |
3.2.1 start method
/** * Start the slot pool to accept RPC calls. * * Start the slot pool to accept RPC calls. * * @param jobMasterId The necessary leader id for running the job. * @param newJobManagerAddress for the slot requests which are sent to the resource manager * @param componentMainThreadExecutor The main thread executor for the job master's main thread. */ public void start( @Nonnull JobMasterId jobMasterId, @Nonnull String newJobManagerAddress, @Nonnull ComponentMainThreadExecutor componentMainThreadExecutor) throws Exception { this.jobMasterId = jobMasterId; this.jobManagerAddress = newJobManagerAddress; this.componentMainThreadExecutor = componentMainThreadExecutor; // Timeout related operations scheduleRunAsync(this::checkIdleSlot, idleSlotTimeout); scheduleRunAsync(this::checkBatchSlotTimeout, batchSlotTimeout); if (log.isDebugEnabled()) { scheduleRunAsync(this::scheduledLogStatus, STATUS_LOG_INTERVAL_MS, TimeUnit.MILLISECONDS); } }
3.2.2 suspend
/** * Suspends this pool, meaning it has lost its authority to accept and distribute slots. * * Suspending this pool means that it has lost its permission to accept and distribute slot s. */ @Override public void suspend() { componentMainThreadExecutor.assertRunningInMainThread(); log.info("Suspending SlotPool."); // cancel all pending allocations --> we can request these slots // again after we regained the leadership Set<AllocationID> allocationIds = pendingRequests.keySetB(); for (AllocationID allocationId : allocationIds) { // resourceManagerGateway cancels the SlotRequest operation resourceManagerGateway.cancelSlotRequest(allocationId); } // do not accept any requests jobMasterId = null; resourceManagerGateway = null; // Clear (but not release!) the available slots. The TaskManagers should re-register them // at the new leader JobManager/SlotPool clear(); }
3.2.3 close
@Override public void close() { log.info("Stopping SlotPool."); // cancel all pending allocations // Cancel pending SlotRequests Set<AllocationID> allocationIds = pendingRequests.keySetB(); for (AllocationID allocationId : allocationIds) { resourceManagerGateway.cancelSlotRequest(allocationId); } // Release resources release all registered slots by releasing the corresponding TaskExecutor // release all registered slots by releasing the corresponding TaskExecutors for (ResourceID taskManagerResourceId : registeredTaskManagers) { final FlinkException cause = new FlinkException( "Releasing TaskManager " + taskManagerResourceId + ", because of stopping of SlotPool"); releaseTaskManagerInternal(taskManagerResourceId, cause); } clear(); }
3.3 resource manager connection
Interface | meaning |
---|---|
connectToResourceManager | Establish connection with ResourceManager |
disconnectResourceManager | Close the ResourceManager connection |
registerTaskManager | Register a TaskExecutor with the given ResourceId |
releaseTaskManager | Release TaskExecutor |
3.3.1 connectToResourceManager
/** * Establish a connection with ResourceManager to handle blocked / pending requests * @param resourceManagerGateway The RPC gateway for the resource manager. */ @Override public void connectToResourceManager(@Nonnull ResourceManagerGateway resourceManagerGateway) { this.resourceManagerGateway = checkNotNull(resourceManagerGateway); // Processing pending PendingRequest requests // work on all slots waiting for this connection for (PendingRequest pendingRequest : waitingForResourceManager.values()) { // Request RM / get resource requestSlotFromResourceManager(resourceManagerGateway, pendingRequest); } // all sent off waitingForResourceManager.clear(); }
3.3.2 disconnectResourceManager
Close the ResourceManager connection
@Override public void disconnectResourceManager() { this.resourceManagerGateway = null; }
3.3.3 registerTaskManager
/** * Register TaskManager to this pool, only those slots come from registered TaskManager will be considered valid. * Also it provides a way for us to keep "dead" or "abnormal" TaskManagers out of this pool. * * @param resourceID The id of the TaskManager * * * If you register a TaskManager with this pool, only slot s from a registered TaskManager will be considered valid. * It also provides us with a way to keep "dead" or "abnormal" task managers away from the pool */ @Override public boolean registerTaskManager(final ResourceID resourceID) { componentMainThreadExecutor.assertRunningInMainThread(); log.debug("Register new TaskExecutor {}.", resourceID); return registeredTaskManagers.add(resourceID); }
3.3.4 releaseTaskManager
/** * Unregister TaskManager from this pool, all the related slots will be released and tasks be canceled. Called * when we find some TaskManager becomes "dead" or "abnormal", and we decide to not using slots from it anymore. * Unregistering the TaskManager from the pool releases all relevant slot s and cancels the task. * Called when we find that a task manager becomes "dead" or "abnormal" and we decide not to use the slot in it anymore. * * @param resourceId The id of the TaskManager * @param cause for the releasing of the TaskManager */ @Override public boolean releaseTaskManager(final ResourceID resourceId, final Exception cause) { componentMainThreadExecutor.assertRunningInMainThread(); if (registeredTaskManagers.remove(resourceId)) { releaseTaskManagerInternal(resourceId, cause); return true; } else { return false; } }
3.4 relevant to slot operation
Interface | meaning |
---|---|
offerSlots | Consumption slot |
failAllocation | Identify the slot as failed according to the given allocation id |
getAvailableSlotsInformation | Get the currently available slots information |
getAllocatedSlotsInformation | Get all slot information |
allocateAvailableSlot | Allocate available slot s with the given allocation id under the given request id. If no slot with the given allocation id is available, this method returns {@ code null}. |
requestNewAllocatedSlot | Request the allocation of a new slot from the resource manager. This method does not return a slot from a slot that is already available in the pool, but adds a new slot to the pool, which will be allocated and returned immediately. |
requestNewAllocatedBatchSlot | Request allocation of a new batch slot from the resource manager Unlike ordinary slots, batch slots will time out only when the slot pool does not contain an appropriate slot. In addition, it does not respond to fault signals from the resource manager. |
disableBatchSlotRequestTimeoutCheck | Disable batch slot request timeout check. Called when someone else wants to take over the timeout check responsibility. |
createAllocatedSlotReport | Creates a report on assigned slot s belonging to the specified task manager. |
3.4.1 offerSlots
/** * According to the allocationid, taskexecutor provides Slot * * AllocationID It is initially generated by the pool and transferred to task manager through resource manager * * We use it to distinguish the different distributions we issue. * * If we find that a Slot does not match or do not actually wait for the pending request of this Slot (which may be completed by other returned slots), the Slot offer may be rejected. * * Slot offering by TaskExecutor with AllocationID. The AllocationID is originally generated by this pool and * transfer through the ResourceManager to TaskManager. We use it to distinguish the different allocation * we issued. Slot offering may be rejected if we find something mismatching or there is actually no pending * request waiting for this slot (maybe fulfilled by some other returned slot). * * @param taskManagerLocation location from where the offer comes from * @param taskManagerGateway TaskManager gateway * @param slotOffer the offered slot * @return True if we accept the offering */ boolean offerSlot( final TaskManagerLocation taskManagerLocation, final TaskManagerGateway taskManagerGateway, final SlotOffer slotOffer) { componentMainThreadExecutor.assertRunningInMainThread(); // Check whether the TaskManager is valid // check if this TaskManager is valid final ResourceID resourceID = taskManagerLocation.getResourceID(); final AllocationID allocationID = slotOffer.getAllocationId(); // Must be a slotOffer in a registered taskmanager if (!registeredTaskManagers.contains(resourceID)) { log.debug("Received outdated slot offering [{}] from unregistered TaskManager: {}", slotOffer.getAllocationId(), taskManagerLocation); return false; } // If the AllocationID associated with the current slot already appears in the SlotPool, check whether this slot has been used // check whether we have already using this slot AllocatedSlot existingSlot; if ((existingSlot = allocatedSlots.get(allocationID)) != null || (existingSlot = availableSlots.get(allocationID)) != null) { // we need to figure out if this is a repeated offer for the exact same slot, // or another offer that comes from a different TaskManager after the ResourceManager // re-tried the request // We need to find out that this is a repeated offer for exactly the same slot, // Or another offer from a different task manager after the resource manager retries the request // We write this by comparing the slot ID, because the slot ID is the identifier of the actual slot on the task manager // we write this in terms of comparing slot IDs, because the Slot IDs are the identifiers of // the actual slots on the TaskManagers // Note: The slotOffer should have the SlotID // Get existing SlotID final SlotID existingSlotId = existingSlot.getSlotId(); // Get new SlotID final SlotID newSlotId = new SlotID(taskManagerLocation.getResourceID(), slotOffer.getSlotIndex()); //This slot has been accepted by the SlotPool before, which is equivalent to the TaskExecutor sending a duplicate offer if (existingSlotId.equals(newSlotId)) { log.info("Received repeated offer for slot [{}]. Ignoring.", allocationID); // return true here so that the sender will get a positive acknowledgement to the retry // and mark the offering as a success return true; } else { //There is already another AllocatedSlot associated with this AllocationID, so the current slot cannot be accepted // the allocation has been fulfilled by another slot, reject the offer so the task executor // will offer the slot to the resource manager return false; } } // This means that this slot has not been used yet //The allocation ID associated with this slot has not appeared before //Create an AllocatedSlot object to represent the newly allocated slot final AllocatedSlot allocatedSlot = new AllocatedSlot( allocationID, taskManagerLocation, slotOffer.getSlotIndex(), slotOffer.getResourceProfile(), taskManagerGateway); // Check whether there is a request associated with this AllocationID // check whether we have request waiting for this slot PendingRequest pendingRequest = pendingRequests.removeKeyB(allocationID); if (pendingRequest != null) { // we were waiting for this! //There is a pending request waiting for this slot allocatedSlots.add(pendingRequest.getSlotRequestId(), allocatedSlot); //Try to complete the waiting request if (!pendingRequest.getAllocatedSlotFuture().complete(allocatedSlot)) { // we could not complete the pending slot future --> try to fulfill another pending request //failed allocatedSlots.remove(pendingRequest.getSlotRequestId()); //Try to satisfy other waiting requests, and use slot to complete pending requests in the order of requests tryFulfillSlotRequestOrMakeAvailable(allocatedSlot); } else { log.debug("Fulfilled slot request [{}] with allocated slot [{}].", pendingRequest.getSlotRequestId(), allocationID); } } else { //There is no request waiting for this slot. The request may have been satisfied // we were actually not waiting for this: // - could be that this request had been fulfilled // - we are receiving the slots from TaskManagers after becoming leaders //Try to satisfy other waiting requests tryFulfillSlotRequestOrMakeAvailable(allocatedSlot); } // we accepted the request in any case. slot will be released after it idled for // too long and timed out // We accepted the request anyway // The slot will be released after too long idle time and timeout return true; }
tryFulfillSlotRequestOrMakeAvailable
/** * Tries to fulfill with the given allocated slot a pending slot request or add the * allocated slot to the set of available slots if no matching request is available. * * Attempting to complete a pending slot request with the given allocated slot, * Or if there is no matching request, return the allocated slot to the available slot set. * * @param allocatedSlot which shall be returned */ private void tryFulfillSlotRequestOrMakeAvailable(AllocatedSlot allocatedSlot) { Preconditions.checkState(!allocatedSlot.isUsed(), "Provided slot is still in use."); //Find the pending requests that match the computing resources of the current AllocatedSlot final PendingRequest pendingRequest = pollMatchingPendingRequest(allocatedSlot); if (pendingRequest != null) { //If there are matching requests, AllocatedSlot is assigned to the waiting requests log.debug("Fulfilling pending slot request [{}] early with returned slot [{}]", pendingRequest.getSlotRequestId(), allocatedSlot.getAllocationId()); // Add the currently allocated slots to the allocated allocated slots collection to identify that they have been used allocatedSlots.add(pendingRequest.getSlotRequestId(), allocatedSlot); // Callback request and return allocatedSlot information Identity slot allocation completed pendingRequest.getAllocatedSlotFuture().complete(allocatedSlot); } else { //If not, the allocated slot becomes available // There is no PendingRequest available. Return allocatedSlot log.debug("Adding returned slot [{}] to available slots", allocatedSlot.getAllocationId()); availableSlots.add(allocatedSlot, clock.relativeTimeMillis()); } }
3.4.2 failAllocation
@Override public Optional<ResourceID> failAllocation(final AllocationID allocationID, final Exception cause) { componentMainThreadExecutor.assertRunningInMainThread(); // Get PendingRequest final PendingRequest pendingRequest = pendingRequests.removeKeyB(allocationID); if (pendingRequest != null) { if (isBatchRequestAndFailureCanBeIgnored(pendingRequest, cause)) { // pending batch requests don't react to this signal --> put it back pendingRequests.put(pendingRequest.getSlotRequestId(), allocationID, pendingRequest); } else { // request was still pending failPendingRequest(pendingRequest, cause); } return Optional.empty(); } else { return tryFailingAllocatedSlot(allocationID, cause); } // TODO: add some unit tests when the previous two are ready, the allocation may failed at any phase }
tryFailingAllocatedSlot
private Optional<ResourceID> tryFailingAllocatedSlot(AllocationID allocationID, Exception cause) { // Get AllocatedSlot with allocation failure AllocatedSlot allocatedSlot = availableSlots.tryRemove(allocationID); if (allocatedSlot == null) { allocatedSlot = allocatedSlots.remove(allocationID); } if (allocatedSlot != null) { log.debug("Failed allocated slot [{}]: {}", allocationID, cause.getMessage()); // notify TaskExecutor about the failure // Notification TaskExecutor assignment failed allocatedSlot.getTaskManagerGateway().freeSlot(allocationID, cause, rpcTimeout); // release the slot. // since it is not in 'allocatedSlots' any more, it will be dropped o return' // Release the slot and discard it allocatedSlot.releasePayload(cause); final ResourceID taskManagerId = allocatedSlot.getTaskManagerId(); if (!availableSlots.containsTaskManager(taskManagerId) && !allocatedSlots.containResource(taskManagerId)) { return Optional.of(taskManagerId); } } return Optional.empty(); }
3.4.3 getAvailableSlotsInformation
Get available slot information
/** * Lists the currently available slot s * @return */ @Override @Nonnull public Collection<SlotInfoWithUtilization> getAvailableSlotsInformation() { final Map<ResourceID, Set<AllocatedSlot>> availableSlotsByTaskManager = availableSlots.getSlotsByTaskManager(); final Map<ResourceID, Set<AllocatedSlot>> allocatedSlotsSlotsByTaskManager = allocatedSlots.getSlotsByTaskManager(); return availableSlotsByTaskManager.entrySet().stream() .flatMap(entry -> { final int numberAllocatedSlots = allocatedSlotsSlotsByTaskManager.getOrDefault(entry.getKey(), Collections.emptySet()).size(); final int numberAvailableSlots = entry.getValue().size(); final double taskExecutorUtilization = (double) numberAllocatedSlots / (numberAllocatedSlots + numberAvailableSlots); return entry.getValue().stream().map(slot -> SlotInfoWithUtilization.from(slot, taskExecutorUtilization)); }) .collect(Collectors.toList()); }
3.4.4 getAllocatedSlotsInformation
Get all allocated solt information
private Collection<SlotInfo> getAllocatedSlotsInformation() { return allocatedSlots.listSlotInfo(); }
3.4.5 allocateAvailableSlot
Get all valid solt information
/** * Assign the slot associated with the allocationID to the request corresponding to the slotRequestId * @param slotRequestId identifying the requested slot * @param allocationID the allocation id of the requested available slot * @return */ @Override public Optional<PhysicalSlot> allocateAvailableSlot( @Nonnull SlotRequestId slotRequestId, @Nonnull AllocationID allocationID) { componentMainThreadExecutor.assertRunningInMainThread(); //Remove from availableSlots AllocatedSlot allocatedSlot = availableSlots.tryRemove(allocationID); if (allocatedSlot != null) { //Join the assigned mapping relationship allocatedSlots.add(slotRequestId, allocatedSlot); return Optional.of(allocatedSlot); } else { return Optional.empty(); } }
3.4.6 requestNewAllocatedSlot
Request the allocation of a new slot from the resource manager. This method does not return a slot from a slot that is already available in the pool, but adds a new slot to the pool, which will be allocated and returned immediately.
/** * Apply to RM for a new slot * * Request the allocation of a new slot from the resource manager. This method does not return a slot from a slot that is already available in the pool, * Instead, a new slot will be added to the pool, which will be allocated and returned immediately. * * @param slotRequestId identifying the requested slot * @param resourceProfile resource profile that specifies the resource requirements for the requested slot * @param timeout timeout for the allocation procedure * @return */ @Nonnull @Override public CompletableFuture<PhysicalSlot> requestNewAllocatedSlot( @Nonnull SlotRequestId slotRequestId, @Nonnull ResourceProfile resourceProfile, Time timeout) { componentMainThreadExecutor.assertRunningInMainThread(); final PendingRequest pendingRequest = PendingRequest.createStreamingRequest(slotRequestId, resourceProfile); // register request timeout FutureUtils .orTimeout( pendingRequest.getAllocatedSlotFuture(), timeout.toMilliseconds(), TimeUnit.MILLISECONDS, componentMainThreadExecutor) .whenComplete( (AllocatedSlot ignored, Throwable throwable) -> { if (throwable instanceof TimeoutException) { timeoutPendingSlotRequest(slotRequestId); } }); return requestNewAllocatedSlotInternal(pendingRequest) .thenApply((Function.identity())); }
requestNewAllocatedSlotInternal
/** * * Request a new slot from RM * * Requests a new slot from the ResourceManager. If there is currently not ResourceManager * connected, then the request is stashed and send once a new ResourceManager is connected. * * @param pendingRequest pending slot request * @return An {@link AllocatedSlot} future which is completed once the slot is offered to the {@link SlotPool} */ @Nonnull private CompletableFuture<AllocatedSlot> requestNewAllocatedSlotInternal(PendingRequest pendingRequest) { if (resourceManagerGateway == null) { stashRequestWaitingForResourceManager(pendingRequest); } else { // Request a new slot from RM requestSlotFromResourceManager(resourceManagerGateway, pendingRequest); } return pendingRequest.getAllocatedSlotFuture(); }
3.4.7 requestNewAllocatedBatchSlot
A request to allocate a new batch slot from the resource manager is different from an ordinary slot. The batch slot will time out only when the slot pool does not contain an appropriate slot. In addition, it does not respond to fault signals from the resource manager.
@Nonnull @Override public CompletableFuture<PhysicalSlot> requestNewAllocatedBatchSlot( @Nonnull SlotRequestId slotRequestId, @Nonnull ResourceProfile resourceProfile) { componentMainThreadExecutor.assertRunningInMainThread(); final PendingRequest pendingRequest = PendingRequest.createBatchRequest(slotRequestId, resourceProfile); return requestNewAllocatedSlotInternal(pendingRequest) .thenApply(Function.identity()); }
3.4.8 disableBatchSlotRequestTimeoutCheck
Disable batch slot request timeout check. Called when someone else wants to take over the timeout check responsibility.
@Override public void disableBatchSlotRequestTimeoutCheck() { batchSlotRequestTimeoutCheckEnabled = false; }
3.4.9 createAllocatedSlotReport
Creates a report on assigned slot s belonging to the specified task manager.
/** * Creates a report on assigned slot s belonging to the specified task manager. * @param taskManagerId identifies the task manager * @return */ @Override public AllocatedSlotReport createAllocatedSlotReport(ResourceID taskManagerId) { final Set<AllocatedSlot> availableSlotsForTaskManager = availableSlots.getSlotsForTaskManager(taskManagerId); final Set<AllocatedSlot> allocatedSlotsForTaskManager = allocatedSlots.getSlotsForTaskManager(taskManagerId); List<AllocatedSlotInfo> allocatedSlotInfos = new ArrayList<>( availableSlotsForTaskManager.size() + allocatedSlotsForTaskManager.size()); for (AllocatedSlot allocatedSlot : Iterables.concat(availableSlotsForTaskManager, allocatedSlotsForTaskManager)) { allocatedSlotInfos.add( new AllocatedSlotInfo(allocatedSlot.getPhysicalSlotNumber(), allocatedSlot.getAllocationId())); } return new AllocatedSlotReport(jobId, allocatedSlotInfos); }