package protocol
Type Members
- abstract class AbstractFetchShuffleBlocks extends BlockTransferMessage
Base class for fetch shuffle blocks and chunks.
Base class for fetch shuffle blocks and chunks.
- Since
3.2.0
- class BlockPushReturnCode extends BlockTransferMessage
Error code indicating a non-fatal failure of a block push request.
Error code indicating a non-fatal failure of a block push request. Due to the best-effort nature of push-based shuffle, these failures do not impact the completion of the block push process. The list of such errors is in
org.apache.spark.network.server.BlockPushNonFatalFailure.ReturnCode.- Since
3.2.0
- abstract class BlockTransferMessage extends Encodable
Messages handled by the
ExternalBlockHandler, or by Spark's NettyBlockTransferService.Messages handled by the
ExternalBlockHandler, or by Spark's NettyBlockTransferService.At a high level:
- OpenBlock is logically only handled by the NettyBlockTransferService, but for the capability for old version Spark, we still keep it in external shuffle service. It returns a StreamHandle.
- UploadBlock is only handled by the NettyBlockTransferService.
- RegisterExecutor is only handled by the external shuffle service.
- RemoveBlocks is only handled by the external shuffle service.
- FetchShuffleBlocks is handled by both services for shuffle files. It returns a StreamHandle.
- class BlocksRemoved extends BlockTransferMessage
The reply to remove blocks giving back the number of removed blocks.
- class CorruptionCause extends BlockTransferMessage
Response to the
DiagnoseCorruption - class DiagnoseCorruption extends BlockTransferMessage
Request to get the cause of a corrupted block.
Request to get the cause of a corrupted block. Returns
CorruptionCause - class ExecutorShuffleInfo extends Encodable
Contains all configuration necessary for locating the shuffle files of an executor.
- class FetchShuffleBlockChunks extends AbstractFetchShuffleBlocks
Request to read a set of block chunks.
Request to read a set of block chunks. Returns
StreamHandle.- Since
3.2.0
- class FetchShuffleBlocks extends AbstractFetchShuffleBlocks
Request to read a set of blocks.
Request to read a set of blocks. Returns
StreamHandle. - class FinalizeShuffleMerge extends BlockTransferMessage
Request to finalize merge for a given shuffle.
Request to finalize merge for a given shuffle. Returns
MergeStatuses- Since
3.1.0
- class GetLocalDirsForExecutors extends BlockTransferMessage
Request to get the local dirs for the given executors.
- class LocalDirsForExecutors extends BlockTransferMessage
The reply to get local dirs giving back the dirs for each of the requested executors.
- class MergeStatuses extends BlockTransferMessage
Result returned by an ExternalShuffleService to the DAGScheduler.
Result returned by an ExternalShuffleService to the DAGScheduler. This represents the result of all the remote shuffle block merge operations performed by an ExternalShuffleService for a given shuffle ID. It includes the shuffle ID, an array of bitmaps each representing the set of mapper partition blocks that are merged for a given reducer partition, an array of reducer IDs, and an array of merged shuffle partition sizes. The 3 arrays list information about all the reducer partitions merged by the ExternalShuffleService in the same order.
- Since
3.1.0
- class OpenBlocks extends BlockTransferMessage
Request to read a set of blocks.
Request to read a set of blocks. Returns
StreamHandle. - class PushBlockStream extends BlockTransferMessage
Request to push a block to a remote shuffle service to be merged in push based shuffle.
Request to push a block to a remote shuffle service to be merged in push based shuffle. The remote shuffle service will also include this message when responding the push requests.
- Since
3.1.0
- class RegisterExecutor extends BlockTransferMessage
Initial registration message between an executor and its local shuffle server.
Initial registration message between an executor and its local shuffle server. Returns nothing (empty byte array).
- class RemoveBlocks extends BlockTransferMessage
Request to remove a set of blocks.
- class StreamHandle extends BlockTransferMessage
Identifier for a fixed number of chunks to read from a stream created by an "open blocks" message.
Identifier for a fixed number of chunks to read from a stream created by an "open blocks" message. This is used by
org.apache.spark.network.shuffle.OneForOneBlockFetcher. - class UploadBlock extends BlockTransferMessage
Request to upload a block with a certain StorageLevel.
Request to upload a block with a certain StorageLevel. Returns nothing (empty byte array).
- class UploadBlockStream extends BlockTransferMessage
A request to Upload a block, which the destination should receive as a stream.
A request to Upload a block, which the destination should receive as a stream.
The actual block data is not contained here. It will be passed to the StreamCallbackWithID that is returned from RpcHandler.receiveStream()