Type Members

1.  abstract class AbstractAuthRpcHandler extends RpcHandler

   RPC Handler which performs authentication, and when it's successful, delegates further calls to another RPC handler.

   RPC Handler which performs authentication, and when it's successful, delegates further calls to another RPC handler. The authentication handshake itself should be implemented by subclasses.

2.  class BlockPushNonFatalFailure extends RuntimeException

   A special RuntimeException thrown when shuffle service experiences a non-fatal failure with handling block push requests with push-based shuffle.

   A special RuntimeException thrown when shuffle service experiences a non-fatal failure with handling block push requests with push-based shuffle. Due to the best-effort nature of push-based shuffle, there are cases where the exceptions gets thrown under certain relatively common cases such as when a pushed block is received after the corresponding shuffle is merge finalized or when a pushed block experiences merge collision. Under these scenarios, we throw this special RuntimeException.

3.  class ChunkFetchRequestHandler extends SimpleChannelInboundHandler[ChunkFetchRequest]

   A dedicated ChannelHandler for processing ChunkFetchRequest messages.

   A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying channel could potentially be blocked due to disk contentions. If several hundreds of clients send ChunkFetchRequest to the server at the same time, it could potentially occupying all threads from TransportServer's default EventLoopGroup for waiting for disk reads before it can send the block data back to the client as part of the ChunkFetchSuccess messages. As a result, it would leave no threads left to process other RPC messages, which takes much less time to process, and could lead to client timing out on either performing SASL authentication, registering executors, or waiting for response for an OpenBlocks messages.

4.  abstract class MessageHandler[T <: Message] extends AnyRef

   Handles either request or response messages coming off of Netty.

   Handles either request or response messages coming off of Netty. A MessageHandler instance is associated with a single Netty Channel (though it may have multiple clients on the same Channel.)

5.  class NoOpRpcHandler extends RpcHandler

   An RpcHandler suitable for a client-only TransportContext, which cannot receive RPCs.

6.  class OneForOneStreamManager extends StreamManager

   StreamManager which allows registration of an Iterator<ManagedBuffer>, which are individually fetched as chunks by the client.

   StreamManager which allows registration of an Iterator<ManagedBuffer>, which are individually fetched as chunks by the client. Each registered buffer is one chunk.

7.  abstract class RpcHandler extends AnyRef

   Handler for sendRPC() messages sent by org.apache.spark.network.client.TransportClients.

8.  abstract class StreamManager extends AnyRef

   The StreamManager is used to fetch individual chunks from a stream.

   The StreamManager is used to fetch individual chunks from a stream. This is used in TransportRequestHandler in order to respond to fetchChunk() requests. Creation of the stream is outside the scope of the transport layer, but a given stream is guaranteed to be read by only one client connection, meaning that getChunk() for a particular stream will be called serially and that once the connection associated with the stream is closed, that stream will never be used again.

9.  class TransportChannelHandler extends SimpleChannelInboundHandler[Message]

   The single Transport-level Channel handler which is used for delegating requests to the TransportRequestHandler and responses to the TransportResponseHandler.

   The single Transport-level Channel handler which is used for delegating requests to the TransportRequestHandler and responses to the TransportResponseHandler.

   All channels created in the transport layer are bidirectional. When the Client initiates a Netty Channel with a RequestMessage (which gets handled by the Server's RequestHandler), the Server will produce a ResponseMessage (handled by the Client's ResponseHandler). However, the Server also gets a handle on the same Channel, so it may then begin to send RequestMessages to the Client. This means that the Client also needs a RequestHandler and the Server needs a ResponseHandler, for the Client's responses to the Server's requests.

   This class also handles timeouts from a io.netty.handler.timeout.IdleStateHandler. We consider a connection timed out if there are outstanding fetch or RPC requests but no traffic on the channel for at least requestTimeoutMs. Note that this is duplex traffic; we will not timeout if the client is continuously sending but getting no responses, for simplicity.

10.  class TransportRequestHandler extends MessageHandler[RequestMessage]

    A handler that processes requests from clients and writes chunk data back.

    A handler that processes requests from clients and writes chunk data back. Each handler is attached to a single Netty channel, and keeps track of which streams have been fetched via this channel, in order to clean them up if the channel is terminated (see #channelUnregistered).

    The messages should have been processed by the pipeline setup by TransportServer.

11.  class TransportServer extends Closeable

    Server for the efficient, low-level streaming service.

12.  trait TransportServerBootstrap extends AnyRef

    A bootstrap which is executed on a TransportServer's client channel once a client connects to the server.

    A bootstrap which is executed on a TransportServer's client channel once a client connects to the server. This allows customizing the client channel to allow for things such as SASL authentication.