c

org.apache.spark.network.shuffle

RemoteBlockPushResolver

class RemoteBlockPushResolver extends MergedShuffleFileManager

An implementation of MergedShuffleFileManager that provides the most essential shuffle service processing logic to support push based shuffle.

Since

3.1.0

Linear Supertypes
MergedShuffleFileManager, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RemoteBlockPushResolver
  2. MergedShuffleFileManager
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new RemoteBlockPushResolver(conf: TransportConf)
    Annotations
    @SuppressWarnings()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def applicationRemoved(appId: String, cleanupLocalDirs: Boolean): Unit

    Invoked when an application finishes.

    Invoked when an application finishes. This cleans up any remaining metadata associated with this application, and optionally deletes the application specific directory path.

    appId

    application ID

    cleanupLocalDirs

    flag indicating whether MergedShuffleFileManager should handle deletion of local dirs itself.

    Definition Classes
    RemoteBlockPushResolver → MergedShuffleFileManager
    Annotations
    @Override()
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  10. def finalizeShuffleMerge(msg: FinalizeShuffleMerge): MergeStatuses

    Handles the request to finalize shuffle merge for a given shuffle.

    Handles the request to finalize shuffle merge for a given shuffle.

    msg

    contains appId and shuffleId to uniquely identify a shuffle to be finalized

    returns

    The statuses of the merged shuffle partitions for the given shuffle on this shuffle service

    Definition Classes
    RemoteBlockPushResolver → MergedShuffleFileManager
    Annotations
    @SuppressWarnings() @Override()
    Exceptions thrown
  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getMergedBlockData(appId: String, shuffleId: Int, shuffleMergeId: Int, reduceId: Int, chunkId: Int): ManagedBuffer

    Get the buffer for a given merged shuffle chunk when serving merged shuffle to reducers

    Get the buffer for a given merged shuffle chunk when serving merged shuffle to reducers

    appId

    application ID

    shuffleId

    shuffle ID

    shuffleMergeId

    shuffleMergeId is used to uniquely identify merging process of shuffle by an indeterminate stage attempt.

    reduceId

    reducer ID

    chunkId

    merged shuffle file chunk ID

    returns

    The ManagedBuffer for the given merged shuffle chunk

    Definition Classes
    RemoteBlockPushResolver → MergedShuffleFileManager
    Annotations
    @SuppressWarnings() @Override()
  13. def getMergedBlockDirs(appId: String): Array[String]

    Get the local directories which stores the merged shuffle files.

    Get the local directories which stores the merged shuffle files.

    appId

    application ID

    Definition Classes
    RemoteBlockPushResolver → MergedShuffleFileManager
    Annotations
    @Override()
  14. def getMergedBlockMeta(appId: String, shuffleId: Int, shuffleMergeId: Int, reduceId: Int): MergedBlockMeta

    Get the meta information of a merged block.

    Get the meta information of a merged block.

    appId

    application ID

    shuffleId

    shuffle ID

    shuffleMergeId

    shuffleMergeId is used to uniquely identify merging process of shuffle by an indeterminate stage attempt.

    reduceId

    reducer ID

    returns

    meta information of a merged block

    Definition Classes
    RemoteBlockPushResolver → MergedShuffleFileManager
    Annotations
    @Override()
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. def receiveBlockDataAsStream(msg: PushBlockStream): StreamCallbackWithID

    Provides the stream callback used to process a remotely pushed block.

    Provides the stream callback used to process a remotely pushed block. The callback is used by the org.apache.spark.network.client.StreamInterceptor installed on the channel to process the block data in the channel outside of the message frame.

    msg

    metadata of the remotely pushed blocks. This is processed inside the message frame

    returns

    A stream callback to process the block data in streaming fashion as it arrives

    Definition Classes
    RemoteBlockPushResolver → MergedShuffleFileManager
    Annotations
    @Override()
  21. def registerExecutor(appId: String, executorInfo: ExecutorShuffleInfo): Unit

    Registers an executor with MergedShuffleFileManager.

    Registers an executor with MergedShuffleFileManager. This executor-info provides the directories and number of sub-dirs per dir so that MergedShuffleFileManager knows where to store and look for shuffle data for a given application. It is invoked by the RPC call when executor tries to register with the local shuffle service.

    appId

    application ID

    executorInfo

    The list of local dirs that this executor gets granted from NodeManager

    Definition Classes
    RemoteBlockPushResolver → MergedShuffleFileManager
    Annotations
    @Override()
  22. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  23. def toString(): String
    Definition Classes
    AnyRef → Any
  24. def validateAndGetAppShuffleInfo(appId: String): AppShuffleInfo
    Attributes
    protected[shuffle]
    Annotations
    @VisibleForTesting()
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from MergedShuffleFileManager

Inherited from AnyRef

Inherited from Any

Ungrouped