Package com.databricks.jdbc.api.impl.arrow

Class RemoteChunkProvider

java.lang.Object

com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider<ArrowResultChunk>

com.databricks.jdbc.api.impl.arrow.RemoteChunkProvider

All Implemented Interfaces:

ChunkProvider

public class RemoteChunkProvider
extends AbstractRemoteChunkProvider<ArrowResultChunk>

Field Summary

Fields inherited from class com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider

allowedChunksInMemory, chunkCount, chunkIndexToChunksMap, chunkReadyTimeoutSeconds, compressionCodec, currentChunkIndex, httpClient, isClosed, linkDownloadService, maxParallelChunkDownloadsPerQuery, nextChunkToDownload, rowCount, session, statementId, totalChunksInMemory

Method Summary

Modifier and Type	Method	Description
protected ArrowResultChunk	createChunk(StatementId statementId, long chunkIndex, TSparkArrowResultLink resultLink)	Creates chunk AbstractRemoteChunkProvider based on the TSparkArrowResultLink.
protected ArrowResultChunk	createChunk(StatementId statementId, long chunkIndex, com.databricks.sdk.service.sql.BaseChunkInfo chunkInfo)	Creates chunk AbstractRemoteChunkProvider based on the BaseChunkInfo.
protected void	doClose()	Subclasses should override this method to perform their specific cleanup.
void	downloadNextChunks()

Methods inherited from class com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider

close, getAllowedChunksInMemory, getChunk, getChunkCount, getCompressionCodec, getRowCount, hasNextChunk, isClosed, next

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createChunk

protected ArrowResultChunk createChunk(StatementId statementId,
                                       long chunkIndex,
                                       com.databricks.sdk.service.sql.BaseChunkInfo chunkInfo)
                                throws DatabricksSQLException

Creates chunk AbstractRemoteChunkProvider based on the BaseChunkInfo. Used in SQL Execution API flow.

Specified by:

createChunk in class AbstractRemoteChunkProvider<ArrowResultChunk>

Throws:

DatabricksSQLException

createChunk

protected ArrowResultChunk createChunk(StatementId statementId,
                                       long chunkIndex,
                                       TSparkArrowResultLink resultLink)
                                throws DatabricksSQLException

Creates chunk AbstractRemoteChunkProvider based on the TSparkArrowResultLink. Used in Thrift CLI flow.

Specified by:

createChunk in class AbstractRemoteChunkProvider<ArrowResultChunk>

Throws:

DatabricksSQLException

downloadNextChunks

public void downloadNextChunks()

Downloads the next set of available chunks asynchronously using a thread pool executor. This method:

• Initializes a thread pool executor if not already created
• Submits chunk download tasks to the executor while:
  • The provider is not closed
  • There are more chunks available to download
  • The number of chunks in memory is below the allowed limit
• Tracks the total chunks in memory and the next chunk to download

Each chunk download is handled by a separate ChunkDownloadTask running in the executor service. This implementation provides non-blocking downloads using a custom thread pool for chunk downloads.

doClose

protected void doClose()

Subclasses should override this method to perform their specific cleanup.

Overrides:

doClose in class AbstractRemoteChunkProvider<ArrowResultChunk>