Class RemoteChunkProvider
- java.lang.Object
-
- com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider<ArrowResultChunk>
-
- com.databricks.jdbc.api.impl.arrow.RemoteChunkProvider
-
- All Implemented Interfaces:
ChunkProvider
public class RemoteChunkProvider extends AbstractRemoteChunkProvider<ArrowResultChunk>
-
-
Field Summary
-
Fields inherited from class com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider
allowedChunksInMemory, chunkCount, chunkIndexToChunksMap, chunkReadyTimeoutSeconds, compressionCodec, currentChunkIndex, httpClient, isClosed, linkDownloadService, maxParallelChunkDownloadsPerQuery, nextChunkToDownload, rowCount, session, statementId, totalChunksInMemory
-
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected ArrowResultChunkcreateChunk(StatementId statementId, long chunkIndex, TSparkArrowResultLink resultLink)Creates chunkAbstractRemoteChunkProviderbased on theTSparkArrowResultLink.protected ArrowResultChunkcreateChunk(StatementId statementId, long chunkIndex, com.databricks.sdk.service.sql.BaseChunkInfo chunkInfo)Creates chunkAbstractRemoteChunkProviderbased on theBaseChunkInfo.protected voiddoClose()Subclasses should override this method to perform their specific cleanup.voiddownloadNextChunks()-
Methods inherited from class com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider
close, getAllowedChunksInMemory, getChunk, getChunkCount, getCompressionCodec, getRowCount, hasNextChunk, isClosed, next
-
-
-
-
Method Detail
-
createChunk
protected ArrowResultChunk createChunk(StatementId statementId, long chunkIndex, com.databricks.sdk.service.sql.BaseChunkInfo chunkInfo) throws DatabricksSQLException
Creates chunkAbstractRemoteChunkProviderbased on theBaseChunkInfo. Used in SQL Execution API flow.- Specified by:
createChunkin classAbstractRemoteChunkProvider<ArrowResultChunk>- Throws:
DatabricksSQLException
-
createChunk
protected ArrowResultChunk createChunk(StatementId statementId, long chunkIndex, TSparkArrowResultLink resultLink) throws DatabricksSQLException
Creates chunkAbstractRemoteChunkProviderbased on theTSparkArrowResultLink. Used in Thrift CLI flow.- Specified by:
createChunkin classAbstractRemoteChunkProvider<ArrowResultChunk>- Throws:
DatabricksSQLException
-
downloadNextChunks
public void downloadNextChunks()
Downloads the next set of available chunks asynchronously using a thread pool executor. This method:
- Initializes a thread pool executor if not already created
- Submits chunk download tasks to the executor while:
- The provider is not closed
- There are more chunks available to download
- The number of chunks in memory is below the allowed limit
- Tracks the total chunks in memory and the next chunk to download
ChunkDownloadTaskrunning in the executor service. This implementation provides non-blocking downloads using a custom thread pool for chunk downloads.
-
doClose
protected void doClose()
Subclasses should override this method to perform their specific cleanup.- Overrides:
doClosein classAbstractRemoteChunkProvider<ArrowResultChunk>
-
-