Package com.databricks.jdbc.api.impl.arrow.incubator

Class RemoteChunkProviderV2

java.lang.Object

com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider<ArrowResultChunkV2>

com.databricks.jdbc.api.impl.arrow.incubator.RemoteChunkProviderV2

All Implemented Interfaces:

ChunkProvider

public class RemoteChunkProviderV2
extends AbstractRemoteChunkProvider<ArrowResultChunkV2>

A V2 implementation of chunk provider that handles chunk downloads using Apache's async HTTP client. Each chunk is downloaded sequentially but processed asynchronously through streaming responses.

Field Summary

Fields inherited from class com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider

allowedChunksInMemory, chunkCount, chunkIndexToChunksMap, chunkReadyTimeoutSeconds, compressionCodec, currentChunkIndex, httpClient, isClosed, linkDownloadService, maxParallelChunkDownloadsPerQuery, nextChunkToDownload, rowCount, session, statementId, totalChunksInMemory

Constructor Summary

Constructors

Constructor	Description
RemoteChunkProviderV2(IDatabricksStatementInternal parentStatement, TFetchResultsResp resultsResp, IDatabricksSession session, IDatabricksHttpClient httpClient, int maxParallelChunkDownloadsPerQuery, CompressionCodec compressionCodec)
RemoteChunkProviderV2(StatementId statementId, ResultManifest resultManifest, ResultData resultData, IDatabricksSession session, IDatabricksHttpClient httpClient, int maxParallelChunkDownloadsPerQuery)

Method Summary

Modifier and Type	Method	Description
protected ArrowResultChunkV2	createChunk(StatementId statementId, long chunkIndex, TSparkArrowResultLink resultLink)	Creates chunk AbstractRemoteChunkProvider based on the TSparkArrowResultLink.
protected ArrowResultChunkV2	createChunk(StatementId statementId, long chunkIndex, com.databricks.sdk.service.sql.BaseChunkInfo chunkInfo)	Creates chunk AbstractRemoteChunkProvider based on the BaseChunkInfo.
protected void	doClose()	Subclasses should override this method to perform their specific cleanup.
void	downloadNextChunks()

Methods inherited from class com.databricks.jdbc.api.impl.arrow.AbstractRemoteChunkProvider

close, getAllowedChunksInMemory, getChunk, getChunkCount, getCompressionCodec, getRowCount, hasNextChunk, isClosed, next

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteChunkProviderV2

public RemoteChunkProviderV2(StatementId statementId,
                             ResultManifest resultManifest,
                             ResultData resultData,
                             IDatabricksSession session,
                             IDatabricksHttpClient httpClient,
                             int maxParallelChunkDownloadsPerQuery)
                      throws DatabricksSQLException

Throws:

DatabricksSQLException

RemoteChunkProviderV2

public RemoteChunkProviderV2(IDatabricksStatementInternal parentStatement,
                             TFetchResultsResp resultsResp,
                             IDatabricksSession session,
                             IDatabricksHttpClient httpClient,
                             int maxParallelChunkDownloadsPerQuery,
                             CompressionCodec compressionCodec)
                      throws DatabricksSQLException

Throws:

DatabricksSQLException

Method Detail

createChunk

protected ArrowResultChunkV2 createChunk(StatementId statementId,
                                         long chunkIndex,
                                         com.databricks.sdk.service.sql.BaseChunkInfo chunkInfo)

Description copied from class: AbstractRemoteChunkProvider

Creates chunk AbstractRemoteChunkProvider based on the BaseChunkInfo. Used in SQL Execution API flow.

Specified by:

createChunk in class AbstractRemoteChunkProvider<ArrowResultChunkV2>

createChunk

protected ArrowResultChunkV2 createChunk(StatementId statementId,
                                         long chunkIndex,
                                         TSparkArrowResultLink resultLink)

Description copied from class: AbstractRemoteChunkProvider

Creates chunk AbstractRemoteChunkProvider based on the TSparkArrowResultLink. Used in Thrift CLI flow.

Specified by:

createChunk in class AbstractRemoteChunkProvider<ArrowResultChunkV2>

downloadNextChunks

public void downloadNextChunks()
                        throws DatabricksSQLException

Downloads the next set of available chunks using Apache's async HTTP client. This method processes chunks sequentially but each chunk download is performed asynchronously. For each chunk, it:

• Checks if the provider is not closed
• Verifies more chunks are available to download
• Ensures the number of chunks in memory is below the allowed limit
• Downloads the chunk data if these conditions are met

The actual download is performed using ArrowResultChunkV2's streaming response consumer, which handles:

• Asynchronous data streaming
• Automatic retries with exponential backoff
• Download statistics tracking
• Memory-efficient data processing

Throws:

DatabricksSQLException - If there's an error during the chunk download process, including errors in downloading links or chunk data

doClose

protected void doClose()

Description copied from class: AbstractRemoteChunkProvider

Subclasses should override this method to perform their specific cleanup.

Overrides:

doClose in class AbstractRemoteChunkProvider<ArrowResultChunkV2>