Package com.databricks.jdbc.api.impl.arrow

Class ChunkLinkDownloadService<T extends AbstractArrowResultChunk>

java.lang.Object

com.databricks.jdbc.api.impl.arrow.ChunkLinkDownloadService<T>

Type Parameters:

T - The specific type of AbstractArrowResultChunk this service manages

public class ChunkLinkDownloadService<T extends AbstractArrowResultChunk>
extends Object

A service that manages the downloading and refreshing of external links for chunked data retrieval.

This service maintains a CompletableFuture for each chunk's external link.

Key Features:

1. Download Pipeline:

• Automatically initiates a download chain when using SQL Execution API
• Fetches links in batches, starting from a specified chunk index
• Processes batches serially, with each new request starting from (last fetched index + 1)
• Completes the corresponding futures as soon as links are received

2. Link Expiration Handling:

• Monitors link expiration when chunks request their download links
• When an expired link is detected (and its chunk hasn't been downloaded):
  • Finds the earliest chunk index with an expired link
  • Restarts the download chain from this index

3. Correctness Guarantee:

The service maintains correctness through two mechanisms:

• Monotonically increasing request indexes
• Server's guarantee of returning continuous series of chunk links

This design ensures that no chunks are missed and links remain valid during the download process.

Constructor Summary

Constructors

Constructor	Description
ChunkLinkDownloadService(IDatabricksSession session, StatementId statementId, long totalChunks, ConcurrentMap<Long,T> chunkIndexToChunksMap, long nextBatchStartIndex)

Method Summary

Modifier and Type	Method	Description
CompletableFuture<ExternalLink>	getLinkForChunk(long chunkIndex)	Retrieves the external link for a specified chunk index.
void	shutdown()	Shuts down the service and cancels all pending operations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ChunkLinkDownloadService

public ChunkLinkDownloadService(IDatabricksSession session,
                                StatementId statementId,
                                long totalChunks,
                                ConcurrentMap<Long,T> chunkIndexToChunksMap,
                                long nextBatchStartIndex)

Method Detail

getLinkForChunk

public CompletableFuture<ExternalLink> getLinkForChunk(long chunkIndex)
                                                throws ExecutionException,
                                                       InterruptedException

Retrieves the external link for a specified chunk index.

This method:

• Checks if the service is operational
• Validates the requested chunk index
• Handles expired links if necessary
• Initiates the download chain if not already started

Parameters:

chunkIndex - The index of the chunk for which to get the link

Returns:

A CompletableFuture containing the ExternalLink for the requested chunk

Throws:

ExecutionException - If the link retrieval fails

InterruptedException - If the operation is interrupted

shutdown

public void shutdown()

Shuts down the service and cancels all pending operations.