com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageReadChannel

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel

All Implemented Interfaces:

Closeable, AutoCloseable, ByteChannel, Channel, ReadableByteChannel, SeekableByteChannel, WritableByteChannel

Direct Known Subclasses:

InMemoryObjectReadChannel

public class GoogleCloudStorageReadChannel
extends Object
implements SeekableByteChannel

Provides seekable read access to GCS.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	contentChannelPosition
protected long	currentPosition
protected boolean	metadataInitialized

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs, StorageResourceId resourceId, ApiErrorExtractor errorExtractor, ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper, GoogleCloudStorageReadOptions readOptions) Constructs an instance of GoogleCloudStorageReadChannel.

Method Summary

Modifier and Type	Method and Description
void	close() Closes this channel.
protected void	closeContentChannel() Closes the underlying ReadableByteChannel.
protected com.google.api.services.storage.Storage.Objects.Get	createDataRequest()
protected com.google.api.services.storage.Storage.Objects.Get	createMetadataRequest()
protected GoogleCloudStorageItemInfo	getInitialMetadata() Returns GoogleCloudStorageItemInfo used to initialize metadata in constructor or null if GoogleCloudStorageReadOptions.getFastFailOnNotFound() is set to false.
protected void	initMetadata(com.google.api.client.http.HttpHeaders headers) Initializes metadata (size, encoding, etc) from HTTP headers.
protected void	initMetadata(String encoding, long sizeFromMetadata, long generation) Initializes metadata (size, encoding, etc) from passed parameters.
boolean	isOpen() Tells whether this channel is open.
protected InputStream	openStream(long bytesToRead) Opens the underlying stream, sets its position to the currentPosition.
long	position() Returns this channel's current position.
SeekableByteChannel	position(long newPosition) Sets this channel's position.
int	read(ByteBuffer buffer) Reads from this channel and stores read data in the given buffer.
void	setMaxRetries(int maxRetries) Sets the number of times to automatically retry by re-opening the underlying contentChannel whenever an exception occurs while reading from it.
protected void	setSize(long size) Sets size of this channel to the given value.
long	size() Returns size of the object to which this channel is connected.
SeekableByteChannel	truncate(long size)
protected void	validatePosition(long position) Validates that the given position is valid for this channel.
int	write(ByteBuffer src)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

currentPosition

protected long currentPosition

contentChannelPosition

protected long contentChannelPosition

metadataInitialized

protected boolean metadataInitialized

Constructor Detail

GoogleCloudStorageReadChannel

public GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs,
                                     StorageResourceId resourceId,
                                     ApiErrorExtractor errorExtractor,
                                     ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper,
                                     @Nonnull
                                     GoogleCloudStorageReadOptions readOptions)
                              throws IOException

Constructs an instance of GoogleCloudStorageReadChannel.

Parameters:

gcs - storage object instance

resourceId - contains information about a specific resource

requestHelper - a ClientRequestHelper used to set any extra headers

readOptions - fine-grained options specifying things like retry settings, buffering, etc. Could not be null.

Throws:

IOException - on IO error

Method Detail

getInitialMetadata

@Nullable
protected GoogleCloudStorageItemInfo getInitialMetadata()
                                                           throws IOException

Returns GoogleCloudStorageItemInfo used to initialize metadata in constructor or null if GoogleCloudStorageReadOptions.getFastFailOnNotFound() is set to false.

Throws:

IOException

setMaxRetries

public void setMaxRetries(int maxRetries)

Sets the number of times to automatically retry by re-opening the underlying contentChannel whenever an exception occurs while reading from it. The count of attempted retries is reset whenever at least one byte is successfully read, so this number of retries refers to retries made without achieving any forward progress.

read

public int read(ByteBuffer buffer)
         throws IOException

Reads from this channel and stores read data in the given buffer.

On unexpected failure, will attempt to close the channel and clean up state.

Specified by:

read in interface ReadableByteChannel

Specified by:

read in interface SeekableByteChannel

Parameters:

buffer - buffer to read data into

Returns:

number of bytes read or -1 on end-of-stream

Throws:

IOException - on IO error

truncate

public SeekableByteChannel truncate(long size)
                             throws IOException

Specified by:

truncate in interface SeekableByteChannel

Throws:

IOException

write

public int write(ByteBuffer src)
          throws IOException

Specified by:

write in interface SeekableByteChannel

Specified by:

write in interface WritableByteChannel

Throws:

IOException

isOpen

public boolean isOpen()

Tells whether this channel is open.

Specified by:

isOpen in interface Channel

Returns:

a value indicating whether this channel is open

closeContentChannel

protected void closeContentChannel()

Closes the underlying ReadableByteChannel.

Catches and ignores all exceptions as there is not a lot the user can do to fix errors here and a new connection will be needed. Especially SSLExceptions since the there's a high probability that SSL connections would be broken in a way that causes Channel.close() itself to throw an exception, even though underlying sockets have already been cleaned up; close() on an SSLSocketImpl requires a shutdown handshake in order to shutdown cleanly, and if the connection has been broken already, then this is not possible, and the SSLSocketImpl was already responsible for performing local cleanup at the time the exception was raised.

close

public void close()

Closes this channel.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Channel

position

public long position()
              throws IOException

Returns this channel's current position.

Specified by:

position in interface SeekableByteChannel

Returns:

this channel's current position

Throws:

IOException

position

public SeekableByteChannel position(long newPosition)
                             throws IOException

Sets this channel's position.

This method will throw an exception if newPosition is greater than object size, which contradicts SeekableByteChannel contract. TODO(user): decide if this needs to be fixed.

Specified by:

position in interface SeekableByteChannel

Parameters:

newPosition - the new position, counting the number of bytes from the beginning.

Returns:

this channel instance

Throws:

FileNotFoundException - if the underlying object does not exist.

IOException - on IO error

size

public long size()
          throws IOException

Returns size of the object to which this channel is connected.

Note: this method will return -1 until metadata will be lazily initialized during first read(java.nio.ByteBuffer) method call.

Specified by:

size in interface SeekableByteChannel

Returns:

size of the object to which this channel is connected after metadata was initialized (during first read) or -1 otherwise.

Throws:

IOException - on IO error

setSize

protected void setSize(long size)

Sets size of this channel to the given value.

validatePosition

protected void validatePosition(long position)
                         throws IOException

Validates that the given position is valid for this channel.

Throws:

IOException

initMetadata

protected void initMetadata(com.google.api.client.http.HttpHeaders headers)
                     throws IOException

Initializes metadata (size, encoding, etc) from HTTP headers. Used for lazy initialization when fail fast is disabled.

Throws:

IOException

initMetadata

protected void initMetadata(@Nullable
                            String encoding,
                            long sizeFromMetadata,
                            long generation)
                     throws IOException

Initializes metadata (size, encoding, etc) from passed parameters.

Throws:

IOException

openStream

protected InputStream openStream(long bytesToRead)
                          throws IOException

Opens the underlying stream, sets its position to the currentPosition.

If the file encoding in GCS is gzip (and therefore the HTTP client will decompress it), the entire file is always requested and we seek to the position requested. If the file encoding is not gzip, only the remaining bytes to be read are requested from GCS.

Parameters:

bytesToRead - number of bytes to read from new stream. Ignored if GoogleCloudStorageReadOptions.getFadvise() is equal to GoogleCloudStorageReadOptions.Fadvise.SEQUENTIAL.

Throws:

IOException - on IO error

createDataRequest

protected com.google.api.services.storage.Storage.Objects.Get createDataRequest()
                                                                         throws IOException

Throws:

IOException

createMetadataRequest

protected com.google.api.services.storage.Storage.Objects.Get createMetadataRequest()
                                                                             throws IOException

Throws:

IOException