com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageImpl

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl

All Implemented Interfaces:

GoogleCloudStorage

public class GoogleCloudStorageImpl
extends Object
implements GoogleCloudStorage

Provides read/write access to Google Cloud Storage (GCS), using Java nio channel semantics. This is a basic implementation of the GoogleCloudStorage interface that mostly delegates through to the appropriate API call(s) via the generated JSON API client, while adding reliability and performance features such as setting up low-level retries, translating low-level exceptions, and request batching.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	GoogleCloudStorageImpl.BackOffFactory A factory for producing BackOff objects.

Nested classes/interfaces inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

GoogleCloudStorage.ListPage<T>

Field Summary

Fields inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

MAX_COMPOSE_OBJECTS, MAX_RESULTS_UNLIMITED, PATH_DELIMITER

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.client.auth.oauth2.Credential credential) Constructs an instance of GoogleCloudStorageImpl.
GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.client.auth.oauth2.Credential credential, Function<List<AccessBoundary>,String> downscopedAccessTokenFn) Constructs an instance of GoogleCloudStorageImpl.
GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.client.http.HttpRequestInitializer httpRequestInitializer)
GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.client.http.HttpRequestInitializer httpRequestInitializer, Function<List<AccessBoundary>,String> downscopedAccessTokenFn)
GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.services.storage.Storage storage) Constructs an instance of GoogleCloudStorageImpl.
GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.services.storage.Storage storage, com.google.auth.Credentials credentials) Constructs an instance of GoogleCloudStorageImpl.
GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.services.storage.Storage storage, com.google.auth.Credentials credentials, Function<List<AccessBoundary>,String> downscopedAccessTokenFn) Constructs an instance of GoogleCloudStorageImpl.

Method Summary

Modifier and Type	Method and Description
void	close() See GoogleCloudStorage.close() for details about expected behavior.
void	compose(String bucketName, List<String> sources, String destination, String contentType) Composes inputs into a single GCS object.
GoogleCloudStorageItemInfo	composeObjects(List<StorageResourceId> sources, StorageResourceId destination, CreateObjectOptions options) Composes inputs into a single GCS object.
void	copy(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap) See GoogleCloudStorage.copy(String, List, String, List) for details about expected behavior.
void	copy(String srcBucketName, List<String> srcObjectNames, String dstBucketName, List<String> dstObjectNames) See GoogleCloudStorage.copy(String, List, String, List) for details about expected behavior.
WritableByteChannel	create(StorageResourceId resourceId, CreateObjectOptions options) Creates and opens an object for writing.
void	createBucket(String bucketName, CreateBucketOptions options) See GoogleCloudStorage.createBucket(String, CreateBucketOptions) for details about expected behavior.
void	createEmptyObject(StorageResourceId resourceId) See GoogleCloudStorage.createEmptyObject(StorageResourceId) for details about expected behavior.
void	createEmptyObject(StorageResourceId resourceId, CreateObjectOptions options) Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObjects(List<StorageResourceId> resourceIds) See GoogleCloudStorage.createEmptyObjects(List) for details about expected behavior.
void	createEmptyObjects(List<StorageResourceId> resourceIds, CreateObjectOptions options) Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method.
static GoogleCloudStorageItemInfo	createItemInfoForBucket(StorageResourceId resourceId, com.google.api.services.storage.model.Bucket bucket) Helper for converting a StorageResourceId + Bucket into a GoogleCloudStorageItemInfo.
static GoogleCloudStorageItemInfo	createItemInfoForStorageObject(com.google.api.services.storage.model.StorageObject object)
static GoogleCloudStorageItemInfo	createItemInfoForStorageObject(StorageResourceId resourceId, com.google.api.services.storage.model.StorageObject object) Helper for converting a StorageResourceId + StorageObject into a GoogleCloudStorageItemInfo.
void	deleteBuckets(List<String> bucketNames) See GoogleCloudStorage.deleteBuckets(List) for details about expected behavior.
void	deleteObject(StorageResourceId resourceId, long metaGeneration)
void	deleteObjects(List<StorageResourceId> fullObjectNames) See GoogleCloudStorage.deleteObjects(List) for details about expected behavior.
GoogleCloudStorageItemInfo	getItemInfo(StorageResourceId resourceId) See GoogleCloudStorage.getItemInfo(StorageResourceId) for details about expected behavior.
List<GoogleCloudStorageItemInfo>	getItemInfos(List<StorageResourceId> resourceIds) See GoogleCloudStorage.getItemInfos(List) for details about expected behavior.
GoogleCloudStorageOptions	getOptions() Retrieve the options that were used to create this GoogleCloudStorage.
List<GoogleCloudStorageItemInfo>	listBucketInfo() See GoogleCloudStorage.listBucketInfo() for details about expected behavior.
List<String>	listBucketNames() See GoogleCloudStorage.listBucketNames() for details about expected behavior.
List<GoogleCloudStorageItemInfo>	listObjectInfo(String bucketName, String objectNamePrefix, ListObjectOptions listOptions) Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.
GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo>	listObjectInfoPage(String bucketName, String objectNamePrefix, ListObjectOptions listOptions, String pageToken) The same semantics as GoogleCloudStorage.listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).
SeekableByteChannel	open(StorageResourceId resourceId, GoogleCloudStorageReadOptions readOptions) See GoogleCloudStorage.open(StorageResourceId) for details about expected behavior.
List<GoogleCloudStorageItemInfo>	updateItems(List<UpdatableItemInfo> itemInfoList) Attempt to update metadata of the objects referenced within the passed itemInfo objects.
void	updateMetadata(GoogleCloudStorageItemInfo itemInfo, Map<String,byte[]> metadata)
static void	validateCopyArguments(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap, GoogleCloudStorage gcsImpl) Validates basic argument constraints like non-null, non-empty Strings, using Preconditions in addition to checking for src/dst bucket existence and compatibility of bucket properties such as location and storage-class.
static void	validateCopyArguments(String srcBucketName, List<String> srcObjectNames, String dstBucketName, List<String> dstObjectNames, GoogleCloudStorage googleCloudStorage)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

create, createBucket, listObjectInfo, listObjectInfoPage, open

Constructor Detail

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                              com.google.api.client.auth.oauth2.Credential credential)
                       throws IOException

Constructs an instance of GoogleCloudStorageImpl.

Parameters:

options - options to customize behavior

credential - OAuth2 credential that allows access to GCS

Throws:

IOException - on IO error

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                              com.google.api.client.auth.oauth2.Credential credential,
                              Function<List<AccessBoundary>,String> downscopedAccessTokenFn)
                       throws IOException

Constructs an instance of GoogleCloudStorageImpl.

Parameters:

options - options to customize behavior

credential - OAuth2 credential that allows access to GCS

downscopedAccessTokenFn - Function that generates downscoped access token.

Throws:

IOException - on IO error

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                              com.google.api.client.http.HttpRequestInitializer httpRequestInitializer)
                       throws IOException

Throws:

IOException

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                              com.google.api.client.http.HttpRequestInitializer httpRequestInitializer,
                              Function<List<AccessBoundary>,String> downscopedAccessTokenFn)
                       throws IOException

Throws:

IOException

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                              com.google.api.services.storage.Storage storage)

Constructs an instance of GoogleCloudStorageImpl.

Parameters:

storage - Storage to use for I/O.

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                              com.google.api.services.storage.Storage storage,
                              com.google.auth.Credentials credentials)

Constructs an instance of GoogleCloudStorageImpl.

Parameters:

options - GoogleCloudStorageOptions to use to initialize the object.

storage - Storage to use for I/O.

credentials - OAuth2 credentials that allows access to GCS

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                              com.google.api.services.storage.Storage storage,
                              com.google.auth.Credentials credentials,
                              Function<List<AccessBoundary>,String> downscopedAccessTokenFn)

Constructs an instance of GoogleCloudStorageImpl.

Parameters:

options - GoogleCloudStorageOptions to use to initialize the object.

storage - Storage to use for I/O.

credentials - OAuth2 credentials that allows access to GCS

downscopedAccessTokenFn - Function that generates downscoped access token.

Method Detail

validateCopyArguments

public static void validateCopyArguments(String srcBucketName,
                                         List<String> srcObjectNames,
                                         String dstBucketName,
                                         List<String> dstObjectNames,
                                         GoogleCloudStorage googleCloudStorage)
                                  throws IOException

Throws:

IOException

getOptions

public GoogleCloudStorageOptions getOptions()

Description copied from interface: GoogleCloudStorage

Retrieve the options that were used to create this GoogleCloudStorage.

Specified by:

getOptions in interface GoogleCloudStorage

create

public WritableByteChannel create(StorageResourceId resourceId,
                                  CreateObjectOptions options)
                           throws IOException

Description copied from interface: GoogleCloudStorage

Creates and opens an object for writing. The bucket must already exist. If resourceId contains a known generationId or a generationId of 0, it is used instead of any "overwrite" settings in the provided CreateObjectOptions (overwrite will only succeed if the provided generationId matches the existing object, where a generationId of 0 indicates no existing object expected).

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - Options to use when creating the object

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

createBucket

public void createBucket(String bucketName,
                         CreateBucketOptions options)
                  throws IOException

See GoogleCloudStorage.createBucket(String, CreateBucketOptions) for details about expected behavior.

Specified by:

createBucket in interface GoogleCloudStorage

Parameters:

bucketName - name of the bucket to create

options - options to use when creating bucket

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId,
                              CreateObjectOptions options)
                       throws IOException

Description copied from interface: GoogleCloudStorage

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten. See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - options to use when creating the object

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId)
                       throws IOException

See GoogleCloudStorage.createEmptyObject(StorageResourceId) for details about expected behavior.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Throws:

IOException - on IO error

updateMetadata

public void updateMetadata(GoogleCloudStorageItemInfo itemInfo,
                           Map<String,byte[]> metadata)
                    throws IOException

Throws:

IOException

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds,
                               CreateObjectOptions options)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency. See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds)
                        throws IOException

See GoogleCloudStorage.createEmptyObjects(List) for details about expected behavior.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

open

public SeekableByteChannel open(StorageResourceId resourceId,
                                GoogleCloudStorageReadOptions readOptions)
                         throws IOException

See GoogleCloudStorage.open(StorageResourceId) for details about expected behavior.

Specified by:

open in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

a channel for reading from the given object

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be opened

deleteBuckets

public void deleteBuckets(List<String> bucketNames)
                   throws IOException

See GoogleCloudStorage.deleteBuckets(List) for details about expected behavior.

Specified by:

deleteBuckets in interface GoogleCloudStorage

Parameters:

bucketNames - name of the buckets to delete

Throws:

FileNotFoundException - if the given bucket does not exist

IOException - on IO error

deleteObject

public void deleteObject(StorageResourceId resourceId,
                         long metaGeneration)
                  throws IOException

Throws:

IOException

deleteObjects

public void deleteObjects(List<StorageResourceId> fullObjectNames)
                   throws IOException

See GoogleCloudStorage.deleteObjects(List) for details about expected behavior.

Specified by:

deleteObjects in interface GoogleCloudStorage

Parameters:

fullObjectNames - names of objects to delete with their respective bucketNames.

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be deleted

validateCopyArguments

public static void validateCopyArguments(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap,
                                         GoogleCloudStorage gcsImpl)
                                  throws IOException

Validates basic argument constraints like non-null, non-empty Strings, using Preconditions in addition to checking for src/dst bucket existence and compatibility of bucket properties such as location and storage-class.

Parameters:

gcsImpl - A GoogleCloudStorage for retrieving bucket info via getItemInfo, but only if srcBucketName != dstBucketName; passed as a parameter so that this static method can be used by other implementations of GoogleCloudStorage that want to preserve the validation behavior of GoogleCloudStorageImpl, including disallowing cross-location copies.

Throws:

IOException

copy

public void copy(String srcBucketName,
                 List<String> srcObjectNames,
                 String dstBucketName,
                 List<String> dstObjectNames)
          throws IOException

See GoogleCloudStorage.copy(String, List, String, List) for details about expected behavior.

Specified by:

copy in interface GoogleCloudStorage

Parameters:

srcBucketName - name of the bucket containing the objects to copy

srcObjectNames - names of the objects to copy

dstBucketName - name of the bucket to copy to

dstObjectNames - names of the objects after copy

Throws:

FileNotFoundException - if the source object or the destination bucket does not exist

IOException - in all other error cases

copy

public void copy(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap)
          throws IOException

See GoogleCloudStorage.copy(String, List, String, List) for details about expected behavior.

Specified by:

copy in interface GoogleCloudStorage

Parameters:

sourceToDestinationObjectsMap - map of destination objects to be copied, keyed by source

Throws:

FileNotFoundException - if the source object or the destination bucket does not exist

IOException - in all other error cases

listBucketNames

public List<String> listBucketNames()
                             throws IOException

See GoogleCloudStorage.listBucketNames() for details about expected behavior.

Specified by:

listBucketNames in interface GoogleCloudStorage

Throws:

IOException

listBucketInfo

public List<GoogleCloudStorageItemInfo> listBucketInfo()
                                                throws IOException

See GoogleCloudStorage.listBucketInfo() for details about expected behavior.

Specified by:

listBucketInfo in interface GoogleCloudStorage

Throws:

IOException

listObjectInfo

public List<GoogleCloudStorageItemInfo> listObjectInfo(String bucketName,
                                                       String objectNamePrefix,
                                                       ListObjectOptions listOptions)
                                                throws IOException

Description copied from interface: GoogleCloudStorage

Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.

Note: Although GCS does not implement a file system, it treats objects that contain a delimiter (ListObjectOptions.getDelimiter()) as different from other objects when listing objects. This will be clearer with an example.

Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

• With prefix == null and delimiter == /, we get: d1/, o1
• With prefix == null and delimiter == null, we get: o1, d1/, d1/o1, d1/o2

Thus when delimiter is null, the entire key name is considered an opaque string, otherwise only the part up to the first delimiter is considered.

Specified by:

listObjectInfo in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

listOptions - options to use when listing objects

Returns:

list of objects

Throws:

IOException - on IO error

See Also:

GoogleCloudStorage.listObjectInfo(String, String, ListObjectOptions)

listObjectInfoPage

public GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo> listObjectInfoPage(String bucketName,
                                                                                  String objectNamePrefix,
                                                                                  ListObjectOptions listOptions,
                                                                                  String pageToken)
                                                                           throws IOException

Description copied from interface: GoogleCloudStorage

The same semantics as GoogleCloudStorage.listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).

Specified by:

listObjectInfoPage in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

listOptions - options to use when listing objects

pageToken - the page token

Returns:

GoogleCloudStorage.ListPage object with listed GoogleCloudStorageItemInfos and next page token if any

Throws:

IOException - on IO error

See Also:

GoogleCloudStorage.listObjectInfoPage(String, String, ListObjectOptions, String)

createItemInfoForBucket

public static GoogleCloudStorageItemInfo createItemInfoForBucket(StorageResourceId resourceId,
                                                                 com.google.api.services.storage.model.Bucket bucket)

Helper for converting a StorageResourceId + Bucket into a GoogleCloudStorageItemInfo.

createItemInfoForStorageObject

public static GoogleCloudStorageItemInfo createItemInfoForStorageObject(com.google.api.services.storage.model.StorageObject object)

createItemInfoForStorageObject

public static GoogleCloudStorageItemInfo createItemInfoForStorageObject(StorageResourceId resourceId,
                                                                        com.google.api.services.storage.model.StorageObject object)

Helper for converting a StorageResourceId + StorageObject into a GoogleCloudStorageItemInfo.

getItemInfos

public List<GoogleCloudStorageItemInfo> getItemInfos(List<StorageResourceId> resourceIds)
                                              throws IOException

See GoogleCloudStorage.getItemInfos(List) for details about expected behavior.

Specified by:

getItemInfos in interface GoogleCloudStorage

Parameters:

resourceIds - names of the GCS StorageObjects or Buckets for which to retrieve info.

Returns:

information about the given resourceIds.

Throws:

IOException - on IO error

updateItems

public List<GoogleCloudStorageItemInfo> updateItems(List<UpdatableItemInfo> itemInfoList)
                                             throws IOException

Description copied from interface: GoogleCloudStorage

Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Specified by:

updateItems in interface GoogleCloudStorage

Returns:

Updated GoogleCloudStorageItemInfo objects for the referenced objects.

Throws:

IOException - on IO error

getItemInfo

public GoogleCloudStorageItemInfo getItemInfo(StorageResourceId resourceId)
                                       throws IOException

See GoogleCloudStorage.getItemInfo(StorageResourceId) for details about expected behavior.

Specified by:

getItemInfo in interface GoogleCloudStorage

Parameters:

resourceId - identifies either root, a Bucket, or a StorageObject

Returns:

information about the given item

Throws:

IOException - on IO error

close

public void close()

See GoogleCloudStorage.close() for details about expected behavior.

Specified by:

close in interface GoogleCloudStorage

compose

public void compose(String bucketName,
                    List<String> sources,
                    String destination,
                    String contentType)
             throws IOException

Description copied from interface: GoogleCloudStorage

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.

Specified by:

compose in interface GoogleCloudStorage

Parameters:

bucketName - name of the bucket containing the sources and the destination

sources - names of the objects to be composed

destination - name of the resulting object

contentType - content-type of composed object

Throws:

IOException - if the Compose operation was unsuccessful

composeObjects

public GoogleCloudStorageItemInfo composeObjects(List<StorageResourceId> sources,
                                                 StorageResourceId destination,
                                                 CreateObjectOptions options)
                                          throws IOException

Description copied from interface: GoogleCloudStorage

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object will have metadata set according to options. Overwrite semantics for the destination object will follow the same semantics as GoogleCloudStorage.create(StorageResourceId, CreateObjectOptions). See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set. The bucket must be the same for all sources and the destination.

Specified by:

composeObjects in interface GoogleCloudStorage

Throws:

IOException