com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem

All Implemented Interfaces:

FileSystemDescriptor, Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

public class GoogleHadoopFileSystem
extends GoogleHadoopFileSystemBase

GoogleHadoopFileSystem is a version of GoogleHadoopFileSystemBase which is rooted in a single bucket at initialization time; in this case, Hadoop paths no longer correspond directly to general GCS paths, and all Hadoop operations going through this FileSystem will never touch any GCS bucket other than the bucket on which this FileSystem is rooted.

This implementation sacrifices a small amount of cross-bucket interoperability in favor of more straightforward FileSystem semantics and compatibility with existing Hadoop applications. In particular, it is not subject to bucket-naming constraints, and files are allowed to be placed in root.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

GoogleHadoopFileSystemBase.GcsFileChecksumType, GoogleHadoopFileSystemBase.GlobAlgorithm, GoogleHadoopFileSystemBase.InvocationRaisingIOE<R>, GoogleHadoopFileSystemBase.OutputStreamType

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	SCHEME URI scheme for GoogleHadoopFileSystem

Fields inherited from class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

DEFAULT_FILTER, defaultBlockSize, delegationTokens, GHFS_ID, initUri, PROPERTIES_FILE, REPLICATION_FACTOR_DEFAULT, UNKNOWN_VERSION, VERSION, VERSION_PROPERTY

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopFileSystem() Constructs an instance of GoogleHadoopFileSystem; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

Method Summary

Modifier and Type	Method and Description
protected void	checkPath(org.apache.hadoop.fs.Path path)
protected void	configureBuckets(GoogleCloudStorageFileSystem gcsFs) Sets and validates the root bucket.
org.apache.hadoop.fs.Path	getDefaultWorkingDirectory() Gets the default value of working directory.
org.apache.hadoop.fs.Path	getFileSystemRoot() Returns the Hadoop path representing the root of the FileSystem associated with this FileSystemDescriptor.
URI	getGcsPath(org.apache.hadoop.fs.Path hadoopPath) Translates a "gs:/" style hadoopPath (or relative path which is not fully-qualified) into the appropriate GCS path which is compatible with the underlying GcsFs or gsutil.
org.apache.hadoop.fs.Path	getHadoopPath(URI gcsPath) Validates GCS Path belongs to this file system.
protected String	getHomeDirectorySubpath() Override to allow a homedir subpath which sits directly on our FileSystem root.
String	getScheme() As the global-rooted FileSystem, our hadoop-path "scheme" is exactly equal to the general GCS scheme.

Methods inherited from class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

append, close, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, create, createNonRecursive, delete, deleteOnExit, getCanonicalServiceName, getContentSummary, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDelegationToken, getFileChecksum, getFileStatus, getGcsFs, getHomeDirectory, getStorageStatistics, getUri, getUsed, getWorkingDirectory, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, hasPathCapability, initialize, listStatus, listXAttrs, makeQualified, mkdirs, open, processDeleteOnExit, removeXAttr, rename, setOwner, setPermission, setTimes, setVerifyChecksum, setWorkingDirectory, setXAttr, startLocalOutput

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUsed, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setWriteChecksum, setXAttr, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, collectDelegationTokens

Field Detail

SCHEME

public static final String SCHEME

URI scheme for GoogleHadoopFileSystem

See Also:

Constant Field Values

Constructor Detail

GoogleHadoopFileSystem

public GoogleHadoopFileSystem()

Constructs an instance of GoogleHadoopFileSystem; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

Method Detail

configureBuckets

protected void configureBuckets(GoogleCloudStorageFileSystem gcsFs)
                         throws IOException

Sets and validates the root bucket.

Specified by:

configureBuckets in class GoogleHadoopFileSystemBase

Parameters:

gcsFs - GoogleCloudStorageFileSystem to configure buckets

Throws:

IOException - if bucket name is invalid or cannot be found.

checkPath

protected void checkPath(org.apache.hadoop.fs.Path path)

Overrides:

checkPath in class GoogleHadoopFileSystemBase

getHomeDirectorySubpath

protected String getHomeDirectorySubpath()

Override to allow a homedir subpath which sits directly on our FileSystem root.

Specified by:

getHomeDirectorySubpath in class GoogleHadoopFileSystemBase

getHadoopPath

public org.apache.hadoop.fs.Path getHadoopPath(URI gcsPath)

Validates GCS Path belongs to this file system. The bucket must match the root bucket provided at initialization time.

Specified by:

getHadoopPath in class GoogleHadoopFileSystemBase

Parameters:

gcsPath - Fully-qualified GCS path, of the form gs://bucket/object-path.

getGcsPath

public URI getGcsPath(org.apache.hadoop.fs.Path hadoopPath)

Translates a "gs:/" style hadoopPath (or relative path which is not fully-qualified) into the appropriate GCS path which is compatible with the underlying GcsFs or gsutil.

Specified by:

getGcsPath in class GoogleHadoopFileSystemBase

Parameters:

hadoopPath - Hadoop path.

getScheme

public String getScheme()

As the global-rooted FileSystem, our hadoop-path "scheme" is exactly equal to the general GCS scheme.

Specified by:

getScheme in interface FileSystemDescriptor

Specified by:

getScheme in class GoogleHadoopFileSystemBase

getFileSystemRoot

public org.apache.hadoop.fs.Path getFileSystemRoot()

Description copied from interface: FileSystemDescriptor

Returns the Hadoop path representing the root of the FileSystem associated with this FileSystemDescriptor.

Specified by:

getFileSystemRoot in interface FileSystemDescriptor

Specified by:

getFileSystemRoot in class GoogleHadoopFileSystemBase

getDefaultWorkingDirectory

public org.apache.hadoop.fs.Path getDefaultWorkingDirectory()

Gets the default value of working directory.

Specified by:

getDefaultWorkingDirectory in class GoogleHadoopFileSystemBase