Package org.apache.parquet.hadoop.codec

Class Lz4RawCodec

java.lang.Object

org.apache.parquet.hadoop.codec.Lz4RawCodec

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.io.compress.CompressionCodec

public class Lz4RawCodec
extends Object
implements org.apache.hadoop.conf.Configurable, org.apache.hadoop.io.compress.CompressionCodec

Lz4 raw compression codec for Parquet. This codec type has been introduced into the parquet format since version 2.9.0. It differs from the Lz4Codec shipped with Apache Hadoop by removing the light frame header which includes 4 byte for uncompressed length and 4 byte for compressed length. In the Apache Arrow implementation, these two Lz4 codecs are recognized as LZ4_RAW and LZ4_HADOOP to minimize the confusion at its best. Please check the link below for reference. https://github.com/apache/parquet-format/blob/master/Compression.md

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.io.compress.CompressionCodec

org.apache.hadoop.io.compress.CompressionCodec.Util

Field Summary

Fields

Modifier and Type	Field	Description
static String	BUFFER_SIZE_CONFIG

Constructor Summary

Constructors

Constructor	Description
Lz4RawCodec()

Method Summary

Modifier and Type	Method	Description
org.apache.hadoop.io.compress.Compressor	createCompressor()
org.apache.hadoop.io.compress.Decompressor	createDecompressor()
org.apache.hadoop.io.compress.CompressionInputStream	createInputStream(InputStream stream)
org.apache.hadoop.io.compress.CompressionInputStream	createInputStream(InputStream stream, org.apache.hadoop.io.compress.Decompressor decompressor)
org.apache.hadoop.io.compress.CompressionOutputStream	createOutputStream(OutputStream stream)
org.apache.hadoop.io.compress.CompressionOutputStream	createOutputStream(OutputStream stream, org.apache.hadoop.io.compress.Compressor compressor)
Class<? extends org.apache.hadoop.io.compress.Compressor>	getCompressorType()
org.apache.hadoop.conf.Configuration	getConf()
Class<? extends org.apache.hadoop.io.compress.Decompressor>	getDecompressorType()
String	getDefaultExtension()
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BUFFER_SIZE_CONFIG

public static final String BUFFER_SIZE_CONFIG

See Also:

Constant Field Values

Constructor Detail

Lz4RawCodec

public Lz4RawCodec()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

createCompressor

public org.apache.hadoop.io.compress.Compressor createCompressor()

Specified by:

createCompressor in interface org.apache.hadoop.io.compress.CompressionCodec

createDecompressor

public org.apache.hadoop.io.compress.Decompressor createDecompressor()

Specified by:

createDecompressor in interface org.apache.hadoop.io.compress.CompressionCodec

createInputStream

public org.apache.hadoop.io.compress.CompressionInputStream createInputStream(InputStream stream)
                                                                       throws IOException

Specified by:

createInputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

createInputStream

public org.apache.hadoop.io.compress.CompressionInputStream createInputStream(InputStream stream,
                                                                              org.apache.hadoop.io.compress.Decompressor decompressor)
                                                                       throws IOException

Specified by:

createInputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

createOutputStream

public org.apache.hadoop.io.compress.CompressionOutputStream createOutputStream(OutputStream stream)
                                                                         throws IOException

Specified by:

createOutputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

createOutputStream

public org.apache.hadoop.io.compress.CompressionOutputStream createOutputStream(OutputStream stream,
                                                                                org.apache.hadoop.io.compress.Compressor compressor)
                                                                         throws IOException

Specified by:

createOutputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

getCompressorType

public Class<? extends org.apache.hadoop.io.compress.Compressor> getCompressorType()

Specified by:

getCompressorType in interface org.apache.hadoop.io.compress.CompressionCodec

getDecompressorType

public Class<? extends org.apache.hadoop.io.compress.Decompressor> getDecompressorType()

Specified by:

getDecompressorType in interface org.apache.hadoop.io.compress.CompressionCodec

getDefaultExtension

public String getDefaultExtension()

Specified by:

getDefaultExtension in interface org.apache.hadoop.io.compress.CompressionCodec