org.apache.arrow.vector

Class BaseFixedWidthVector

java.lang.Object

org.apache.arrow.vector.BaseValueVector

org.apache.arrow.vector.BaseFixedWidthVector

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<ValueVector>, FieldVector, FixedWidthVector, ValueVector, VectorDefinitionSetter

Direct Known Subclasses:

BigIntVector, BitVector, DateDayVector, DateMilliVector, DecimalVector, FixedSizeBinaryVector, Float4Vector, Float8Vector, IntervalDayVector, IntervalYearVector, IntVector, SmallIntVector, TimeMicroVector, TimeMilliVector, TimeNanoVector, TimeSecVector, TimeStampVector, TinyIntVector, UInt1Vector, UInt2Vector, UInt4Vector, UInt8Vector

public abstract class BaseFixedWidthVector
extends BaseValueVector
implements FixedWidthVector, FieldVector, VectorDefinitionSetter

BaseFixedWidthVector provides an abstract interface for implementing vectors of fixed width values. The vectors are nullable implying that zero or more elements in the vector could be NULL.

Field Summary

Fields

Modifier and Type	Field and Description
protected Field	field
protected int	validityAllocationSizeInBytes
protected ArrowBuf	validityBuffer
protected int	valueAllocationSizeInBytes
protected ArrowBuf	valueBuffer
protected int	valueCount

Fields inherited from class org.apache.arrow.vector.BaseValueVector

allocator, INITIAL_VALUE_ALLOCATION, MAX_ALLOCATION_SIZE, MAX_ALLOCATION_SIZE_PROPERTY, name

Constructor Summary

Constructors

Constructor and Description
BaseFixedWidthVector(String name, BufferAllocator allocator, FieldType fieldType, int typeWidth)

Method Summary

Modifier and Type	Method and Description
void	allocateNew() Same as allocateNewSafe().
void	allocateNew(int valueCount) Allocate memory for the vector to support storing at least the provided number of elements in the vector.
boolean	allocateNewSafe() Allocate memory for the vector.
void	clear() Same as close()
void	close() Close the vector and release the associated buffers.
protected void	decrementAllocationMonitor()
ArrowBuf[]	getBuffers(boolean clear) Return the underlying buffers associated with this vector.
int	getBufferSize() Get the size (number of bytes) of underlying buffers used by this vector
int	getBufferSizeFor(int count) Get the potential buffer size for a particular number of records.
List<FieldVector>	getChildrenFromFields() Get the inner child vectors.
ArrowBuf	getDataBuffer() Get the buffer that stores the data for elements in the vector.
long	getDataBufferAddress() Get the memory address of buffer that stores the data for elements in the vector.
Field	getField() Get information about how this field is materialized.
List<ArrowBuf>	getFieldBuffers() Get the buffers belonging to this vector
List<BufferBacked>	getFieldInnerVectors() Deprecated.
int	getNullCount() Get the number of elements that are null in the vector
ArrowBuf	getOffsetBuffer() buffer that stores the offsets for elements in the vector.
long	getOffsetBufferAddress() Get the memory address of buffer that stores the offsets for elements in the vector.
TransferPair	getTransferPair(BufferAllocator allocator) Construct a transfer pair of this vector and another vector of same type.
abstract TransferPair	getTransferPair(String ref, BufferAllocator allocator) Construct a transfer pair of this vector and another vector of same type.
TransferPair	getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) Construct a transfer pair of this vector and another vector of same type.
ArrowBuf	getValidityBuffer() Get buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.
long	getValidityBufferAddress() Get the memory address of buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.
int	getValueCapacity() Get the current value capacity for the vector
int	getValueCount() Get the value count of vector.
protected void	handleSafe(int index) * helper methods for setters * *
protected void	incrementAllocationMonitor()
void	initializeChildrenFromFields(List<Field> children) Initialize the children in schema for this Field.
boolean	isNull(int index) Check if element at given index is null.
boolean	isSafe(int index) Check if the given index is within the current value capacity of the vector
int	isSet(int index) Same as isNull(int).
void	loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers) Load the buffers of this vector with provided source buffers.
void	reAlloc() Resize the vector to increase the capacity.
void	reset() Reset the vector to initial state.
void	set(int index, byte[] value, int start, int length)
void	set(int index, ByteBuffer value, int start, int length)
void	setIndexDefined(int index) Mark the particular position in the vector as non-null.
void	setInitialCapacity(int valueCount) Sets the desired value capacity for the vector.
void	setSafe(int index, byte[] value, int start, int length)
void	setSafe(int index, ByteBuffer value, int start, int length)
void	setValueCount(int valueCount) Set value count for the vector.
void	splitAndTransferTo(int startIndex, int length, BaseFixedWidthVector target) Slice this vector at desired index and length and transfer the corresponding data to the target vector.
void	transferTo(BaseFixedWidthVector target) Transfer this vector'data to another vector.
void	zeroVector() zero out the vector and the data in associated buffers.

Methods inherited from class org.apache.arrow.vector.BaseValueVector

checkBufRefs, compareTypes, getAllocator, getValidityBufferSizeFromCount, iterator, releaseBuffer, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.arrow.vector.ValueVector

getAllocator, getMinorType, getObject, getReader, makeTransferPair

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Field Detail

valueAllocationSizeInBytes

protected int valueAllocationSizeInBytes

validityAllocationSizeInBytes

protected int validityAllocationSizeInBytes

field

protected final Field field

validityBuffer

protected ArrowBuf validityBuffer

valueBuffer

protected ArrowBuf valueBuffer

valueCount

protected int valueCount

Constructor Detail

BaseFixedWidthVector

public BaseFixedWidthVector(String name,
                            BufferAllocator allocator,
                            FieldType fieldType,
                            int typeWidth)

Method Detail

getValidityBufferAddress

public long getValidityBufferAddress()

Get the memory address of buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.

Specified by:

getValidityBufferAddress in interface FieldVector

Returns:

starting address of the buffer

getDataBufferAddress

public long getDataBufferAddress()

Get the memory address of buffer that stores the data for elements in the vector.

Specified by:

getDataBufferAddress in interface FieldVector

Returns:

starting address of the buffer

getOffsetBufferAddress

public long getOffsetBufferAddress()

Get the memory address of buffer that stores the offsets for elements in the vector. This operation is not supported for fixed-width vectors.

Specified by:

getOffsetBufferAddress in interface FieldVector

Returns:

starting address of the buffer

Throws:

UnsupportedOperationException - for fixed width vectors

getValidityBuffer

public ArrowBuf getValidityBuffer()

Get buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector. Consider it as a buffer for internal bit vector data structure.

Specified by:

getValidityBuffer in interface ValueVector

Returns:

buffer

getDataBuffer

public ArrowBuf getDataBuffer()

Get the buffer that stores the data for elements in the vector.

Specified by:

getDataBuffer in interface ValueVector

Returns:

buffer

getOffsetBuffer

public ArrowBuf getOffsetBuffer()

buffer that stores the offsets for elements in the vector. This operation is not supported for fixed-width vectors.

Specified by:

getOffsetBuffer in interface ValueVector

Returns:

buffer

Throws:

UnsupportedOperationException - for fixed width vectors

setInitialCapacity

public void setInitialCapacity(int valueCount)

Sets the desired value capacity for the vector. This function doesn't allocate any memory for the vector.

Specified by:

setInitialCapacity in interface ValueVector

Parameters:

valueCount - desired number of elements in the vector

getValueCapacity

public int getValueCapacity()

Get the current value capacity for the vector

Specified by:

getValueCapacity in interface ValueVector

Returns:

number of elements that vector can hold.

zeroVector

public void zeroVector()

zero out the vector and the data in associated buffers.

Specified by:

zeroVector in interface FixedWidthVector

reset

public void reset()

Reset the vector to initial state. Same as zeroVector(). Note that this method doesn't release any memory.

Specified by:

reset in interface ValueVector

close

public void close()

Close the vector and release the associated buffers.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ValueVector

Overrides:

close in class BaseValueVector

clear

public void clear()

Same as close()

Specified by:

clear in interface ValueVector

Overrides:

clear in class BaseValueVector

incrementAllocationMonitor

protected void incrementAllocationMonitor()

decrementAllocationMonitor

protected void decrementAllocationMonitor()

allocateNew

public void allocateNew()

Same as allocateNewSafe().

Specified by:

allocateNew in interface ValueVector

allocateNewSafe

public boolean allocateNewSafe()

Allocate memory for the vector. We internally use a default value count of 4096 to allocate memory for at least these many elements in the vector. See allocateNew(int) for allocating memory for specific number of elements in the vector.

Specified by:

allocateNewSafe in interface ValueVector

Returns:

false if memory allocation fails, true otherwise.

allocateNew

public void allocateNew(int valueCount)

Allocate memory for the vector to support storing at least the provided number of elements in the vector. This method must be called prior to using the ValueVector.

Specified by:

allocateNew in interface FixedWidthVector

Parameters:

valueCount - the desired number of elements in the vector

Throws:

OutOfMemoryException

getBufferSizeFor

public int getBufferSizeFor(int count)

Get the potential buffer size for a particular number of records.

Specified by:

getBufferSizeFor in interface ValueVector

Parameters:

count - desired number of elements in the vector

Returns:

estimated size of underlying buffers if the vector holds a given number of elements

getBufferSize

public int getBufferSize()

Get the size (number of bytes) of underlying buffers used by this vector

Specified by:

getBufferSize in interface ValueVector

Returns:

size of underlying buffers.

getField

public Field getField()

Get information about how this field is materialized.

Specified by:

getField in interface ValueVector

Returns:

the field corresponding to this vector

getBuffers

public ArrowBuf[] getBuffers(boolean clear)

Return the underlying buffers associated with this vector. Note that this doesn't impact the reference counts for this buffer so it only should be used for in-context access. Also note that this buffer changes regularly thus external classes shouldn't hold a reference to it (unless they change it).

Specified by:

getBuffers in interface ValueVector

Parameters:

clear - Whether to clear vector before returning; the buffers will still be refcounted but the returned array will be the only reference to them

Returns:

The underlying buffers that is used by this vector instance.

reAlloc

public void reAlloc()

Resize the vector to increase the capacity. The internal behavior is to double the current value capacity.

Specified by:

reAlloc in interface ValueVector

getFieldInnerVectors

@Deprecated
public List<BufferBacked> getFieldInnerVectors()

Deprecated.

Specified by:

getFieldInnerVectors in interface FieldVector

Returns:

the inner vectors for this field as defined by the TypeLayout

initializeChildrenFromFields

public void initializeChildrenFromFields(List<Field> children)

Initialize the children in schema for this Field. This operation is a NO-OP for scalar types since they don't have any children.

Specified by:

initializeChildrenFromFields in interface FieldVector

Parameters:

children - the schema

Throws:

IllegalArgumentException - if children is a non-empty list for scalar types.

getChildrenFromFields

public List<FieldVector> getChildrenFromFields()

Get the inner child vectors.

Specified by:

getChildrenFromFields in interface FieldVector

Returns:

list of child vectors for complex types, empty list for scalar vector types

loadFieldBuffers

public void loadFieldBuffers(ArrowFieldNode fieldNode,
                             List<ArrowBuf> ownBuffers)

Load the buffers of this vector with provided source buffers. The caller manages the source buffers and populates them before invoking this method.

Specified by:

loadFieldBuffers in interface FieldVector

Parameters:

fieldNode - the fieldNode indicating the value count

ownBuffers - the buffers for this Field (own buffers only, children not included)

getFieldBuffers

public List<ArrowBuf> getFieldBuffers()

Get the buffers belonging to this vector

Specified by:

getFieldBuffers in interface FieldVector

Returns:

the inner buffers.

getTransferPair

public TransferPair getTransferPair(String ref,
                                    BufferAllocator allocator,
                                    CallBack callBack)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

ref - name of the target vector

allocator - allocator for the target vector

callBack -

Returns:

TransferPair

getTransferPair

public TransferPair getTransferPair(BufferAllocator allocator)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Overrides:

getTransferPair in class BaseValueVector

Parameters:

allocator - allocator for the target vector

Returns:

TransferPair

getTransferPair

public abstract TransferPair getTransferPair(String ref,
                                             BufferAllocator allocator)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

ref - name of the target vector

allocator - allocator for the target vector

Returns:

TransferPair

transferTo

public void transferTo(BaseFixedWidthVector target)

Transfer this vector'data to another vector. The memory associated with this vector is transferred to the allocator of target vector for accounting and management purposes.

Parameters:

target - destination vector for transfer

splitAndTransferTo

public void splitAndTransferTo(int startIndex,
                               int length,
                               BaseFixedWidthVector target)

Slice this vector at desired index and length and transfer the corresponding data to the target vector.

Parameters:

startIndex - start position of the split in source vector.

length - length of the split.

target - destination vector

getNullCount

public int getNullCount()

Get the number of elements that are null in the vector

Specified by:

getNullCount in interface ValueVector

Returns:

the number of null elements.

getValueCount

public int getValueCount()

Get the value count of vector. This will always be zero unless setValueCount(int) has been called prior to calling this.

Specified by:

getValueCount in interface ValueVector

Returns:

valueCount for the vector

setValueCount

public void setValueCount(int valueCount)

Set value count for the vector.

Specified by:

setValueCount in interface ValueVector

Parameters:

valueCount - value count to set

isSafe

public boolean isSafe(int index)

Check if the given index is within the current value capacity of the vector

Parameters:

index - position to check

Returns:

true if index is within the current value capacity

isNull

public boolean isNull(int index)

Check if element at given index is null.

Specified by:

isNull in interface ValueVector

Parameters:

index - position of element

Returns:

true if element at given index is null, false otherwise

isSet

public int isSet(int index)

Same as isNull(int).

Parameters:

index - position of element

Returns:

1 if element at given index is not null, 0 otherwise

setIndexDefined

public void setIndexDefined(int index)

Mark the particular position in the vector as non-null.

Specified by:

setIndexDefined in interface VectorDefinitionSetter

Parameters:

index - position of the element.

set

public void set(int index,
                byte[] value,
                int start,
                int length)

setSafe

public void setSafe(int index,
                    byte[] value,
                    int start,
                    int length)

set

public void set(int index,
                ByteBuffer value,
                int start,
                int length)

setSafe

public void setSafe(int index,
                    ByteBuffer value,
                    int start,
                    int length)

handleSafe

protected void handleSafe(int index)

* helper methods for setters * *