MergingDigest (t-digest 3.3 API)

java.lang.Object
- com.tdunning.math.stats.TDigest
- - com.tdunning.math.stats.AbstractTDigest
  - - com.tdunning.math.stats.MergingDigest

All Implemented Interfaces:

Serializable
```
public class MergingDigest
extends AbstractTDigest
```
Maintains a t-digest by collecting new points in a buffer that is then sorted occasionally and merged into a sorted array that contains previously computed centroids.
This can be very fast because the cost of sorting and merging is amortized over several insertion. If we keep N centroids total and have the input array is k long, then the amortized cost is something like
N/k + log k
These costs even out when N/k = log k. Balancing costs is often a good place to start in optimizing an algorithm. For different values of compression factor, the following table shows estimated asymptotic values of N and suggested values of k:

Compression N k

50 78 25

100 157 42

200 314 73

Sizing considerations for t-digest

The virtues of this kind of t-digest implementation include:
- No allocation is required after initialization
- The data structure automatically compresses existing centroids when possible
- No Java object overhead is incurred for centroids since data is kept in primitive arrays
The current implementation takes the liberty of using ping-pong buffers for implementing the merge resulting in a substantial memory penalty, but the complexity of an in place merge was not considered as worthwhile since even with the overhead, the memory cost is less than 40 bytes per centroid which is much less than half what the AVLTreeDigest uses and no dynamic allocation is required at all.
See Also:

Serialized Form

Sizing considerations for t-digest
Compression	N	k
50	78	25
100	157	42
200	314	73

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class MergingDigest.Encoding

Nested Classes
Modifier and Type	Class and Description
`static class`	`MergingDigest.Encoding`

Field Summary

Fields
Modifier and Type Field and Description

boolean useAlternatingSort

boolean useTwoLevelCompression

static boolean useWeightLimit
- Fields inherited from class com.tdunning.math.stats.TDigest
  scale

Fields
Modifier and Type	Field and Description
`boolean`	`useAlternatingSort`
`boolean`	`useTwoLevelCompression`
`static boolean`	`useWeightLimit`

Constructor Summary

Constructors
Constructor and Description
`MergingDigest(double compression)` Allocates a buffer merging t-digest.
`MergingDigest(double compression, int bufferSize)` If you know the size of the temporary buffer for incoming points, you can use this entry point.
`MergingDigest(double compression, int bufferSize, int size)` Fully specified constructor.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`add(double x, int w)` Adds a sample to a histogram.
`void`	`add(List<? extends TDigest> others)`
`void`	`asBytes(ByteBuffer buf)` Serialize this TDigest into a byte buffer.
`void`	`asSmallBytes(ByteBuffer buf)` Serialize this TDigest into a byte buffer.
`int`	`byteSize()` Returns the number of bytes required to encode this TDigest using #asBytes().
`double`	`cdf(double x)` Returns the fraction of all points added which are ≤ x.
`int`	`centroidCount()`
`Collection<Centroid>`	`centroids()` A `Collection` that lets you go through the centroids in ascending order by mean.
`void`	`compress()` Merges any pending inputs and compresses the data down to the public setting.
`double`	`compression()` Returns the current compression factor.
`static MergingDigest`	`fromBytes(ByteBuffer buf)`
`ScaleFunction`	`getScaleFunction()`
`double`	`quantile(double q)` Returns an estimate of a cutoff such that a specified fraction of the data added to this TDigest would be less than or equal to the cutoff.
`TDigest`	`recordAllData()` Turns on internal data recording.
`void`	`setScaleFunction(ScaleFunction scaleFunction)`
`long`	`size()` Returns the number of points that have been added to this TDigest.
`int`	`smallByteSize()` Returns the number of bytes required to encode this TDigest using #asSmallBytes().
`String`	`toString()`

Methods inherited from class com.tdunning.math.stats.AbstractTDigest
add, add, createCentroid, isRecording

Methods inherited from class com.tdunning.math.stats.TDigest
createAvlTreeDigest, createDigest, createMergingDigest, getMax, getMin

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - useAlternatingSort
```
public boolean useAlternatingSort
```
  - useTwoLevelCompression
```
public boolean useTwoLevelCompression
```
  - useWeightLimit
```
public static boolean useWeightLimit
```
- Constructor Detail
  - MergingDigest
```
public MergingDigest(double compression)
```
    Allocates a buffer merging t-digest. This is the normally used constructor that allocates default sized internal arrays. Other versions are available, but should only be used for special cases.
    
    Parameters:
    
    compression - The compression factor
  - MergingDigest
```
public MergingDigest(double compression,
                     int bufferSize)
```
    If you know the size of the temporary buffer for incoming points, you can use this entry point.
    
    Parameters:
    
    compression - Compression factor for t-digest. Same as 1/\delta in the paper.
    
    bufferSize - How many samples to retain before merging.
  - MergingDigest
```
public MergingDigest(double compression,
                     int bufferSize,
                     int size)
```
    Fully specified constructor. Normally only used for deserializing a buffer t-digest.
    
    Parameters:
    
    compression - Compression factor
    
    bufferSize - Number of temporary centroids
    
    size - Size of main buffer
- Method Detail
  - recordAllData
```
public TDigest recordAllData()
```
    Turns on internal data recording.
    
    Overrides:
    
    recordAllData in class AbstractTDigest
    
    Returns:
    
    This TDigest so that configurations can be done in fluent style.
  - add
```
public void add(double x,
                int w)
```
    Description copied from class: TDigest
    
    Adds a sample to a histogram.
    
    Specified by:
    
    add in class TDigest
    
    Parameters:
    
    x - The value to add.
    
    w - The weight of this point.
  - add
```
public void add(List<? extends TDigest> others)
```
    Specified by:
    
    add in class TDigest
  - compress
```
public void compress()
```
    Merges any pending inputs and compresses the data down to the public setting. Note that this typically loses a bit of precision and thus isn't a thing to be doing all the time. It is best done only when we want to show results to the outside world.
    
    Specified by:
    
    compress in class TDigest
  - size
```
public long size()
```
    Description copied from class: TDigest
    
    Returns the number of points that have been added to this TDigest.
    
    Specified by:
    
    size in class TDigest
    
    Returns:
    
    The sum of the weights on all centroids.
  - cdf
```
public double cdf(double x)
```
    Description copied from class: TDigest
    
    Returns the fraction of all points added which are ≤ x. Points that are exactly equal get half credit (i.e. we use the mid-point rule)
    
    Specified by:
    
    cdf in class TDigest
    
    Parameters:
    
    x - The cutoff for the cdf.
    
    Returns:
    
    The fraction of all data which is less or equal to x.
  - quantile
```
public double quantile(double q)
```
    Description copied from class: TDigest
    
    Returns an estimate of a cutoff such that a specified fraction of the data added to this TDigest would be less than or equal to the cutoff.
    
    Specified by:
    
    quantile in class TDigest
    
    Parameters:
    
    q - The desired fraction
    
    Returns:
    
    The smallest value x such that cdf(x) ≥ q
  - centroidCount
```
public int centroidCount()
```
    Specified by:
    
    centroidCount in class TDigest
  - centroids
```
public Collection<Centroid> centroids()
```
    Description copied from class: TDigest
    
    A Collection that lets you go through the centroids in ascending order by mean. Centroids returned will not be re-used, but may or may not share storage with this TDigest.
    
    Specified by:
    
    centroids in class TDigest
    
    Returns:
    
    The centroids in the form of a Collection.
  - compression
```
public double compression()
```
    Description copied from class: TDigest
    
    Returns the current compression factor.
    
    Specified by:
    
    compression in class TDigest
    
    Returns:
    
    The compression factor originally used to set up the TDigest.
  - byteSize
```
public int byteSize()
```
    Description copied from class: TDigest
    
    Returns the number of bytes required to encode this TDigest using #asBytes().
    
    Specified by:
    
    byteSize in class TDigest
    
    Returns:
    
    The number of bytes required.
  - smallByteSize
```
public int smallByteSize()
```
    Description copied from class: TDigest
    
    Returns the number of bytes required to encode this TDigest using #asSmallBytes(). Note that this is just as expensive as actually compressing the digest. If you don't care about time, but want to never over-allocate, this is fine. If you care about compression and speed, you pretty much just have to overallocate by using allocating #byteSize() bytes.
    
    Specified by:
    
    smallByteSize in class TDigest
    
    Returns:
    
    The number of bytes required.
  - getScaleFunction
```
public ScaleFunction getScaleFunction()
```
  - setScaleFunction
```
public void setScaleFunction(ScaleFunction scaleFunction)
```
    Overrides:
    
    setScaleFunction in class TDigest
  - asBytes
```
public void asBytes(ByteBuffer buf)
```
    Description copied from class: TDigest
    
    Serialize this TDigest into a byte buffer. Note that the serialization used is very straightforward and is considerably larger than strictly necessary.
    
    Specified by:
    
    asBytes in class TDigest
    
    Parameters:
    
    buf - The byte buffer into which the TDigest should be serialized.
  - asSmallBytes
```
public void asSmallBytes(ByteBuffer buf)
```
    Description copied from class: TDigest
    
    Serialize this TDigest into a byte buffer. Some simple compression is used such as using variable byte representation to store the centroid weights and using delta-encoding on the centroid means so that floats can be reasonably used to store the centroid means.
    
    Specified by:
    
    asSmallBytes in class TDigest
    
    Parameters:
    
    buf - The byte buffer into which the TDigest should be serialized.
  - fromBytes
```
public static MergingDigest fromBytes(ByteBuffer buf)
```
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object

Class MergingDigest

Nested Class Summary

Field Summary

Fields inherited from class com.tdunning.math.stats.TDigest

Constructor Summary

Method Summary

Methods inherited from class com.tdunning.math.stats.AbstractTDigest

Methods inherited from class com.tdunning.math.stats.TDigest

Methods inherited from class java.lang.Object

Field Detail

useAlternatingSort

useTwoLevelCompression

useWeightLimit

Constructor Detail

MergingDigest

MergingDigest

MergingDigest

Method Detail

recordAllData

add

add

compress

size

cdf

quantile

centroidCount

centroids

compression

byteSize

smallByteSize

getScaleFunction

setScaleFunction

asBytes

asSmallBytes

fromBytes

toString