org.yamcs.yarch

Class PartitionManager

java.lang.Object

org.yamcs.yarch.PartitionManager

Direct Known Subclasses:

RdbPartitionManager, RdbPartitionManager

public abstract class PartitionManager
extends Object

Keeps track of partitions and histograms for one table. The partitioning is organised in a set of time based partitioning intervals (one day/month/year long) Each interval has associated a set of value based partitions. In addition each interval has a set of histograms - one for each table column for which histograms have been created. In the RocksDB implementation (both old and new) each interval corresponds to one rocksdb database directory. In case there is no time based partitioning, there is only one interval. In case there is no value based partitioning, there is only one partition in each interval.

Author:

nm

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PartitionManager.Interval Keeps a value -> partition map for a specific time interval

Field Summary

Fields

Modifier and Type	Field and Description
protected PartitionedTimeInterval<PartitionManager.Interval>	intervals
protected PartitioningSpec	partitioningSpec
protected PartitionManager.Interval	pcache
protected TableDefinition	tableDefinition

Constructor Summary

Constructors

Constructor and Description
PartitionManager(TableDefinition tableDefinition)

Method Summary

Modifier and Type	Method and Description
HistogramInfo	createAndGetHistogram(long instant, String columnName)
Partition	createAndGetPartition(long instant, Object value) Creates (if not already existing) and returns the partition in which the instant,value should be written.
protected abstract HistogramInfo	createHistogram(String columnName)
protected abstract HistogramInfo	createHistogramByTime(TimePartitionInfo pinfo, String columnName)
protected abstract Partition	createPartition(Object value) Create a partition for value based partitioning
protected abstract Partition	createPartitionByTime(TimePartitionInfo pinfo, Object value) Create a partition for time (and possible value) based partitioning
Partition	getPartitionForTuple(Tuple t) Gets partition where tuple has to be written.
List<Partition>	getPartitions() Retrieves the existing partitions
Iterator<PartitionManager.Interval>	intervalIterator(TimeInterval timeInterval) Iterates over all intervals overlapping with the timeInterval.
Iterator<List<Partition>>	iterator(long start, Set<Object> partitionValueFilter) See iterator(Set)
Iterator<List<Partition>>	iterator(Set<Object> partitionValueFilter) Returns an iterator which at each step gives the list of partition corresponding to a time interval (so when we do a replay those partitions have to be played in parallel).
Iterator<List<Partition>>	reverseIterator(long start, Set<Object> partitionValueFilter)
Iterator<List<Partition>>	reverseIterator(Set<Object> partitionValueFilter) same as above, only in reverse direction

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tableDefinition

protected final TableDefinition tableDefinition

partitioningSpec

protected final PartitioningSpec partitioningSpec

intervals

protected PartitionedTimeInterval<PartitionManager.Interval> intervals

pcache

protected PartitionManager.Interval pcache

Constructor Detail

PartitionManager

public PartitionManager(TableDefinition tableDefinition)

Method Detail

iterator

public Iterator<List<Partition>> iterator(Set<Object> partitionValueFilter)

Returns an iterator which at each step gives the list of partition corresponding to a time interval (so when we do a replay those partitions have to be played in parallel). The iterator returns intervals sorted on time.

Parameters:

partitionValueFilter - - return only partitions whose value are in the filter. If null, return all partitions;

Returns:

iterator going over partitions

reverseIterator

public Iterator<List<Partition>> reverseIterator(Set<Object> partitionValueFilter)

same as above, only in reverse direction

Parameters:

partitionValueFilter -

Returns:

iterator

public Iterator<List<Partition>> iterator(long start,
                                          Set<Object> partitionValueFilter)

See iterator(Set)

Parameters:

start -

partitionValueFilter - values - return only partitions whose value are in the filter. If null, return all partitions;

Returns:

an iterator over the partitions starting at the specified start time

intervalIterator

public Iterator<PartitionManager.Interval> intervalIterator(TimeInterval timeInterval)

Iterates over all intervals overlapping with the timeInterval. Note that the timeInterval is considered closed at both ends (if set): [timeInterval.start, timeInterval.stop] whereas the partition intervals are considered closed at start and open at stop: [Interval.start, Interval.stop)

reverseIterator

public Iterator<List<Partition>> reverseIterator(long start,
                                                 Set<Object> partitionValueFilter)

createAndGetPartition

public Partition createAndGetPartition(long instant,
                                       Object value)
                                throws IOException

Creates (if not already existing) and returns the partition in which the instant,value should be written. value can be null (in case of no value partitioning)

Parameters:

instant - - time for which the partition has to be created - can be TimeEncoding.INVALID in case value only or no partitioning

value - - value for which the partition has to be created - can be null in case of time only or no partitioning. For the enum partitions, the value is the index (type Short) rather than the string.

Returns:

a Partition

Throws:

IOException

createAndGetHistogram

public HistogramInfo createAndGetHistogram(long instant,
                                           String columnName)
                                    throws IOException

Throws:

IOException

getPartitionForTuple

public Partition getPartitionForTuple(Tuple t)
                               throws IOException

Gets partition where tuple has to be written. Creates the partition if necessary.

Parameters:

t -

Returns:

the partition where the tuple has to be written

Throws:

IOException

createPartitionByTime

protected abstract Partition createPartitionByTime(TimePartitionInfo pinfo,
                                                   Object value)
                                            throws IOException

Create a partition for time (and possible value) based partitioning

Parameters:

pinfo -

value -

Returns:

Throws:

IOException

createPartition

protected abstract Partition createPartition(Object value)
                                      throws IOException

Create a partition for value based partitioning

Parameters:

value -

Returns:

Throws:

IOException

createHistogramByTime

protected abstract HistogramInfo createHistogramByTime(TimePartitionInfo pinfo,
                                                       String columnName)
                                                throws IOException

Throws:

IOException

createHistogram

protected abstract HistogramInfo createHistogram(String columnName)
                                          throws IOException

Throws:

IOException

getPartitions

public List<Partition> getPartitions()

Retrieves the existing partitions

Returns:

list of all existing partitions