Package org.apache.lucene.codecs.blocktree

Class BlockTreeTermsWriter

java.lang.Object

org.apache.lucene.codecs.FieldsConsumer

org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter

All Implemented Interfaces:

Closeable, AutoCloseable

public final class BlockTreeTermsWriter
extends FieldsConsumer

Block-based terms index and dictionary writer.

Writes terms dict and index, block-encoding (column stride) each term's metadata for each set of terms between two index terms.

Files:

• .tim: Term Dictionary
• .tip: Term Index

Term Dictionary

The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and per-term metadata (typically pointers to the postings list for that term in the inverted index).

The .tim is arranged in blocks: with blocks containing a variable number of entries (by default 25-48), where each entry is either a term or a reference to a sub-block.

NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the Postings Metadata and Term Metadata sections.

• TermsDict (.tim) --> Header, PostingsHeader, NodeBlock^NumBlocks, FieldSummary, DirOffset, Footer
• NodeBlock --> (OuterNode | InnerNode)
• OuterNode --> EntryCount, SuffixLength, Byte^SuffixLength, StatsLength, < TermStats >^EntryCount, MetaLength, <TermMetadata>^EntryCount
• InnerNode --> EntryCount, SuffixLength[,Sub?], Byte^SuffixLength, StatsLength, < TermStats ? >^EntryCount, MetaLength, <TermMetadata ? >^EntryCount
• TermStats --> DocFreq, TotalTermFreq
• FieldSummary --> NumFields, <FieldNumber, NumTerms, RootCodeLength, Byte^{RootCodeLength}, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, MinTerm, MaxTerm>^NumFields
• Header --> CodecHeader
• DirOffset --> Uint64
• MinTerm,MaxTerm --> VInt length followed by the byte[]
• EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength,NumFields, FieldNumber,RootCodeLength,DocCount,LongsSize --> VInt
• TotalTermFreq,NumTerms,SumTotalTermFreq,SumDocFreq --> VLong
• Footer --> CodecFooter

Notes:

• Header is a CodecHeader storing the version information for the BlockTree implementation.
• DirOffset is a pointer to the FieldSummary section.
• DocFreq is the count of documents which contain the term.
• TotalTermFreq is the total number of occurrences of the term. This is encoded as the difference between the total number of occurrences and the DocFreq.
• FieldNumber is the fields number from FieldInfos. (.fnm)
• NumTerms is the number of unique terms for the field.
• RootCode points to the root block for the field.
• SumDocFreq is the total number of postings, the number of term-document pairs across the entire field.
• DocCount is the number of documents that have at least one posting for this field.
• LongsSize records how many long values the postings writer/reader record per term (e.g., to hold freq/prox/doc file offsets).
• MinTerm, MaxTerm are the lowest and highest term in this field.
• PostingsHeader and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files).
• For inner nodes of the tree, every entry will steal one bit to mark whether it points to child nodes(sub-block). If so, the corresponding TermStats and TermMetaData are omitted

Term Index

The .tip file contains an index into the term dictionary, so that it can be accessed randomly. The index is also used to determine when a given term cannot exist on disk (in the .tim file), saving a disk seek.

• TermsIndex (.tip) --> Header, FSTIndex^NumFields <IndexStartFP>^NumFields, DirOffset, Footer
• Header --> CodecHeader
• DirOffset --> Uint64
• IndexStartFP --> VLong
• FSTIndex --> FST<byte[]>
• Footer --> CodecFooter

Notes:

• The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST.
• DirOffset is a pointer to the start of the IndexStartFPs for all fields
• It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.

See Also:

• BlockTreeTermsReader

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static final class	BlockTreeTermsWriter.PendingBlock
private static class	BlockTreeTermsWriter.PendingEntry
private static final class	BlockTreeTermsWriter.PendingTerm
private static class	BlockTreeTermsWriter.StatsWriter
(package private) class	BlockTreeTermsWriter.TermsWriter

Field Summary

Fields

Modifier and Type	Field	Description
private boolean	closed
static final int	DEFAULT_MAX_BLOCK_SIZE	Suggested default value for the maxItemsInBlock parameter to BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int).
static final int	DEFAULT_MIN_BLOCK_SIZE	Suggested default value for the minItemsInBlock parameter to BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int).
(package private) static final BytesRef	EMPTY_BYTES_REF
(package private) final FieldInfos	fieldInfos
private final List<ByteBuffersDataOutput>	fields
private final IndexOutput	indexOut
(package private) final int	maxDoc
(package private) final int	maxItemsInBlock
private final IndexOutput	metaOut
(package private) final int	minItemsInBlock
(package private) final PostingsWriterBase	postingsWriter
private final RAMOutputStream	scratchBytes
private final IntsRefBuilder	scratchIntsRef
private final IndexOutput	termsOut

Constructor Summary

Constructors

Constructor	Description
BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock)	Create a new writer.

Method Summary

Modifier and Type	Method	Description
(package private) static String	brToString(byte[] b)
(package private) static String	brToString(BytesRef b)
void	close()
(package private) static long	encodeOutput(long fp, boolean hasTerms, boolean isFloor)
static void	validateSettings(int minItemsInBlock, int maxItemsInBlock)	Throws IllegalArgumentException if any of these settings is invalid.
void	write(Fields fields, NormsProducer norms)	Write all fields, terms and postings.
private static void	writeBytesRef(DataOutput out, BytesRef bytes)

Methods inherited from class org.apache.lucene.codecs.FieldsConsumer

merge

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_MIN_BLOCK_SIZE

public static final int DEFAULT_MIN_BLOCK_SIZE

Suggested default value for the minItemsInBlock parameter to BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int).

See Also:

• Constant Field Values

DEFAULT_MAX_BLOCK_SIZE

public static final int DEFAULT_MAX_BLOCK_SIZE

Suggested default value for the maxItemsInBlock parameter to BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int).

See Also:

• Constant Field Values

metaOut

private final IndexOutput metaOut

termsOut

private final IndexOutput termsOut

indexOut

private final IndexOutput indexOut

maxDoc

final int maxDoc

minItemsInBlock

final int minItemsInBlock

maxItemsInBlock

final int maxItemsInBlock

postingsWriter

final PostingsWriterBase postingsWriter

fieldInfos

final FieldInfos fieldInfos

fields

private final List<ByteBuffersDataOutput> fields

scratchBytes

private final RAMOutputStream scratchBytes

scratchIntsRef

private final IntsRefBuilder scratchIntsRef

EMPTY_BYTES_REF

static final BytesRef EMPTY_BYTES_REF

closed

private boolean closed

Constructor Details

BlockTreeTermsWriter

public BlockTreeTermsWriter(SegmentWriteState state,
 PostingsWriterBase postingsWriter,
 int minItemsInBlock,
 int maxItemsInBlock)
                     throws IOException

Create a new writer. The number of items (terms or sub-blocks) per block will aim to be between minItemsPerBlock and maxItemsPerBlock, though in some cases the blocks may be smaller than the min.

Throws:

IOException

Method Details

validateSettings

public static void validateSettings(int minItemsInBlock,
 int maxItemsInBlock)

Throws IllegalArgumentException if any of these settings is invalid.

write

public void write(Fields fields,
 NormsProducer norms)
           throws IOException

Description copied from class: FieldsConsumer

Write all fields, terms and postings. This the "pull" API, allowing you to iterate more than once over the postings, somewhat analogous to using a DOM API to traverse an XML tree.

Notes:

• You must compute index statistics, including each Term's docFreq and totalTermFreq, as well as the summary sumTotalTermFreq, sumTotalDocFreq and docCount.
• You must skip terms that have no docs and fields that have no terms, even though the provided Fields API will expose them; this typically requires lazily writing the field or term until you've actually seen the first term or document.
• The provided Fields instance is limited: you cannot call any methods that return statistics/counts; you cannot pass a non-null live docs when pulling docs/positions enums.

Specified by:

write in class FieldsConsumer

Throws:

IOException

encodeOutput

static long encodeOutput(long fp,
 boolean hasTerms,
 boolean isFloor)

brToString

static String brToString(BytesRef b)

brToString

static String brToString(byte[] b)

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in class FieldsConsumer

Throws:

IOException

writeBytesRef

private static void writeBytesRef(DataOutput out,
 BytesRef bytes)
                           throws IOException

Throws:

IOException