Package org.apache.lucene.store
Class NRTCachingDirectory
java.lang.Object
org.apache.lucene.store.Directory
org.apache.lucene.store.FilterDirectory
org.apache.lucene.store.NRTCachingDirectory
- All Implemented Interfaces:
Closeable,AutoCloseable,Accountable
Wraps a
RAMDirectory
around any provided delegate directory, to
be used during NRT search.
This class is likely only useful in a near-real-time context, where indexing rate is lowish but reopen rate is highish, resulting in many tiny files being written. This directory keeps such segments (as well as the segments produced by merging them, as long as they are small enough), in RAM.
This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd.
Here's a simple example usage:
Directory fsDir = FSDirectory.open(new File("/path/to/index").toPath());
NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0);
IndexWriterConfig conf = new IndexWriterConfig(analyzer);
IndexWriter writer = new IndexWriter(cachedFSDir, conf);
This will cache all newly flushed segments, all merges
whose expected segment size is <= 5 MB, unless the net
cached bytes exceeds 60 MB at which point all writes will
not be cached (until the net bytes falls below 60 MB).
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final RAMDirectoryprivate final longprivate final longprivate final Objectprivate static final booleanFields inherited from class org.apache.lucene.store.FilterDirectory
inFields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE -
Constructor Summary
ConstructorsConstructorDescriptionNRTCachingDirectory(Directory delegate, double maxMergeSizeMB, double maxCachedMB) We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is<= maxMergeSizeMB, and 2) the total cached bytes is<= maxCachedMB -
Method Summary
Modifier and TypeMethodDescriptionvoidclose()Close this directory, which flushes any cached files to the delegate and then closes the delegate.createOutput(String name, IOContext context) Creates a new, empty file in the directory and returns anIndexOutputinstance for appending data to this file.createTempOutput(String prefix, String suffix, IOContext context) Creates a new, empty, temporary file in the directory and returns anIndexOutputinstance for appending data to this file.voiddeleteFile(String name) Removes an existing file in the directory.protected booleandoCacheWrite(String name, IOContext context) Subclass can override this to customize logic; return true if this file should be written to the RAMDirectory.longfileLength(String name) Returns the byte length of a file in the directory.Returns nested resources of this class.String[]listAll()Returns names of all files stored in this directory.String[]Opens a stream for reading an existing file.longReturn the memory usage of this object in bytes.voidRenamessourcefile todestfile wheredestmust not already exist in the directory.(package private) static booleanslowFileExists(Directory dir, String fileName) Returns true if the file exists (can be opened), false if it cannot be opened, and (unlike Java's File.exists) throws IOException if there's some unexpected error.voidsync(Collection<String> fileNames) Ensures that any writes to these files are moved to stable storage (made durable).toString()private voidMethods inherited from class org.apache.lucene.store.FilterDirectory
getDelegate, getPendingDeletions, obtainLock, syncMetaData, unwrapMethods inherited from class org.apache.lucene.store.Directory
copyFrom, ensureOpen, getTempFileName, openChecksumInput
-
Field Details
-
cache
-
maxMergeSizeBytes
private final long maxMergeSizeBytes -
maxCachedBytes
private final long maxCachedBytes -
VERBOSE
private static final boolean VERBOSE- See Also:
-
uncacheLock
-
-
Constructor Details
-
NRTCachingDirectory
We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is<= maxMergeSizeMB, and 2) the total cached bytes is<= maxCachedMB
-
-
Method Details
-
toString
- Overrides:
toStringin classFilterDirectory
-
listAll
Description copied from class:DirectoryReturns names of all files stored in this directory. The output must be in sorted (UTF-16, java'sString.compareTo(java.lang.String)) order.- Overrides:
listAllin classFilterDirectory- Throws:
IOException- in case of I/O error
-
deleteFile
Description copied from class:DirectoryRemoves an existing file in the directory. This method must throw eitherNoSuchFileExceptionorFileNotFoundExceptionifnamepoints to a non-existing file.- Overrides:
deleteFilein classFilterDirectory- Parameters:
name- the name of an existing file.- Throws:
IOException- in case of I/O error
-
fileLength
Description copied from class:DirectoryReturns the byte length of a file in the directory. This method must throw eitherNoSuchFileExceptionorFileNotFoundExceptionifnamepoints to a non-existing file.- Overrides:
fileLengthin classFilterDirectory- Parameters:
name- the name of an existing file.- Throws:
IOException- in case of I/O error
-
listCachedFiles
-
createOutput
Description copied from class:DirectoryCreates a new, empty file in the directory and returns anIndexOutputinstance for appending data to this file. This method must throwFileAlreadyExistsExceptionif the file already exists.- Overrides:
createOutputin classFilterDirectory- Parameters:
name- the name of the file to create.- Throws:
IOException- in case of I/O error
-
sync
Description copied from class:DirectoryEnsures that any writes to these files are moved to stable storage (made durable). Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.- Overrides:
syncin classFilterDirectory- Throws:
IOException- See Also:
-
rename
Description copied from class:DirectoryRenamessourcefile todestfile wheredestmust not already exist in the directory. It is permitted for this operation to not be truly atomic, for example bothsourceanddestcan be visible temporarily inDirectory.listAll(). However, the implementation of this method must ensure the content ofdestappears as the entiresourceatomically. So oncedestis visible for readers, the entire content of previoussourceis visible. This method is used by IndexWriter to publish commits.- Overrides:
renamein classFilterDirectory- Throws:
IOException
-
openInput
Description copied from class:DirectoryOpens a stream for reading an existing file. This method must throw eitherNoSuchFileExceptionorFileNotFoundExceptionifnamepoints to a non-existing file.- Overrides:
openInputin classFilterDirectory- Parameters:
name- the name of an existing file.- Throws:
IOException- in case of I/O error
-
close
Close this directory, which flushes any cached files to the delegate and then closes the delegate.- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Overrides:
closein classFilterDirectory- Throws:
IOException
-
doCacheWrite
Subclass can override this to customize logic; return true if this file should be written to the RAMDirectory. -
createTempOutput
public IndexOutput createTempOutput(String prefix, String suffix, IOContext context) throws IOException Description copied from class:DirectoryCreates a new, empty, temporary file in the directory and returns anIndexOutputinstance for appending data to this file. The temporary file name (accessible viaIndexOutput.getName()) will start withprefix, end withsuffixand have a reserved file extension.tmp.- Overrides:
createTempOutputin classFilterDirectory- Throws:
IOException
-
slowFileExists
Returns true if the file exists (can be opened), false if it cannot be opened, and (unlike Java's File.exists) throws IOException if there's some unexpected error.- Throws:
IOException
-
unCache
- Throws:
IOException
-
ramBytesUsed
public long ramBytesUsed()Description copied from interface:AccountableReturn the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsedin interfaceAccountable
-
getChildResources
Description copied from interface:AccountableReturns nested resources of this class. The result should be a point-in-time snapshot (to avoid race conditions).- Specified by:
getChildResourcesin interfaceAccountable- See Also:
-