Attivio v5.6 Documentation
Page tree
Skip to end of metadata
Go to start of metadata

Overview

Query limits and caching configuration for searchers can be configured by specifying properties on the index searchers configuration.

This page documents all available searcher properties. 

Example:

<f:index>
  <f:searchers>
    <f:property name="searchMaxRows" value="10000"/>
  </f:searchers>
</f:index>

Max Search Rows

By default, any query that requires retrieval of more than 5,000 rows will fail. This prevents unbounded memory usage of queries. In practice, this limit applies to the sum of the query's rows and offset parameter values.

Property

Data TypeDefault

Description

searchMaxRows

int5000

The maximum number of rows that can be returned for a search (0 = unlimited)

If you need to retrieve more than the default 5000 documents, we recommend that you use the Streaming Query API instead of altering searchMaxRows.

Example

<project-dir>\conf\features\core\Index.index.xml
<f:index name="index">
  <f:searchers>
    <f:property name="searchMaxRows" value="5000" />
  </f:searchers>
</f:index>

Search Cache

You can use the search cache to cache the top N results from queries in memory. This caching speeds up retrieving a query's additional pages (for example, get search results 1-10, 11-20, etc.), and improves search performance for frequently-run queries because the results are not read from disk.

Property

Data Type

Default

Description

searchCacheSize

int

512

maximum number of search results to cache

searchAutowarmFactor

float

0.5

percentage of cached search results to use to autowarm the next index

searchCacheMaxRows

int

500

maximum number of rows to cache for a search result

searchCacheWindow 

int

50

window of rows to cache for a search result (search internally always collects this many rows at a minimum)

Index Warming

As part of the COMMIT processing for an AIE Index, the most recent N queries (where N = searchAutoWarmFactor * (number of cached queries)) from the Search Cache are re-run to warm the new index by repopulating the Search Cache with the latest results for those queries. 

The searchAutowarmFactor property has an impact on AIE Index COMMIT time. Setting this value to 0.0 results in no impact at COMMIT time, however, it also means that no previous searches will cache for new indexes, and after a COMMIT, every search will require reading from disk until the cache is repopulated.

Example

<f:index name="index">
  <f:searchers>
    <f:property name="searchCacheSize"      value="512"/>
    <f:property name="searchAutowarmFactor" value="0.5"/>
    <f:property name="searchCacheMaxRows"   value="500"/>
    <f:property name="searchCacheWindow"    value="50"/>
  </f:searchers>
</f:index>

Filter Cache

The filter Cache caches the matching documents for filter queries. If the same filters are commonly used in queries, caching these filters results in a high cache hit ratio (# of times a filter was found in the cache/total # of query requests with filters) on this cache, speeding up of average query execution.

Property

Data Type

Default

Description

filterCacheSize

int

512

maximum number of filters to cache

filterAutowarmFactor

float

0.5

percentage of cached filters to use to autowarm the next index

Index Warming

As part of the COMMIT processing for an AIE Index, the most recent N queries (where N = filterAutowarmFactor * (number of cached queries)) from the Filter Cache re-run to warm the new index by repopulating the Search Cache with the latest results for those queries. 

The filterAutowarmFactor property has an impact on AIE Index COMMIT time. Setting this value to 0.0 results in no impact at COMMIT time, however, it also means that no previous searches will cache for new indexes, and after a COMMIT, every search will require reading from disk until the cache is repopulated.

Example

<f:index name="index">
  <f:searchers>
    <f:property name="filterCacheSize"      value="512"/>
    <f:property name="filterAutowarmFactor" value="0.5"/>
  </f:searchers>
</f:index>

Document Cache

The document cache caches frequently requested documents from the AIE Index. If the same documents are returned by many searches, this cache reduces the (Input-Output) IO processing required to return results. 

Property

Data Type

Default

Description

documentCacheMinSize

int

100

minimum size for per segment document cache

documentCacheMaxSize

int

512

maximum size for per segment document cache

documentCacheLoadFactor

float

0.01

percentage of documents to cache

Per-Segment Caching

The document cache is kept on a per-segment basis. This means that during AIE Index COMMIT processing, the Document Cache for previously committed index segments will not be affected. New segments (created for new documents/merged segments/optimized index) will not add any documents to the cache until after the COMMIT completes and queries results from the new segment start filling the cache. 

Per-Segment Cache Size Calculation

Use the following formula to determine the Document Cache size for each AIE Index segment:

Math.min(documentCacheMaxSize, Math.max(documentCacheMinSize, (segment size * documentCacheLoadFactor)).

Example

<f:index name="index">
  <f:searchers>
    <f:property name="documentCacheMinSize" value="100"/>
    <f:property name="documentCacheMaxSize" value="512"/>
    <f:property name="documentCacheLoadFactor" value="0.01"/> <!-- cache size will be 1% of the segment size -->
  </f:searchers>
</f:index>

Advanced Properties

 The following properties are also available, however they are for expert use only.

  • rtfCachePolicy
  • clauseCacheSize
  • tempTableBufferSize
  • joinBufferSize
  • groupByBufferSize
  • groupByMaxFileSize
  • distinctBufferSize
  • rootDocsBufferSize
  • childDocsBufferSize
  • rootDocsSkipInterval
  • childDocsSkipInterval
  • No labels