Overview

An Attivio project scales up through partitioning and replication.

• Partitioning divides a large index into multiple parts, with each part running in parallel (on separate hardware) to speed up ingestion and searching.
• Replication creates additional copies of the partition set, called rows, so that you can scale to meet your application's query load.

The Index Feature automatically generates these components based on a higher-level index configuration. The Index Feature coordinates very complex interactions among these nodes based on:

• The number of desired partitions.
• The names of the indexing nodes (in blue, above).
• The names of the searching nodes (in green, above).

For the above system, the Index Feature configuration could be (almost) as simple as this:

<f:index name="index">
  <f:partitionSet size="5"/>
</f:index>

This configuration allocates documents to partitions by hashing on the document ID. If you really wanted to divide the documents alphabetically (as implied by the diagram) you would need to invoke Partition Routing.

View incoming links.

• Attivio Features ,
• Attivio Glossary ,
• Attivio Platform 5.6.1 Release Notes ,
• Attivio Platform 5.6.2 Release Notes ,
• Configure the FacetFinder ,
• Configuring Query Caches ,
• Create an Attivio Project on Hadoop ,
• Creating Custom Field Expressions ,
• Facets ,
• In Depth Architecture Diagrams ,
• Index Engines ,
• Indexes ,
• Managing Environments ,
• Message Grouping ,
• Multi-Node Topologies ,
• Query Performance Tuning ,
• Real-Time Updates ,
• System Status ,
• Updating Content ,
• Zoned Indexing

Index Feature

Configuration for an index feature is provided by the <f:index> element in the <PROJECT_DIR>/conf/features/core/Index.<INDEX_NAME>.xml file.

By default, a new Attivio project includes a single index, named index, configured in the <PROJECT_DIR>/conf/features/core/Index.index.xml file.

<f:index> Attributes

<f:index> Elements

Minimal Example Config

<f:index name="index"/>

Index Process Memory

The <f:memory> element is used to configure memory allocation for index processes in a clustered Attivio project. 

Example

<f:index name="index">
  <f:memory size="4G" diskCacheSize="50%"/>
</f:index>

Index Storage

The index storage can be configured using the <f:storage> element of the <f:index> feature.

For clustered projects, the index is stored in HDFS. For unclustered projects, the index is stored on the local file system.

Unclustered Storage Properties

The following properties are available for configuring storage for unclustered projects.

Example

<f:index>
  <f:storage>
    <f:property name="path" value="/opt/attivio/disk_alt/data-index" />
  </f:storage>
</f:index>

MMap Example

<f:index>
  <f:storage>
    <f:property name="mmap" value="true" />
    <!-- disable mmap for stored fields/term vectors -->
    <f:property name="mmapStoredFields" value="false"/>
    <f:property name="mmapTermVectors" value="false"/>
  </f:storage>
</f:index>

Clustered Storage Properties

The following properties are available for configuring storage for clustered projects.

Example

<f:index>
  <f:storage>
    <f:property name="totalMemory" value="17179869184"/> <!-- 16G -->
    <f:property name="directMemory" value="true"/> <!-- use non-heap memory -->
  </f:storage>
</f:index>

Index Writer

The <f:writer> element is used to configure the index writer row. This is the Attivio engine instance which writes out index files as new documents are indexed.

Index Writer Properties

Advanced settings on the Index Writer can be configured using the <f:property> element on the <f:writer> element.  See  Index Engines for more detail on the possible properties, and  Index Writer Properties for a complete list of available index writer properties.

Example: Set the maximum number of segments produced by optimization to 3 (instead of the default of 5):

<f:index name="index">
  <f:writer>
    <f:property name="optimizeMaxSegments" value="3" />
  </f:writer>
</f:index>

Zones

It is sometimes more efficient to create a project where the index has been subdivided into zones. These zones behave like separate indexes; they can be queried separately or in parallel, creating opportunities for more efficient queries. Zones can be confured on the index writer using the <f:zone> element. For full discussion on zones, see: Zoned Indexing.

<f:zone> Attributes

Zone Routing

The <f:route> element of the <f:zone> element allows configuring a field value based routing to a zone.

For instance, this will typically be used to route documents to a zone based on a table (or similar collection identification) field.

Example: Route documents with table schema field values of "foo" to the bar zone. All other documents will be routed to the default zone.

<f:index name="index">
  <f:writer defaultZoneName="default">
    <f:zone name="default"/>
    <f:zone name="bar">
      <f:route field="table" value="foo"/>
    </f:zone>
  </f:writer>
</f:index>

Zone Properties

Index Writer properties can be configured on a per-zone basis by using the <f:property> element of the <f:zone> element. See Index Writer Properties for a complete list of available index writer properties.

The Zone Adding Feature

Sometimes it is desirable to add a zone to an index which is configured somewhere else. For example, a module may require its own separate zone in the default index. The following defintion will add a hidden zone called myzone to the index named index:

    <f:addIndexZone index="index">
      <f:zone name="myzone">
        <f:route field="table" value="somevalue"/>
        <f:property name="hidden" value="true"/>
      </f:zone>
    </f:addIndexZone>

Index Searchers

The <f:searchers> element is used to configure index searcher rows. This configuration applies to all searcher rows and the writer row (if search="true" and a dedicated searcher is not available or configured).

Index Searcher Properties

Advanced settings on the Index Searcher can be configured using the <f:property> element on the <f:searchers> element. See Index Searcher Properties for a complete list of available index searcher properties.

Example: Allow queries to request offsets up to 20000:

<f:index name="index">
  <searchers>
    <f:property name="searchMaxRows" value="20000" />
  </searchers>
</f:index>

Join Configuration

Join execution performance can be explicitly configured to execute locally per-partition, or to execute across all partitions. The default is to execute join queries across all partitions for correctness. This default can be changed, or overridden on a per join key basis. The default join configuration can be changed by setting the defaultJoinMode attribute of the <f:searchers> element. Per-field join execution can be configured using the <f:join> element of <f:searchers>.

Join Modes

<f:join> Attributes

Example: Set default join configuration to execute join queries locally per partition:

<f:index name="index">
  <f:searchers defaultJoinMode="local"/>
</f:index>

Example: Override join execution on a per-key basis:

<f:index name="index">
  <f:searchers defaultJoinMode="remote">
    <f:join primary="primaryKey" foreign="foreignKey" mode="local"/>
  </f:searchers>
</f:index>

Index Partitioning

Most Attivio users index very large collections of data. Trying to hold all that data in a single, monolithic index is not practical. Some indexes are just too large for any one server to handle. Index building, maintenance, and querying all impose loads on the system that are proportional to the size of the index. For all of these reasons, it is advantageous to break up the monolithic index into partitions, and treat each of the partitions as if it were a small independent index. Attivio automatically distributes incoming documents across partitions, and dispatches each query to a set of relevant partitions before combining their partial results into a single response document.

During content ingestion, each document is sent to a specific partition based on the value of one or more fields. Each partition has a dedicated indexer node that ingests the document and adds it to the local index. Therefore, each partition contains only a fraction of the full index, and is only a fraction of the size of the full index.

Partitioning is configured by adding the <f:partitionSet> element to the Index Feature definition. This element has a size attribute that indicates the number of partitions for the index. Just tell Attivio how many partitions you want and Attivio will set them up.

This is a simple two-partition index:

<f:index name="index">
  <f:partitionSet size="2">
  </f:partitionSet>
</f:index>

Partition Routing

Attivio indexes can route incoming documents to specific partitions based on document properties. Conditional routing is accomplished by adding multiple <f:route> elements to the index declaration. Routing elements act as a series of filters: the first one to apply to a document is used. The list of route elements should end with an empty <f:route/> element to catch documents which do not match any preceding routing element.

The <f:route> element lets us set up a series of conditional routing tests to apply to each incoming document. The first successful match determines the partition where the document will be stored. It also optionally lets us send a document to all partitions.

The <f:filter>  element allows applying filters to an <f:route> element .

The partition element allows specifying which partitions to route across. If no partition elements are configured for a route, routing will occur across all partitions.

Example: Route documents using hashfield as the partition field:

<f:index>
  <f:partitionSet size="2">
    <!-- Default route - hash by value of "hashfield" -->
    <f:route field="hashfield"/>
  </f:partitionSet> 
</f:index>

Example: Route documents with table schema field values of "acl" to all index partitions:

<f:index>
  <f:partitionSet size="2">
    <f:route broadcast="true">
      <f:filter field="table" value="acl"/>
    </f:route>

    <!-- Default Route -->
    <f:route/>
  </f:partitionSet>
</f:index>

Example: Route documents with table schema field values of "user" with different properties than those routed by the default route:

<f:index>
  <f:partitionSet size="2">
    <f:route multiValue="true" consistent="true">
      <f:filter field="table" value="user"/>
    </f:route>

    <!-- Default Route -->
    <f:route consistent="false"/>
  </f:partitionSet>
</f:index>

Example: Field value-based partition routing:

<f:index name="index">
  <f:partitionSet size="4">
    <!-- route documents with table=country to partition 0 -->
    <f:route>
      <f:filter field="table" value="country"/>
      <f:partition index="0"/>
    </f:route>

    <!-- route documents with table=city to partition 1 -->
    <f:route>
      <f:filter field="table" value="city"/>
      <f:partition index="1"/>
    </f:route>

    <!-- route documents with table=news to partition 2 -->
    <f:route>
      <f:filter field="table" value="news"/>
      <f:partition index="2"/>
    </f:route>

    <!-- route documents with table=medal to partition 3 -->
    <f:route>
      <f:filter field="table" value="medal"/>
      <f:partition index="3"/>
    </f:route>
  </f:partitionSet>
</f:index>

Content and Query Dispatchers

Sometimes it is useful to specify which workflow the content dispatchers (the entry point for documents to the index) or query dispatchers (the entry point for queries to the index) will be placed in. In the example below, the content dispatcher is placed in a workflow named customIndexer and the query dispatcher is placed in a workflow named customSearcher. Multiple content and query dispatcher specifications may be provided as needed.

<f:index>
  <f:contentDispatcher workflow="customIndexer"/>
  <f:queryDispatcher workflow="customSearcher"/>
</f:index>

Content and Query Dispatcher Elements

The <f:contentDispatcher> element and the <f:queryDispatcher> element share the following elements:

Query Dispatcher Properties

Query dispatcher properties can be configured on the <f:queryDispatcher> element.

Example: Configure a 5-second connection timeout for the Query Dispatcher:

 <f:index>
  <f:queryDispatcher>
    <f:property name="connectTimeout" value="5000"/>
  </f:queryDispatcher>
  ...
</f:index>

Content Dispatcher Properties

Content dispatcher properties can be configured on the <f:contentDispatcher> element.

Example: Configure a 2-minute connection timeout for the Content Dispatcher:

 <f:index>
  <f:contentDispatcher>
    <f:property name="connectTimeout" value="120000"/>
  </f:contentDispatcher>
  ...
</f:index>