Overview

The Attivio platform schema defines the fields that can appear in the Attivio Universal Index.

The default schema file location is: <project-dir>\conf\schema\default.xml

The index fields have a variety of attributes and properties that control how the field will be analyzed and indexed, such as the type of stemming that should be performed on different types of text fields.

Fields derived from your incoming documents must be mapped into Attivio schema fields in order to insert the data into the index.

View incoming links.

• Access Control JOIN ,
• Advanced Text Extraction Module ,
• Advanced Text Extraction Module - Supported Fields ,
• Attivio Configuration ,
• Attivio Connector SDK Quick Start ,
• Attivio Glossary ,
• Attivio Platform ,
• Configure the Advanced Text Extraction Module ,
• Configure the FacetFinder ,
• Configuring Languages in Attivio ,
• Configuring Query Caches ,
• Content Ingestion - Concepts and Tools ,
• Create a New Project ,
• Creating Custom Field Expressions ,
• Creating Custom Ingest Transformers ,
• Dates and Date Formats ,
• Dictionary Entity Extraction ,
• Exposing Locale Properties ,
• Facets ,
• Field Properties ,
• Highlighting Results ,
• Ingestion Quick Start ,
• Joining Database Scanner ,
• Load Data and Content ,
• Loading Factbook XML ,
• Loading Parquet Content ,
• Managing Projects ,
• Multiple Docs per Large XML File ,
• One Doc per XML File ,
• Pattern-Based Categorization ,
• Phonetic Matching ,
• Processing of HTML Files with Text Extraction ,
• Query Modification ,
• Real-Time Updates ,
• Scope Search ,
• Simple Query Language ,
• Stopword Removal ,
• Tokenization ,
• Tokenizer Registration ,
• Updating Documents that Match a Query

Field Mapping in Attivio

The Attivio Schema is intimately related to the larger issue of mapping database fields into the Attivio Universal Index. For instance, database fields name and summary might need to be mapped into Attivio Schema fields title and text. The correct mapping is necessary in order to get the content properly analyzed and into the index.

This mapping from database fields to schema fields can occur in a variety of ways:

• You can edit the Attivio Schema for your project to use the existing database field names as new fields in the Attivio Universal Index.
• Mapping can be performed in the SQL query that pulls content from the database, through the use of field aliases.
• Mapping can be set up quite easily as part of the database connector definition in the Connector Editor. You just fill out a table.
• Mapping can also occur by editing various components of the input workflow.

This page is largely about the first option in this list: modifying the Attivio Schema to handle incoming fields that are not provided by default. The other opportunities for mapping are described where they are encountered in context in the documentation.

Working with the Attivio Schema

During ingestion, connectors create IngestDocument  instances and pass them into ingestion workflows, where the IngestDocuments are processed by workflow components.  Eventually the IngestDocuments reach the indexer subflow, which processes their content and inserts it into the Attivio index.

The connectors and components typically add many fields to the IngestDocuments during ingestion. There is no restriction on the names of these fields; they are simply properties of the IngestDocument that are used to store information temporarily. When you configure a connector or workflow component, you can invent any field names that seem useful.

When the IngestDocuments reach the indexer subflow, the fields whose names match entries in the Attivio schema are inserted into the index. Undefined fields are ignored by default, with the exception of the "wildcard" or "dynamic" fields mentioned below.

To send content to the index, the containing field must be:

• A field that is defined in the default schema, or
• A field that you have customized in the schema, or
• A field that you have added to the schema, or
• Any field name that matches a "wildcard" or "dynamic" field definition in the schema.

Selecting Fields from the Schema

The easy way to index content from IngestDocuments is to use fields that are already defined in the default schema. Just open the project schema file and browse. You may find a preconfigured field that describes your content, such as "author."

This is the default author field:

<field name="author" displayName="Author" type="STRING" multivalue="true" indexed="true" stored="true" sort="false"/>

This means when your connector or workflow component puts the author's name into an IngestDocument field called "author", it will be indexed. Further, the author's name will be stored for retrieval.

The default schema defines over sixty fields, and sometimes contains up to 300 depending on which modules you have loaded.

Customizing Fields in the Schema

Some changes, such as modifying the type of a field, can make the schema incompatible with an existing index. Attivio will be unable to start. The remedy is to revert the changes or delete the existing index files.

Adding Fields to the Schema

To add a field to the schema, open your schema file (The default schema file location is: <project-dir>\conf\schema\default.xml) and simply add another <field> element.

1. Scan the file for a field that is similar to the one you wish to create. This will show you appropriate default settings for that type of field. Copy this entry.
2. Paste the entry at the bottom of the list of fields, just before the </fields> tag. Fields are evaluated top-down, so if there are name conflicts, the last entry will overwrite any previous ones.
3. Customize the entry and save the file.
4. In the AIE CLI, deploy the modified project configuration to the Configuration Server(s); this should restart all running nodes with the new configuration.

Note that schema changes are applied during ingestion, so you may have to re-ingest your previous content.

When customizing a field, there are several attributes that one typically sets:

• name: Choose a unique name, unless you are deliberately overriding an existing field definition.
• type: string, text, boolean, integer, long, float, double, money, decimal, date or shape. (Data-type details are listed in the reference tables below.)
• indexed: Will this field be indexed? Most fields are indexed but some utility fields, such as the uri of an image, are unlikely to be searched.
• stored: Should Attivio store this field value for use in query results? Usually we store field values, but there are exceptions. Some fields like the content field are assembled by concatenating other fields that area already indexed and stored independently. It is useful to index this concatenated content to provide a default-search-field, but there is no need to store it. That information is already stored.
• displayName: If the field name is cryptic and will be displayed in query results, you can optionally give it a more readable display name.

Many other field attributes are listed in the tables below. There are also many field properties that can be invoked. The field properties are listed here.

Indexing Dynamic Fields using Wildcards

Attivio provides a shortcut that lets you index the content of some of your dynamic fields without first modifying the schema. This shortcut is very convenient during prototyping and when creating a demo in real time.

When you create the dynamic field (in your connector or workflow component) you can add a special suffix to the field name. The suffix tells Attivio how to index the field. These are the dynamic-field suffixes that are defined in the default Attivio schema:

The dynamic-field suffixes are often referred to as "wildcards" in the documentation because they are defined using wildcard notation in the Attivio schema.  This is how the "_s" field is defined in the schema:

<field name="*_s" type="STRING" multivalue="false" indexed="true" stored="true" sort="false"/>

As you can see, the "*_s" field matches all dynamic fields that end with "_s".

Defining a "Catch-All" Field

The wildcard "*" standing alone matches all dynamic fields, whether they have special suffixes or not.

<field name="*" type="string" indexed="true" stored="true"/>

Note that Attivio attempts to match unknown fields to these wildcard entries in a top-down manner, and stops with the first successful match. Therefore, if you use the "*" wildcard to describe a "catch-all" field, it needs to be the final wildcard field in the schema. Otherwise it will intercept dynamic fields that were intended to match other wildcards.

<schema> Element

When you create a project with the createproject utility, Attivio creates a local schema file based on the modules that you included in the proejct. This file is <project_dir>\conf\schema\default.xml. Any schema extensions or modifications for the project should be made in this file.

<schema> Attributes

The top-level XML element defining a schema is the <schema> element. The <schema> element has two attributes:

• name: The primary Attivio schema is called "default". Do not change the name of the default schema. Multiple components of Attivio look for the default schema by name.

• merge: Your project schema file may contain multiple "default" schema definitions. This is due to createproject adding small schema definitions from various modules that were included in the project. If merge is true, the schema definition will be merged with the existing one. If set to false, the schema will replace the existing one.

For instance, the keyphrases module adds a single field to the project's schema.xml file. Note that it refers to schema name "default" with merge set to "true."

  <!--
  From module: keyphrases
  -->
  <schema name="default" merge="true">
    <fields>
      <field name="keyphrases" type="string" tokenize="false" indexed="true" stored="true"  displayName="Key Phrases"/>
    </fields>
  </schema>

The default schema contains many field definitions with preconfigured properties that are usually appropriate to that type of field. In any specific project, most of those fields will never be used. These unused fields entail no overhead, so there's no benefit to removing fields from the schema.

<schema> Properties

The following property must be set on the <schema> element.

Example:

<schema name="default">
  <properties>
    <property name="fieldNames.mode" value="multi_value"/>
  </properties>
  ...
</schema>

<fields> Element

The <fields> element is a container for all the <field> and <realtimeField> declarations, which define the attributes of the fields in the index.

<fields> Attributes

The <fields> element has one attribute:

• default-search-field: Defines which field is used to match query terms that do not specify a field. This attribute is set to the content field in the default Attivio schema.

The content field concatenates the document's title, author and text values, plus the values of all *_s and *_i dynamic fields, into a single field value. When the user issues an unfielded query, Attivio looks for a match in the content field.  If a match is found, the search results are compiled using the stored values of the individual fields.  The concatenated value is indexed, but is not stored.

<field> Elements

<field> Attributes

The following attributes can be set as <field> tag attributes.

<field> Properties

See Field Properties for a list of all properties that apply to fields.

<include-field> Elements

Include-field elements are used to create a single field by concatenating the values of other fields. For example, the content field is comprised of many other fields as follows:

<field name="content" type="string" indexed="true" stored="false">
  <include-field name="title" />
  <include-field name="author" />
  <include-field name="text" />
  <include-field name="*_s" />
</field>

<realtimeField> Elements

Real-time fields are configured using the <realtimeField> element.

<realtimeField> Attributes

Real-time field attributes are the same as <field> attributes.

<udfs> Element

The <udfs> element is a container for all the <udf> declarations, which define User Defined Fields.

<udf> Elements

Defines a User Defined Field. This inclues the name of the User Defined Field, the class that provides the implementation, and output type information.

<udf> Attributes

The following attributes can be set as <udf> tag attributes.

Additional Properties Defined in the Schema

The <schema>, <field> and <realtimeField> elements may contain additional properties that can be accessed by various workflow components. They may include, for example, the type of tokenization to use for a field. Properties are typically defined in the schema, rather than in other configuration files, when they are related to the index in some way and are shared by multiple components.

Field Caches and Memory Usage

Schema field configuration can impact the amount of RAM used by the system:

Field caches contain data that is stored in RAM while the Attivio engine is running. They are used for the following:

• faceting on a field
• sorting by a field
• boosting by a field's value
• performing inner/outer/allow/deny join on a field

See Configuring Query Caches for more detailed information about configuring when these caches are loaded into memory, and monitoring their memory usage.

Default schema fields

The Attivio schema has a number of default fields.  This section calls out some of these fields that have universal application to Attivio stages.

Processing Feedback

Processing feedback is potentially generated by any ingestion workflow stage in the system.  In general, any debug, warning, or error information associated processing of the document can be attached to the document.  This allows the processing results for the content to be accessed by applications using the search index.   For example, the following list (non-exhaustive) of workflow events will generate processing feedback to be attached to the document:

• processing timeouts
• failure to extract content due to file encryption or password protected content
• data truncation
• unknown content types
• I/O errors