Overview
The Attivio Intelligence Engine (AIE) uses AIE Connectors to retrieve data documents, from sources such as files and databases, and ingest them. Connectors scan the sources and retrieve the documents via a set of classes called Scanners. The AIE installation includes a number of such scanners for different data sources.
Some of the scanners support an Incremental Update feature, which allows them to track the documents that have been loaded, and re-ingest only those that have been added or changed since the previous ingest.
View incoming links.
Activating Incremental Updates
To activate the Incremental Update feature, the scanner's incrementalModeActivated parameter must be set to "true". You can specify the incrementalModeActivated via a Connector Editor in the AIE Administrator, in the scanner's XML configuration file, or at the command line when running the aie-connect.exe. The scanner keeps a record of the ID and the signature of each document it retrieves in the Store. The signature identifies the document version, and its content depends on the data source that the scanner handles. For example, a FileScanner uses the last file modification time; but a SharePoint Scanner uses a SharePoint-specific token.
The following points are critical to success:
- The amount of space used to maintain the history varies, depending on the data source. It averages somewhere below 1K bytes per document.
- Incremental updates CAN NOT be used with the copy/move after load features, which are available in file based scanners.
Parameters
Option name | Argument data type | Default value | Option description | Is Required |
|---|---|---|---|---|
--incremental-activated | boolean | false | Enables incremental updates. | No |
--incremental-deletes | boolean | true | Used with 'incremental-activated' parameter to control if AIE should delete documents that have been removed. | No |
Example
See Incremental Updating Example for a tutorial example of setting up Incremental Updating.
Connectors that Support Incremental Updating
The following links lead to topics that discuss incremental updating in the context of various types of AIE connectors.