Page tree
Skip to end of metadata
Go to start of metadata

Overview

The legacy Connectors admin page is deprecated. Users should now use the new Connector Admin to create, edit and run connectors.


The AIE Administrator's Connectors page lets you create, edit, edit xml, copy, delete, start, stop and pause Connectors.

Main Article: Load Data and Content  

View incoming links.

Working with Connectors

To work with connectors, open the AIE Administrator and select Connectors in the System Management area. The Connectors page appears. From here you can:

  • Create, modify, reset, run, and stop connectors.
  • Click Scheduler to view scheduled tasks.
  • Sort/resort the table by the desired data by clicking the desired column heading.
  • Hide the navigation pane with the show/hide button to see more or less of the events table as desired.

The Connector table shows the following fields for each connector.

Statistic

Description

Name 

The Connector's name. 

Status

Connector status, such as Running, Stopped, Paused, Completing.

Docs Sent 

The number of documents found in the current Connector run.

Processing

The number of documents for which the Connector has yet to receive acknowledgment from the index.

Previous RunThe number of documents processed on the last Connector run.

Type 

Indicates the type of connector (the type of the underlying AIE scanner). 

Workflow

The Connector's destination workflow.

Node Set 

The name of the nodeset where this connector runs.  In simple examples, this is the local node. 

Labels 

You can label a Connector to help distinguish it. Autocomplete in the search box is based on these labels. 

Performing Actions on Multiple Connectors at a Time

To perform actions on multiple Connectors, select the Connectors upon which you want to perform actions using the checkboxes to the left of the connector names, then click the desired action from the menu bar.

Menu Item

Description

NewCreate a new connector. The list of connectors reflects the modules that have been added to the project.

Delete

Delete the selected Connector(s).  Note that there is no undo capability for this action.

Start

Start the selected Connector(s).  Statistics update every five seconds.

Pause

Pause the selected Connector(s).  Click Start to resume.

ResetDeletes incremental updating status information.  Prepares the connector to begin a new full scan.  (Note that Reset does not delete documents from the index. It is a Best Practice to delete the previous documents before loading a new set. See Deleting Content.)

Stop

Stop the selected Connector(s) before the finish. 

SchedulerAssign a connector to run on a schedule.

Performing Actions on Individual Connectors

To quickly perform actions on individual Connectors:

 

  1. Right-click on the desired Connector. A context menu for that Connector appears.
  2. Select the desired option from the context menu. Available options described below.

Menu Item

Description

Edit

Edit the Connector Properties.

Start

Start the Connector running.  Statistics update every five seconds.

ResetDeletes incremental updating status information.  Prepares the connector to begin a new full scan.  (Note that Reset does not delete documents from the index. It is a Best Practice to delete the previous documents before loading a new set. See Deleting Content.)

Copy

Copy an existing Connector so you can rename and modify it.

Delete

Delete the Connector.  Note that there is no undo capability for this action.

Test

Retrieve and view ten items from the Connector.

Creating New Connectors

Most AIE connectors are associated with modules.  The module must be included in the project before the connector can be created.

To create a new Connector:

  1. Click New on the menu bar. The New Connector dialog box appears showing a list of Connector types you can create. The available types depend on the AIE modules loaded.
    NewConnectorDB
  2. Select the type of Connector you want to create, and click OK. A dialog box for creating the selected type of Connector appears showing the Scanner tab.

Scanner Tab

The Scanner tab contains the scanner parameters for the connector. The options vary depending on the Connector type.  This example shows the Scanner Tab options for a File Connector.

ConnectorScannerTab {

The following properties are common to all connectors.  See connector-specific documentation pages for additional properties.

Scanner Editor

Remarks

Connector Name

The name of the connector as seen in the UI or in XML.

Node Set

The nodeset the connector should run on. Defaults to default-service-nodeset. The Editor can set this value only on new, unsaved connectors.

Document ID Prefix

Ingest Workflow


Ingestion workflow

to receive the ingested documents. String.

Notes Tab

The Notes tab lets you add/edit comments about the Connector. Use the label-editing controls at the bottom of the page to add meaningful labels to your Connector. These labels can help you locate specific connectors in a multi-connector environment.

COnnectorNotesTab

Scheduler Tab

The Scheduler tab lets you schedule automatic reloads of this feed at the specified interval, from once per minute to once per month.  

  1. Click the + sign to add an event. The Schedule dialog box appears.
  2. Specify the desired schedule.
  3. Click OK.
  4. Click the Field Mappings tab to continue, or if you are done, click Save, or Save & Test.
  • To edit an existing event, click the pencil icon next to that event.
  • To delete an existing event, click the trash icon next to that event.

SchedulerTab

Field Mappings Tab

The Field Mappings tab lets you specify dynamic mappings and static mappings. 

ConnectorMappingTab

The following properties are common to all connectors:

CSV Scanner Editor

Remarks

Field Mappings

Rename fields to set for each document in format oldfield:newfield

(option may be specified multiple times).

Static Field Values

Static field value to set for each document in format myfield:myvalue

(option may be specified multiple times).

 

Advanced Tab

Best Practice: Leave these settings alone!

In most connectors, the Advanced Tab contains settings that should not be changed without consulting Professional Services first.  Many of the settings influence indexing and processing behaviors that interact complexly with one another.  Setting them correctly requires a deep knowledge of AIE internals.

The following properties are common to most connectors.  See connector-specific documentation pages for additional properties.

Scanner Editor

Remarks

Document Batch Size

Batch size, integer, default 5000.

Commit Interval

Commit interval in number of documents. Integer, Default 0.

Log Interval

Post a status message in the log file every N documents. Default is 1000.

Wait for Completion

When a commit message is sent, all previously submitted documents must be processed into the index before the commit can finish. Default is true. See the javadoc for ContentFeeder

 and Message Ordering for more information.


Wait for Completion Timeout

Maximum amount of time to keep a commit action waiting for confirmation that all documents have finished processing. Integer. Default is one hour.

Flush IntervalThis parameter is deprecated. Use the Document Batch Size field instead.
Result Polling Interval

Sets an interval (in milliseconds) on which audit messages are retrieved from the AIE server. Default is -1, meaning that messages are sent continuously.

Optimize Interval

Optimize interval expressed in number of documents. Default is zero (don't optimize in the middle of feeding a batch of documents).

Optimize After Crawl

Enables sending an optimize message after the crawl is completed. Default is "false".

Commit After CrawlEnables sending an commit message after the crawl is completed. Default is "true".

Ordered Commits

Wait for documents that are still in flight to be indexed before executing commit or other index message. Default is to wait. See Message Ordering for more information

Create Unique Content Pointer IDs

If true, makes all ContentPointer IDs unique to avoid collisions at ingest workflow time. Otherwise, uses the id passed into the put() methods of the DirectMessagePublisher

. Default is "true".

Maximum Ingested Documents

Stops crawling when given number of documents is processed. Default is -1 (no maximum).

Size to Domain MapThis is a map of document sizes versus domain labels, for use with the MessageDomains feature. Items on the left are minimum file sizes in megabytes. Items on the right are corresponding domain labels.
Store Documents Before FeedingIf true, all documents ingested by this connector will be stored in the Document Store for possible re-ingestion at a later date. See Reloading Documents.
Collection NameCollection name for the documents in the store.
Store Content PointersStore contentPointers from the documents in the document store.
Filter FieldsList of field names that support filtering using only the document store APIs for later retrieval. The field values in these fields should contain a few frequently-used values, that is, should have low cardinality.

 

Save, Save and Test

In the bottom margin of the Connector Editor there are buttons for saving and testing the connection.

Save Button

The Save button uses dynamic configuration to make the new (or modified) connector active in this instance of AIE.  It also writes out a description of the connector in the project's configuration servers.

Save and Test Button

The Save and Test button makes the connector active on all nodes, just like the Save button.  It also tests the connector by reading in the first ten records and displaying them in a pop-up table.  The left column of the table shows the record ID.  Select a record to view its fields and values in the right half of the table.

SaveAndTest

 

Viewing Scheduled Events

To view all scheduled events on all connectors, click Scheduler.  The Scheduler screen appears:

Scheduler

  • Click delete to delete the desired events. Doing so removes the events from the Schedule tabs on the associated connectors.
  • Click Stop Scheduler/Start Scheduler to pause the Scheduler as needed.

 

Boolean Properties in Connector Editors

All of the true/false property controls in the Connector Editors offer a third option: Attivio Properties.

This option provides access to various Boolean properties used in the project.  To add a custom property, see Configure Properties.

When things go wrong

When a connector fails, a fatal system event is generated and the connector row is highlighted in red.   The status for the connector must be cleared (acknowledging the system event) before it may be restarted.

In the event of a system error or node death where connector execution is interrupted, the system can become temporarily confused about the status of a connector.  The connector may be reported as running or completing even though the process that started the connector is no longer functional.  Periodically the system checks connectors and clients to update their status.  During this process if a connector has been inactive for a significant period it will be marked as so.  When this occurs, the connector UI will start to report the connector as having failed.  This wait time distinguishes between systems that are operating poorly and those that have crashed.  If a user wishes to accelerate this process (so that for instance the connector may be restarted), the existing audit data may be purged via the Store UI.  This purging triggers the system to consider the connector as inactive.

 

  • No labels