Overview
The legacy Connectors admin page is deprecated. Users should now use the new Connector Admin to create, edit and run connectors.
The AIE Administrator's Connectors page lets you create, edit, edit xml, copy, delete, start, stop and pause Connectors.
Main Article: Load Data and Content
View incoming links.
Working with Connectors
To work with connectors, open the AIE Administrator and select Connectors in the System Management area. The Connectors page appears. From here you can:
- Create, modify, reset, run, and stop connectors.
- Click Scheduler to view scheduled tasks.
- Sort/resort the table by the desired data by clicking the desired column heading.
- Hide the navigation pane with the show/hide button to see more or less of the events table as desired.
The Connector table shows the following fields for each connector.
Statistic | Description |
|---|---|
Name | The Connector's name. |
Status | Connector status, such as Running, Stopped, Paused, Completing. |
Docs Sent | The number of documents found in the current Connector run. |
| Processing | The number of documents for which the Connector has yet to receive acknowledgment from the index. |
| Previous Run | The number of documents processed on the last Connector run. |
Type | Indicates the type of connector (the type of the underlying AIE scanner). |
Workflow | The Connector's destination workflow. |
Node Set | The name of the nodeset where this connector runs. In simple examples, this is the local node. |
Labels | You can label a Connector to help distinguish it. Autocomplete in the search box is based on these labels. |
Performing Actions on Multiple Connectors at a Time
To perform actions on multiple Connectors, select the Connectors upon which you want to perform actions using the checkboxes to the left of the connector names, then click the desired action from the menu bar.
Menu Item | Description |
|---|---|
| New | Create a new connector. The list of connectors reflects the modules that have been added to the project. |
Delete | Delete the selected Connector(s). Note that there is no undo capability for this action. |
Start | Start the selected Connector(s). Statistics update every five seconds. |
Pause | Pause the selected Connector(s). Click Start to resume. |
| Reset | Deletes incremental updating status information. Prepares the connector to begin a new full scan. (Note that Reset does not delete documents from the index. It is a Best Practice to delete the previous documents before loading a new set. See Deleting Content.) |
Stop | Stop the selected Connector(s) before the finish. |
| Scheduler | Assign a connector to run on a schedule. |
Performing Actions on Individual Connectors
To quickly perform actions on individual Connectors:
- Right-click on the desired Connector. A context menu for that Connector appears.
- Select the desired option from the context menu. Available options described below.
Menu Item | Description |
|---|---|
Edit | Edit the Connector Properties. |
Start | Start the Connector running. Statistics update every five seconds. |
| Reset | Deletes incremental updating status information. Prepares the connector to begin a new full scan. (Note that Reset does not delete documents from the index. It is a Best Practice to delete the previous documents before loading a new set. See Deleting Content.) |
Copy | Copy an existing Connector so you can rename and modify it. |
Delete | Delete the Connector. Note that there is no undo capability for this action. |
Test | Retrieve and view ten items from the Connector. |
Creating New Connectors
Most AIE connectors are associated with modules. The module must be included in the project before the connector can be created.
To create a new Connector:
- Click New on the menu bar. The New Connector dialog box appears showing a list of Connector types you can create. The available types depend on the AIE modules loaded.
- Select the type of Connector you want to create, and click OK. A dialog box for creating the selected type of Connector appears showing the Scanner tab.
Scanner Tab
The Scanner tab contains the scanner parameters for the connector. The options vary depending on the Connector type. This example shows the Scanner Tab options for a File Connector.
{
The following properties are common to all connectors. See connector-specific documentation pages for additional properties.
Scanner Editor | Remarks |
|---|---|
Connector Name | The name of the connector as seen in the UI or in XML. |
Node Set | The nodeset the connector should run on. Defaults to default-service-nodeset. The Editor can set this value only on new, unsaved connectors. |
Document ID Prefix | Set the document ID prefix. |
Ingest Workflow | to receive the ingested documents. String. |
Notes Tab
The Notes tab lets you add/edit comments about the Connector. Use the label-editing controls at the bottom of the page to add meaningful labels to your Connector. These labels can help you locate specific connectors in a multi-connector environment.
Scheduler Tab
The Scheduler tab lets you schedule automatic reloads of this feed at the specified interval, from once per minute to once per month.
- Click the + sign to add an event. The Schedule dialog box appears.
- Specify the desired schedule.
- Click OK.
- Click the Field Mappings tab to continue, or if you are done, click Save, or Save & Test.
- To edit an existing event, click the pencil icon next to that event.
- To delete an existing event, click the trash icon next to that event.
Field Mappings Tab
The Field Mappings tab lets you specify dynamic mappings and static mappings.
The following properties are common to all connectors:
CSV Scanner Editor | Remarks |
|---|---|
Field Mappings | Rename fields to set for each document in format oldfield:newfield (option may be specified multiple times). |
Static Field Values | Static field value to set for each document in format myfield:myvalue (option may be specified multiple times). |
Advanced Tab
Best Practice: Leave these settings alone!
In most connectors, the Advanced Tab contains settings that should not be changed without consulting Professional Services first. Many of the settings influence indexing and processing behaviors that interact complexly with one another. Setting them correctly requires a deep knowledge of AIE internals.
The following properties are common to most connectors. See connector-specific documentation pages for additional properties.
Scanner Editor | Remarks |
|---|---|
Document Batch Size | Batch size, integer, default 5000. |
Commit Interval | Commit interval in number of documents. Integer, Default 0. |
Log Interval | Post a status message in the log file every N documents. Default is 1000. |
Wait for Completion | When a commit message is sent, all previously submitted documents must be processed into the index before the commit can finish. Default is true. See the javadoc for ContentFeeder and Message Ordering for more information. |
Wait for Completion Timeout | Maximum amount of time to keep a commit action waiting for confirmation that all documents have finished processing. Integer. Default is one hour. |
| Flush Interval | This parameter is deprecated. Use the Document Batch Size field instead. |
| Result Polling Interval | Sets an interval (in milliseconds) on which audit messages are retrieved from the AIE server. Default is -1, meaning that messages are sent continuously. |
Optimize Interval | Optimize interval expressed in number of documents. Default is zero (don't optimize in the middle of feeding a batch of documents). |
Optimize After Crawl | Enables sending an optimize message after the crawl is completed. Default is "false". |
| Commit After Crawl | Enables sending an commit message after the crawl is completed. Default is "true". |
Ordered Commits | Wait for documents that are still in flight to be indexed before executing commit or other index message. Default is to wait. See Message Ordering for more information |
| Create Unique Content Pointer IDs | If true, makes all ContentPointer IDs unique to avoid collisions at ingest workflow time. Otherwise, uses the id passed into the put() methods of the DirectMessagePublisher . Default is "true". |
Maximum Ingested Documents | Stops crawling when given number of documents is processed. Default is -1 (no maximum). |
| Size to Domain Map | This is a map of document sizes versus domain labels, for use with the MessageDomains feature. Items on the left are minimum file sizes in megabytes. Items on the right are corresponding domain labels. |
| Store Documents Before Feeding | If true, all documents ingested by this connector will be stored in the Document Store for possible re-ingestion at a later date. See Reloading Documents. |
| Collection Name | Collection name for the documents in the store. |
| Store Content Pointers | Store contentPointers from the documents in the document store. |
| Filter Fields | List of field names that support filtering using only the document store APIs for later retrieval. The field values in these fields should contain a few frequently-used values, that is, should have low cardinality. |
Save, Save and Test
In the bottom margin of the Connector Editor there are buttons for saving and testing the connection.
Save Button
The Save button uses dynamic configuration to make the new (or modified) connector active in this instance of AIE. It also writes out a description of the connector in the project's configuration servers.
Save and Test Button
The Save and Test button makes the connector active on all nodes, just like the Save button. It also tests the connector by reading in the first ten records and displaying them in a pop-up table. The left column of the table shows the record ID. Select a record to view its fields and values in the right half of the table.
Viewing Scheduled Events
To view all scheduled events on all connectors, click Scheduler. The Scheduler screen appears:
- Click delete to delete the desired events. Doing so removes the events from the Schedule tabs on the associated connectors.
- Click Stop Scheduler/Start Scheduler to pause the Scheduler as needed.
Boolean Properties in Connector Editors
All of the true/false property controls in the Connector Editors offer a third option: Attivio Properties.
This option provides access to various Boolean properties used in the project. To add a custom property, see Configure Properties.
When things go wrong
When a connector fails, a fatal system event is generated and the connector row is highlighted in red. The status for the connector must be cleared (acknowledging the system event) before it may be restarted.
In the event of a system error or node death where connector execution is interrupted, the system can become temporarily confused about the status of a connector. The connector may be reported as running or completing even though the process that started the connector is no longer functional. Periodically the system checks connectors and clients to update their status. During this process if a connector has been inactive for a significant period it will be marked as so. When this occurs, the connector UI will start to report the connector as having failed. This wait time distinguishes between systems that are operating poorly and those that have crashed. If a user wishes to accelerate this process (so that for instance the connector may be restarted), the existing audit data may be purged via the Store UI. This purging triggers the system to consider the connector as inactive.