Overview

AIE can ingest a variety of text-like files from a file system using the File Connector . This page demonstrates how to use the File Connector to crawl a target document container, scan its documents, and extract text to be loaded into the AIE index.

Note that the File Connector sends output to the fileIngest workflow, which invokes Advanced Text Extraction features.  These features extract text from over 500 file types, and can break up composite files (zip files, etc.,) into individual documents automatically.  

View incoming links.

• Activating Incremental Updating ,
• Advanced Linguistics Quick Start ,
• Advanced Text Extraction Module ,
• Building Classification Models ,
• Configure the Advanced Text Extraction Module ,
• Content Ingestion - Concepts and Tools ,
• Dates and Date Formats ,
• Loading Factbook XML ,
• Pattern-Based Entity Extraction ,
• Structure Extraction Module ,
• Thumbnail and Preview Images ,
• Using Classification Models ,
• Using Sentiment Models ,
• Workflow Routing

Before You Begin

Ensure that the environment is prepared as follows:

1. Create a new project that includes the advancedtextextraction and SAIL modules, or just include the demo group of modules, which includes both.
   
   • The advancedtextextraction module provides the fileIngest workflow used in the examples.
   • SAIL lets us view the ingested files as search results.

2. Start AIE using the AIE Agent and its Command-Line Interface (CLI).

Setting Up the Target Files

To experiment with the File Connector, create a new directory and put a few document files in it. In these examples, using a Windows environment, we created this directory:

c:\documents

The top-level directory is not a requirement; it just makes the example code easier to read. We copied a few .docx and .html files into the directory.

Configuring a File Connector

The Connector Editor is part of the AIE Administrator.

Once you have started the AIE node, the AIE Administrator will be available at http://<host>:17000/admin.

In the AIE Administrator, navigate to System Management > Connectors. Click New in the menu bar. Select the File Connector from the list.

On the Scanner tab of the resulting editor, enter the Connector Name (textFileConnector) and the Start Directory (c:\documents). With the example we have set up on this page, you can accept all other default values.

The remaining fields in the editor are listed in the section on  File Connector Properties, below on this page.

Click Save. The Connector UI sends the connection's XML configuration to the Configuration Server.  To view this configuration, use the CLI update command, and then look for the new connector file in the <project-dir>\conf\connectors directory.  It will have the same name that you gave to the connector. 

File Connector Properties

The File Connector is configured by setting properties on the editor.

The table above is for the File Connector Scanner Tab.  The other tabs in the Connector Editor are described on the Connectors page.

Running the File Connector

Open the AIE Administrator in a web browser, using a URL similar to http://localhost:17000/admin/.

Navigate to System Management > Connectors. Here is the textFileConnector we just defined:

Click the checkbox to select the connector. Click the Start link to load the files.

Navigate to the SAIL interface: Query > SAIL. You may search by clicking on a Key Phrase, or by typing keywords into the Search field. To match all available records, search for *:*.

SAIL will show you a great deal more information if you click the Search Options link and check the Debug checkbox.

Loading Single Record XML Files with the File Connector

If the XML content to be loaded is formatted such that each XML file corresponds to a single IngestDocument , the content can be loaded by using the standard file connector and the xmlIngest ingestion workflow.

For example, say your XML input data looks like this:

<?xml version="1.0" encoding="utf-8"?>
<doc id="1">
  <field name="name1" value="value1" />
  <field name="name2" value="value2" />
  <field name="less_useful_data" value="value3" />
</doc>

<?xml version="1.0" encoding="utf-8"?>
<doc id="2">
  <field name="name1" value="value4" />
  <field name="name2" value="value5" />
  <field name="less_useful_data" value="value6" />
</doc>

and you want to bring each XML file into AIE as a separate IngestDocument, preserving the name1 and name2 fields and omitting the less_useful_data field. Do this by configuring a standard file connector to operation on XML files, to route the incoming documents to the xmlIngest workflow, and modify the xPathExtractor component of that workflow to map name1 and name2 into AIE schema fields.

Of these steps, configuring the file connector will be left as an exercise for the reader, since the process is almost identical to the example that is on this page.  The following discussion shows how to map XML values into AIE schema fields using the XPathExtractor component.

Configuring XPathExtractor

AIE uses XPath Expressions to map XML elements and attributes to AIE document fields. Mappings are realized through the xPathExtractor ingestion component. The default AIE configuration (<install_dir>\conf\core-app\attivio-components.xml) contains an xPathExtractor transformer in the standard xmlIngest ingestion workflow.

The configuration of the default xPathExtractor transformer is very basic. To edit xPathExtractor in the AIE Administrator, navigate to the System Management > Palette page. Use the Search Components field to search for "xPathExtractor." Click on the name of the component to open an editor.

You can accept all of the existing settings except for the XPaths settings. These settings map XML field values to AIE schema fields, so they must be customized to match your XML data.

The format of the XPath expressions (the strings "/doc/field[@name = 'name1']/@value" and "/doc/field[@name = 'name2']/@value", above) is described in more detail at http://en.wikipedia.org/wiki/XPath. Suffice to say that these expressions have the following interpretation:

• Find the top level <doc> element. Find the <field> element underneath it whose name attribute equals "name2". The value of this element's value attribute will be mapped to the AIE schema field "text".
• Find the top level <doc> element. Find the <field> element underneath it whose name attribute equals "name1". The value of this element's value attribute will be mapped to the AIE schema field "title".

Note that the field whose name attribute is set to "less_useful_data" will be ignored when ingesting these documents.

Additional Examples

The Factbook Quick Start Tutorial configuration provides another example of XPath mapping of fields.

Main Article: Loading Factbook XML

Content Encoding

This connector extracts content in binary form, and thus has no encoding limitations. However, the ingest workflow (typically fileIngest), which generates field values from the binary content, may not support all content encoding schemes. 

Incremental Updating

This is one of the connectors that supports the Activating Incremental Updating features. There is a tutorial example of incremental updating here.