Quick Start

The following exercise builds on the Attivio Quick Start tutorial to introduce the concept of workflows and custom ingest transformers. We'll deploy the Factbook project and then edit an ingestion workflow into which we'll insert a custom transformer which will do something odd but fun: we'll make some of our content searchable in Pig Latin. We'll do this by stacking additional tokens.

Prerequisites

In order to complete the following exercise, you'll need to have the following software installed:

Application	Recommended Version	Download Location	Notes
Maven	3.5 or later	https://maven.apache.org/
Eclipse	Oxygen or later	https://www.eclipse.org/	Be sure to install the M2Eclipse plugin or use the Eclipse IDE for Java Developers installer which includes Maven integration.

Sidebar on Tokens

Since we'll be inserting a custom transformer that will be stacking tokens in Pig Latin, we should first have a basic understanding of how tokens work.

Introduction

In order to support free text search, fields in an Attivio index are tokenized by default in the ingest workflow if they are indexed and are a string field. This means that the field's text value is split into "tokens", which are typically words. Each token and its position in the document is indexed on its own and can be searched individually or as part of a more complex query.

The Anatomy of a Token

Let's say a document we ingested contains the phrase "the lazy dog". Examining the generated tokens, we have:

['the'@800001#35~38] ['lazy'@1000#39~43] ['dog'@2800#44~47]

The token output is constructed as follows:

Each separate token is grouped within square brackets. [ ... ]
The token value is written between single quotes. 'the'
Token annotation information appears after the @ character. It is output as a long (in hex format). @800001
Token startOffset is output after the # character. This is a character count from the start of the field and is zero-based (a token starting at the beginning of a field would have a startOffset of 0). #35
Token endOffset is output after the ~ character. This is the index of the character immediately following the end of the token text, such that token endOffset - startOffset = token length. ~38

Token Stacking

Stacking is a concept whereby tokens can be added to the TokenList overlaid on top of an existing token. A token found in the text is referred to as a surface token. Token stacking is used by certain linguistic operations to generate additional tokens to improve recall or provide additional query functionality.

Operations that produce stacked tokens include:

Lemmatization – additional tokens are generated for lemmas and stacked at the surface token position.
Entity Extraction – additional tokens are stacked denoting the type of located entity, e.g. person, and the start and end position of the entity.
SoundsLike – additional tokens are produced by the Sounds Like operation and are stacked at the surface token position.

If we take the example "the quick brown fox jumped over the lazy dog", the lemmatization tokens added for the word jumped are stacked underneath the surface token and inherit the same start and end offset values. In addition, these stacked lemma tokens are annotated with the TokenAnnotation.LEMMA annotation flag @8.

0         10        20        30        40        50
|         |         |         |         |         |
012345678901234567890123456789012345678901234567890
|         |         |         |         |         |
the    quick brown fox jumped over the lazy dog
 
Tokens:
the    quick brown fox jumped over the lazy dog
                       jump
                       jumps
Raw Tokens:
['the'@800001#0~3] ['quick'#7~12] ['brown'#13~18] ['fox'#19~22] ['jumped'#23~29, 'jump'@8#23~29, 'jump'@8#23~29, 'jumps'@8#23~29] ['over'#30~34] ['the'@800001#35~38] ['lazy'#39~43] ['dog'#44~47]

Now that we know what tokens are, we need to have a way to examine them during ingestion.

1. Deploy the Attivio Factbook Project

Open the Attivio Quick Start Tutorial in a new tab in your browser, follow the instructions to deploy the Factbook project, and return to this page when complete.

2. Create a LogDocument component and insert it into the workflow

Click System Management > Palette
Click New
Type LogDocument in the search box and click Filter
Expand Document Transformers and click the LogDocument component type that is found and click OK
In the popup that appears enter the following:
Field Value
Name logDocument
Click Save
Click System Management > Workflows > Ingest
Click ingestPostProcess
Click the Add Existing Component Button
Type logDocument in the search box and click Filter
Expand Platform Components > Document and click on the logDocument component and click OK
Click Save
Click System Management > Connectors and re-run the country connector

When it is complete, open the following file in a text editor:

 <data-agent-dir>\projects\Factbook\default\logs\logs-local\docs-ingestPostProcess.logDocument-17000.xml

The file will contain every document ingested during the connector's run. At this point in the workflow, fields will be tokenized such as the text field.

    <field name="text" language="en">
      <value>
        <tokens>['background'@20000002] ['sentence'@20000002] ['Discovered'#0~10, 'discover'@8#0~10] ['and'#11~14] ['claimed'#15~22, 'claim'@8#15~22] ['for'#23~26] ['location'@20000002#27~27] ['Spain'#27~32] ['location'@40000002#32~32] ['in'#33~35] ['number'@20000002#36~36] ['date'@20000002#36~36] ['1499'#36~40] ['date'@40000002#40~40] ['number'@40000002#40~40] [','@3#40~41] ['location'@20000002#42~42] ['Aruba'#42~47] ['location'@40000002#47~47] ['was'#48~51, 'be'@8#48~51] ['acquired'#52~60, 'acquire'@8#52~60] ['by'#61~63] ['the'#64~67] ['nationality'@20000002#68~68] ['Dutch'#68~73] ['nationality'@40000002#73~73] ['in'#74~76] ['number'@20000002#77~77] ['date'@20000002#77~77] ['1636'#77~81] ['date'@40000002#81~81] ['number'@40000002#81~81] ['.'@3#81~82] ['sentence'@40000002#82~82] ['sentence'@20000002#83~83] ['the'#83~86] ['island'#87~93] ['\''@3#93~94] ['s'#94~95] ['economy'#96~103] ['has'#104~107, 'have'@8#104~107] ['been'#108~112, 'be'@8#108~112] ['dominated'#113~122, 'dominate'@8#113~122] ['by'#123~125] ['number'@20000002#126~126] ['three'#126~131] ['number'@40000002#131~131] ['main'#132~136] ['industries'#137~147, 'industry'@8#137~147] ['.'@3#147~148] ['sentence'@40000002#148~148] ['sentence'@20000002#149~149] ['a'#149~150] ['19th'#151~155] ['century'#156~163] ['gold'#164~168] ['rush'#169~173] ['was'#174~177, 'be'@8#174~177] ['followed'#178~186, 'follow'@8#178~186] ['by'#187~189] ['prosperity'#190~200] ['brought'#201~208, 'bring'@8#201~208] ['on'#209~211] ['by'#212~214] ['the'#215~218] ['opening'#219~226, 'open'@8#219~226] ['in'#227~229] ['number'@20000002#230~230] ['date'@20000002#230~230] ['1924'#230~234] ['date'@40000002#234~234] ['number'@40000002#234~234] ['of'#235~237] ['an'#238~240] ['keyphrase'@20000002#241~241] ['oil'#241~244] ['refinery'#245~253] ['keyphrase'@40000002#253~253] ['.'@3#253~254] ['sentence'@40000002#254~254] ['sentence'@20000002#255~255] ['the'#255~258] ['last'#259~263] ['decades'#264~271, 'decade'@8#264~271] ['of'#272~274] ['the'#275~278] ['20th'#279~283] ['century'#284~291] ['saw'#292~295, 'see'@8#292~295] ['a'#296~297] ['boom'#298~302] ['in'#303~305] ['the'#306~309] ['tourism'#310~317] ['industry'#318~326] ['.'@3#326~327] ['sentence'@40000002#327~327] ['sentence'@20000002#328~328] ['location'@20000002#328~328] ['Aruba'#328~333] ['location'@40000002#333~333] ['seceded'#334~341, 'secede'@8#334~341] ['from'#342~346] ['the'#347~350] ['location'@20000002#351~351] ['Netherlands'#351~362] ['Antilles'#363~371] ['location'@40000002#371~371] ['in'#372~374] ['number'@20000002#375~375] ['date'@20000002#375~375] ['1986'#375~379] ['date'@40000002#379~379] ['number'@40000002#379~379] ['and'#380~383] ['became'#384~390, 'become'@8#384~390] ['a'#391~392] ['separate'#393~401] [','@3#401~402] ['autonomous'#403~413] ['member'#414~420] ['of'#421~423] ['the'#424~427] ['Kingdom'#428~435] ['of'#436~438] ['the'#439~442] ['location'@20000002#443~443] ['Netherlands'#443~454] ['location'@40000002#454~454] ['.'@3#454~455] ['sentence'@40000002#455~455] ['sentence'@20000002#456~456] ['Movement'#456~464] ['toward'#465~471] ['full'#472~476] ['independence'#477~489] ['was'#490~493, 'be'@8#490~493] ['halted'#494~500, 'halt'@8#494~500] ['at'#501~503] ['location'@20000002#504~504] ['Aruba'#504~509] ['location'@40000002#509~509] ['\''@3#509~510] ['s'#510~511] ['request'#512~519] ['in'#520~522] ['number'@20000002#523~523] ['date'@20000002#523~523] ['1990'#523~527] ['date'@40000002#527~527] ['number'@40000002#527~527] ['.'@3#527~528] ['sentence'@40000002#528~528] ['background'@40000002#528~528]</tokens>
        <string>Discovered and claimed for Spain in 1499, Aruba was acquired by the Dutch in 1636. the island's economy has been dominated by three main industries. a 19th century gold rush was followed by prosperity brought on by the opening in 1924 of an oil refinery. the last decades of the 20th century saw a boom in the tourism industry. Aruba seceded from the Netherlands Antilles in 1986 and became a separate, autonomous member of the Kingdom of the Netherlands. Movement toward full independence was halted at Aruba's request in 1990.</string>
      </value>
    </field>

The closing <documents> tag is NOT added until Attivio shuts down.

3. Create an Attivio SDK project

The Attivio Java SDK allows custom code (most commonly ingestion or query transformers) to be added to your Attivio projects. In the next several steps, we will create a custom module which contains a transformer which will alter the tokens of our incoming documents.

Navigate to the directory where you want to create your module. This should be a separate location from the Attivio installation or any Attivio projects you have created.

cd C:\attivio\sdkprojects\

Create a new module using the Maven archetype command:

 mvn archetype:generate -DarchetypeGroupId=com.attivio.platform.archetypes -DarchetypeArtifactId=attivio-archetype-module -DarchetypeVersion=5.6.1.0

This will download a number of artifacts and prompt you for the following:

groupId - enter 'com.attivio.platform'
artifactId: enter 'piglatin'
package - accept the default package
Confirm the settings by typing 'Y'

This command will result in a module project directory as follows:

│   dist-assembly.xml
│   pom.xml
│
└───src
    ├───main
    │   ├───java
    │   │   └───com
    │   │       └───attivio
    │   │           └───platform
    │   │               └───piglatin
    │   │                       ModuleInit.java
    │   │                       SampleAttivioRunnable.java
    │   │                       SampleDocumentModifyingTransformer.java
    │   │                       SampleFieldValueCreatingTransformer.java
    │   │                       SampleQueryRewriteTransformer.java
    │   │                       SampleQueryTransformer.java
    │   │                       SampleResponseTransformer.java
    │   │
    │   └───resources
    │       │   attivio.module.json
    │       │
    │       └───piglatin
    │               beans.xml
    │               features.xml
    │               module.xml
    │               piglatin.properties
    │
    └───test
        ├───java
        │   └───com
        │       └───attivio
        │           └───platform
        │               └───piglatin
        │                       SampleAttivioRunnableTest.java
        │                       SampleDocumentModifyingTransformerTest.java
        │                       SampleFieldValueCreatingTransformerTest.java
        │                       SampleQueryRewriterTransformerTest.java
        │                       SampleQueryTransformerTest.java
        │                       SampleResponseTransformerTest.java
        │
        └───resources
                attivio.test.json

A number of sample source code files and their respective tests are included by default. Feel free to review them, such as the SampleDocumentModifyingTransformer.java class.

Import the piglatin project into Eclipse:

In Eclipse, choose File > Import.
Expand the Maven group and select Existing Maven Projects and click Next.
The piglatin project's pom.xml file should be displayed and selected. Click Finish.
In Eclipse's Package Explorer, expand the src/main/java > com.attivio.platform.piglatin package.
Right-click the com.attivio.platform.piglatin package and select New > Class.
Name the new class TranslateToPigLatin.
Click Finish.

Edit the class so that it contains the following minimal code. The processDocument method is where we can intercept each IngestDocument and loop through its fields and modify each field's tokens.

package com.attivio.platform.piglatin;

import java.util.ArrayList;
import java.util.List;
import com.attivio.sdk.AttivioException;
import com.attivio.sdk.ingest.*;
import com.attivio.sdk.token.Token;
import com.attivio.sdk.token.Position;
import com.attivio.sdk.token.TokenAnnotation;
import com.attivio.sdk.token.TokenList;
import com.attivio.sdk.server.component.configuration.HasInputListProperty;
import com.attivio.sdk.server.component.ingest.DocumentModifyingTransformer;
public class TranslateToPigLatin implements DocumentModifyingTransformer, HasInputListProperty{
	protected List<String> input = new ArrayList<String>();
	
	public boolean processDocument(IngestDocument doc)throws AttivioException{
		for(String field : input){
			if(doc.containsField(field)){
				IngestField allValues = doc.getField(field);
				
				for(IngestFieldValue value : allValues){
					TokenList tokenList = value.getTokenList();
					if(tokenList == null)
						continue;
					for(Position position : tokenList.positions()){
						final Token token = position.get();
						String str = token.toString();
						String pigLatinString = pigLatin(str);
						if(!pigLatinString.equals(str)){
							Token newToken = token.clone();
							newToken.setValue(pigLatinString);
							newToken.setAnnotation(TokenAnnotation.TOKENIZED);
							position.add(newToken);
						}
					}
				}
			}
		}
		return true;
	}
	/**
     * Method to translate a sentence word by word.
     *
     * @param s
     *            The sentence in English
     * @return The pig latin version
     */
	
	private String pigLatin(String s) {
        String latin = "";
        int i = 0;
        while (i < s.length()) {
            // Take care of punctuation and spaces
            while (i < s.length() && !isLetter(s.charAt(i))) {
                latin = latin + s.charAt(i);
                i++;
            }
            // If there aren't any words left, stop.
            if (i >= s.length())
                break;
            // Otherwise we're at the beginning of a word.
            int begin = i;
            while (i < s.length() && isLetter(s.charAt(i))) {
                i++;
            }
            // Now we're at the end of a word, so translate it.
            int end = i;
            latin = latin + pigWord(s.substring(begin, end));
        }
        return latin;
    }
	/**
     * Method to test whether a character is a letter or not.
     *
     * @param c
     *            The character to test
     * @return True if it's a letter
     */
    private static boolean isLetter(char c) {
        return ((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z'));
    }
    /**
     * Method to translate one word into pig latin.
     *
     * @param word
     *            The word in english
     * @return The pig latin version
     */
    private static String pigWord(String word) {
        int split = firstVowel(word);
        // return word.substring(split)+"-"+word.substring(0, split)+"ay";
        return word.substring(split) + word.substring(0, split) + "ay";
    }
    /**
     * Method to find the index of the first vowel in a word.
     *
     * @param word
     *            The word to search
     * @return The index of the first vowel
     */
    private static int firstVowel(String word) {
        word = word.toLowerCase();
        for (int i = 0; i < word.length(); i++)
            if (word.charAt(i) == 'a' || word.charAt(i) == 'e'
                    || word.charAt(i) == 'i' || word.charAt(i) == 'o'
                    || word.charAt(i) == 'u')
                return i;
        return 0;
    }
    @Override
    public List<String> getInput() {
        return input;
    }
    @Override
    public void setInput(List<String> input) {
        this.input = input;
    }
}

Edit the src/main/resources/attivio.module.json file to add the new component:

{
  "name":"piglatin",
  "moduleVersion":"${project.version}",
  "description":"A description of my module",
  "initClassName":"com.attivio.platform.piglatin.ModuleInit",
  "components": [
    "com.attivio.platform.piglatin.SampleFieldValueCreatingTransformer",
    "com.attivio.platform.piglatin.TranslateToPigLatin"
  ],
  "newFiles": {
    "lib/piglatin.jar":"lib/piglatin-${project.version}.jar"
  }
}

Build the module using Maven:

cd C:\attivio\sdkprojects\piglatin
mvn clean install
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.290 s
[INFO] Finished at: 2018-06-04T15:49:09-04:00
[INFO] Final Memory: 27M/272M
[INFO] ------------------------------------------------------------------------

This will create a new file in the target directory of your module named piglatin-0.1.0-SNAPSHOT.zip. Next, we'll install this module into our Factbook project.

4. Install the piglatin Module

Run the following command to install the piglatin module:

 <install-dir>\bin\aie-exec modulemanager -i file:///c:/attivio/sdkprojects/piglatin/target/piglatin-0.1.0-SNAPSHOT.zip

Confirm the module has been installed:

 <install-dir>\bin\aie-exec modulemanager -l
 
Name            Version        User      Installed On        Description
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
cloudsupport    1.0.1          localuser 2018-05-24T13:00:27 Adds support for interacting with Cloud services
module-alm      1.0.1.aie55    localuser 2018-05-24T12:59:57 Provides English language advanced linguistics capabilities, including language detection and entity extraction
piglatin        0.1.0-SNAPSHOT apaquette 2018-06-04T16:06:13 A description of my module
searchanalytics 1.0.1          localuser 2018-05-24T13:00:05 Archive, scan and query search information
searchui        1.0.1          localuser 2018-05-24T13:00:14 Used to host the Search UI application from within Attivio nodes
webcrawler      1.0.4          localuser 2018-05-24T13:00:21 Configure connectors to crawl web content and process the pages as ingest documents.

Incrementally add the piglatin module to the Factbook project:

<install-dir>\bin\createproject.exe --name Factbook -i -m piglatin -o C:\attivio\projects

Open the Attivio CLI:

<install-dir>\bin\aie-cli.exe -p C:\attivio\projects\Factbook

Type the update command and hit Enter

Type the deploy command and hit Enter

Once the project is running again, move on to the next step.

5. Add an instance of the TranslateToPigLatin component to the workflow

Click System Management > Palette
Click New
Type TranslateToPigLatin in the search box and click Filter
Expand Document Transformers and click the TranslateToPigLatin component type that is found and click OK
In the popup that appears enter the following:
Field Value
Name translateToPigLatin
Input Fields title
text
Click Save
Click System Management > Workflows > Ingest
Click ingestPostProcess
Click the Add Existing Component Button
Type translateToPigLatin in the search box and click Filter
Expand Platform Components > Document and click on the translateToPigLatin component and click OK
Select the translateToPigLatin component and click the Move Up button to move it before the logDocument component
Click Save
Click System Management > Connectors and re-run the country connector

When it is complete, re-open the following file in a text editor:

 <data-agent-dir>\projects\Factbook\default\logs\logs-local\docs-ingestPostProcess.logDocument-17000.xml