Overview

Field Expressions increase the usability of field values found in search results.  For instance, a multi-value numeric field can be transformed into a sum or an average value.  A location can be transformed into a distance from a known point. A date in an unfamiliar format can be localized to suit the user's cultural conventions.  

FieldExpressions are a generic mechanism for:

• Controlling which fields will be returned from matching documents.
• Transforming returned values in various interesting ways.
• Adding metadata (such as a mean value or geographic distance from a point) to the returned document.
• Sorting returned documents in customized ways.  (Complex expressions can penalize performance.)
• Boosting match scores.  (Complex expressions can penalize performance.)

If no Field Expressions are specified for a query, all stored fields are returned.

You can experiment with Field Expressions using the Debug Search page of the AIE Administrator. Type the Field Expression in the Fields field. Use the LEGACY XML output format for easier visual interpretation of the results.

View incoming links.

• Advanced Query Language ,
• Category Boost (Relevancy Feature) ,
• Creating Custom Field Expressions ,
• Creating Custom Response Transformers ,
• Dates and Date Formats ,
• Debug Search ,
• Field Boost (Relevancy Feature) ,
• Highlighting Results ,
• HTTP REST APIs ,
• Machine Learning Relevancy Overview ,
• Multi-Level Facet Guide ,
• Query and Response - Concepts and Tools ,
• Query Data and Content in Attivio ,
• Scope Search ,
• Search Application Example ,
• Sorting Results ,
• Streaming Query API ,
• XML REST API

Field Expression Syntax

Field Expressions are applied through the Java Client API or the JSON REST API. 

Java Client API

Field Expressions may be requested through various methods of the  QueryRequest  message, most often by using the addField method.

The examples on this page show the creation of the QueryRequest to provide context, followed by one or more lines of code that formulate the Field Expression and attach it to the QueryRequest.  

REST API

Field expressions can also be requested using the JSON REST API's fields parameter. This parameter introduces a comma-separated list of field expressions. 

The examples on this page begin with &fields= to provide context.  The full HTTP REST syntax of the examples looks more like this:

http://localhost:17000/books/search?q=table:books&lang=advanced&fields=title,author

This example returns all of the books in an AIE index, listing the title and author of each.

String Syntax

Field expressions in the REST API follow some string conventions that provide syntactic shortcuts in some circumstances.

String literals and constants can be represented in field expressions by wrapping text in double quotes.

Single quotes can be used to quote field literals.

Examples:

• fieldName – a field name == new StoredField("fieldName")
• 'fieldName' – a quoted field name == new StoredField("fieldName")
• "string literal" – a constant string == new ConstantExpression("string literal")
• 'field name' – a quoted field name == new StoredField("field name")

Alias Example

Each field expression can be given an alias, much like the SELECT statement in SQL. This alias can be specified by saying fieldexpression AS alias. The alias will be the name of the field returned in response documents for this field expression. If no alias is specified for the field, AIE will generate a field name.

# Return the sum of numeric fields a and b as "A-plus-B"
&fields=ADD(a, b) AS "A-plus-B"

This expression adds a new field ("A-plus-B") to the returned document.  

Multiple Expressions Example

# Request fields a, b, and c, as well as the result of the ADD field expression over a and b (as "A-plus-B")
&fields=a,b,c,ADD(a, b) AS "A-plus-B"

This returns three stored fields from the index (a, b and c) plus one new field ("A-plus-B") that has been generated from the stored information.

General Functions

• StoredField - Request the value(s) of a stored field for a document.
• Constant - Request a constant value, often used as input to more complex field expressions.
• QueryParameter - Request a constant value that is replaced with the value for an arbitrary query parameter on the QueryRequest.
• Cast - Convert the data type of a field expression.
• ClauseContext - Request that field expression values be computed for a specified table in a join query.
• Union - Merge multiple fields into a multi-value field.

Index Statistics

The following field expressions are used for returning statistical information related to the index:

• PartitionId - Returns the partition id for each document indicating which partition contains the document.
• NumDocs - Returns the total number of documents in the index (including deleted documents).
• FieldLength - Returns the field length for a field of a document (Approximate/Lossy).
• TotalTermFrequency - Returns the total term frequency for all terms in a field across all documents in the index (including deleted documents).
• ScopeFrequency - Returns the frequency of occurrences of a requested scope in a field for a document.
• TotalScopeFrequency - Returns the total frequency of all occurrences of a requested scope across all documents in the index (including deleted documents).

Math Functions

The following field expressions are used for computing mathematical functions over values.

• Binary Math - Collection of math functions that take 2 arguments, including basic arithmetic operations
  
• Unary Math - Collection of math functions that take a single argument, including many trig operations.
• StandardScore - Compute the z-score for a value.
• GeoDistance - Compute the distance between a point and a specified center point.
• GeoBoost - Compute a score based on the distance between a point and a specified center point.

String Functions

The following field expressions are used for manipulating string values:

• SubString - Extract a substring.
• Concat - Concatenate string values.
• MultiValueJoin - Join values of a multi-value field using a delimiter.
• UpperCase - Convert a string to upper case.
• LowerCase - Convert a string to lower case.
• TitleCase - Convert a string to title case.
• StringLength - Return the length of a string.
• Locate - Locate the position for a substring in a string value.
• Trim - Remove whitespace from the beginning and end of a string.
• RegexMatch - Check a string for a match against a specified regex.
• Collate - Apply a collation to a string value.

Date Functions

The following field expressions reformat and interpret dates:

• CurrentTime - Returns the current time.
• DateFormat - Format a date as a string.
• DateExtract - Extract a field from a date.
• DateAdd - Add a specified amount to a specified calendar field of a date value.
• DateDiff - Compute the difference between dates.

• Freshness - Compute a score based on document recency.

Logical Functions

• Compare - Comparison operators (==, <=, >=, >, <, !=)
• Logical - Logical operators (AND/OR/NOT).
• Coalesce - Returns the value for the first non-null argument.
• Condition - Branching condition operator
• Switch - Branching switch operator.

Aggregate Expressions

Aggregate Expressions are available for aggregating values using standard functions over multi-value fields.

• Aggregate - Aggregate operators

Matching and Scoring

These field expressions help us understand exactly what fields contributed to a match, and how a document's match score was derived.

• PhraseMatch - Returns true if a document contains a match for the specified phrase.
• PhraseScore - Returns the score for a match for a document for the specified phrase.
• ScoreExplain - Returns an explanation of how the score was produced for a document.
• DocBoost - Returns a boost value for documents requested.

Highlighting

Fields requested using the StoredField field expression will be highlighted if the query request has highlighting enabled and the field is configured to be highlighted. If the StoredField field expression is used as an argument for another field expression, it will not be highlighted. Use the Teaser field expression or the ScopeTeaser expression if you need to highlight values for a field that is an argument for a more complex field expression.

The Following Field Expressions are used for highlighting fields according to matching query terms:

• Teaser - Apply highlighting to a field.

• ScopeTeaser - Apply highlighting to a field, using a scope to determine fragment boundaries.

• ScopeField - Extract scopes from a field.

• MatchingTerms - Provides a list of terms for a document that match the query.

• MatchingFields - Provides a list of field names that indicate which fields contained the matches for the query.

User Defined Fields

Attivio supports the ability to extend field expressions. See Creating Custom Field Expressions for documentation on creating and working with custom field expressions.