Release Date: 29 June 2019

This page summarizes the modules, new features, enhancements, and bug fixes included in the Attivio Platform 5.6.1 release, as well as listing any known issues.

Modules

Attivio Platform 5.6.1 installs the following module versions when the Full installation type (the default) is selected:

These modules are not installed if the Core installation type is selected.

New Features

Multi-Level Facets

Attivio Platform 5.6.1 adds support for multi-level faceting. This feature supports visualization of a hierarchy for the documents matching a query, and enables filtering documents using arbitrary levels of this hierarchy. As an example, documents with location data can be faceted using a country > state or province > city hierarchy, enabling users to easily refine their results (to show only documents from France or California or London, say) from a single hierarchical facet.

See the Multi-Level Facet Guide for more information.

Connector Admin UI

The new Connector Admin UI simplifies creation, editing, and management of connectors and provides insights into ingested data.

Connector SDK

The Attivio Java SDK now supports the implementation of custom connector scanner classes to deliver content into the Attivio workflow from any data source. The SDK specifies the interfaces and annotations the scanner should implement so that it can be configured and controlled from the Connector Admin UI or the legacy Administrator UI. The SDK also specifies Attivio exposed interfaces which the scanner can use to feed content and access Attivio services.

See the public SDK's Scanners page for more information on custom scanner development.

Customizable JRE for Attivio processes ("Bring Your Own JRE")

Users can now configure their own Java Runtime Environment (JRE) to use for running Attivio processes. Any Java SE 11 JRE can be specified.

See Updating Attivio JREs for more information.

Enhancements

OpenJDK 11

• PLAT-43014: The Attivio installer now includes OpenJDK 11.0.2 as the default Java Runtime Environment (JRE) for Attivio processes.

Improved Hadoop cluster stability and user experience

• PLAT-42943: Distribution of index containers across YARN nodes now includes logic to ensure High Availability.
• PLAT-43027: pmem enforcement changed to be false by default for Attivio Hadoop clusters.
• PLAT-43027: The CLI now validates project configurations to ensure that all YARN containers will be allocable when the index sliderPlacementPolicy property is set to its default value of 32 (anti-affinity).

• PLAT-43027: When deploying an index, the CLI now displays progress while adding the index's .tar file to the Slider .zip archive.

• PLAT-43027: The CLI no longer builds the Slider .zip archive twice (once for abc-index, once for index), instead updating the existing archive with relevant index configuration settings.

• PLAT-43027: The CLI displays status STOPPING for Hadoop services when a stop (or restart) has been requested but the service has not timed out yet in ZooKeeper.

• PLAT-43027: The CLI status command's output now sorts services on the same host alphabetically instead of randomly to improve readability.
• PLAT-43027: The CLI start hadoop and stop hadoop commands now use case-insensitive regular expressions, allow for multiple spaces between <hostRegexp> and <serviceRegex> instead of just one, and print out a message reporting whether any entries match the regular expression or not.

• PLAT-43027: Added resiliency to the HdfsFailoverController and HdfsNameNode services.

• PLAT-43027: Increased timeout periods for writes to HDFS journal log and start of new segments from 20 seconds to 40 seconds to better handle stress situations.

• PLAT-43269: The minimum timeout period for the ZooKeeper service was raised from 4 seconds to 20 seconds to provide more resiliency against SAN slowness.

• PLAT-43269: Users can now specify dataLogDir and other properties for ZooKeeper service.

• PLAT-43277: Enabled user configuration of abc-index memory usage, partition count, and row count.

Smaller memory footprint for clustered projects

• PLAT-43027: Reduced memory allocated for Slider app master to 512 MB for indexes of 16 or fewer partitions, or 1024 MB (1 GB) for indexes of more than 16 partitions
  Slider app master's memory allocations are controlled by these properties in attivio.core-app.properties:

  # slider app master memory, default is 512, unit are megabytes
  slider.am.memory=512
  # ... for larger indices (> 16 indexwriter/indexsearcher processes)
  slider.am.memory-large=1024

• PLAT-43027: Changed Attivio Hadoop cluster default yarn.scheduler.minimum-allocation-mb property value to 512 to permit smaller configurations.

• PLAT-43027: Changed Attivio Hadoop cluster default yarn.scheduler.maximum-allocation-mb property value from 3072 to 4096.

• PLAT-43027: Reduced default heap memory allocations for many services in Attivio Hadoop cluster:

  • HdfsFailoverController from 1024 MB to 256 MB.

  • HdfsNameNode from 5376 MB to 2048 MB.

  • YarnResourceManager from 2048 MB to 1024 MB.

  • YarnNodeManager from 4096 MB to 1024 MB.

  • HBaseMaster from 4096 MB to 1024 MB.

  • HBaseRegion from 3093 MB to 3072 MB.

• PLAT-43027: Reduced default HDFS replication factor from 3 to 2.

Other improvements

• PLAT-40938: Added payload support to trigger dictionaries.
• PLAT-41380: Optimized FacetQuery for text schema field type.
• PLAT-41808: Cleaned up REST APIs to reduce use of custom deserializers.
• PLAT-41811: Added user-selected FacetFilters to REST query response.
• PLAT-41912: Added multi-threaded facet execution.
• PLAT-41816: Facet interchanges between engine and query dispatcher now use JSON format to reduce I/O
• PLAT-42022: Autowarm queries are now executed in parallel.
• PLAT-42038: Static teaser field is not displayed in query results unless explicitly requested.
• PLAT-42058: Java 8 features are now used to simplify DelegatingFileSystem.
• PLAT-42068: Query requests now use binary serialization to reduce memory overhead (and improve performance) for reads and writes of query objects across network.
• PLAT-42085: Objects read from the query dispatcher are now buffered to streamline read operations.
• PLAT-42116: Added Ignore Scopes blacklist property to AddEntitySentiment transformer class. By default, this blacklist includes the number, date, extracteddate, time, and url scopes.

Bug Fixes

• PLAT-40376: Resolved an issue where REST Perfmon API returned "NaN" metrics values.
• PLAT-41495: Resolved an issue where NormalizeUnicode ingest transformer only processed surface tokens.
• PLAT-41562: Dictionary Administration web application no longer allows addition of terms to Derived-mode Entity dictionaries.
• PLAT-41755: Resolved an issue where the unclustered (HSQL) Attivio store could throw a NullPointerException when updating sent-documents count for a completed connector run.
• PLAT-41925: Resolved an issue where not all Attivio Hadoop services appeared in CLI when all were stopped.
• PLAT-41956: Resolved an issue which could cause the ExtractHashTags transformer class to throw a NullPointerException.
• PLAT-41988: Resolved an issue where connector history's sent-documents count for current execution record was incorrect for unclustered projects.
• PLAT-42006: Resolved an issue where CLI help zk command output pointed to non-existent URL.
• PLAT-42186: Removed deprecated addrow and removerow CLI commands. These functions are now accessed via options for the flexindex command.
• PLAT-43340: Resolved security vulnerabilities by upgrading to commons-compress-1.14.jar, commons-collections-3.2.2.jar, derby-10.12.1, and netty-3.9.9.Final library versions.
• PLAT-43509: Resolved an issue where the Publish Dictionary button was not properly disabled in the Rules Dictionary Administration web application.

• PLAT-43620: Resolved an issue which could cause deadlocks in the unclustered (HSQL) Attivio store.

• PLAT-43815: Removed deprecated sail module.

• PLAT-43930: Resolved an issue where the Ingest Audit Viewer's Show Details function could fail with a "Premature EOF" IOException

Known Issues

• PLAT-44000: If you update the Attivio JRE to point to any version other than Java SE 11, Attivio processes may display inconsistent startup failure messages.
  • Solution: Ensure that the Attivio JRE points to a Java SE 11 installation.
• PLAT-44001: The Excel Connector cannot read files from Excel versions prior to Excel 97/2000/XP/2003.
  • Solution: Use the excelcnv.exe tool shipped with recent versions of Microsoft Office to convert older Excel files to the modern .xlsx format prior to ingestion.