Page tree
Skip to end of metadata
Go to start of metadata

Overview

All logging inside the Attivio Intelligence Engine (AIE) is managed through the

com.attivio.util.AttivioLogger

class. The AttivioLogger provides methods to log information, warning and error messages, and requires all warnings and errors to have proper error codes.  

AIE Logging Configuration

 

View incoming links.

Logging Architecture

Once the Attivio Logger has finished preparing a message for logging, the message is sent to the underlying logging stack which is managed by SLF4j. For example, for some log messages, the Attivio Logger will merge error or warning message parameters into standard error or warning message templates before the resulting final message is passed to the underlying logging infrastructure.

By default, AIE ships with the log4j SLF4J adapter installed such that all messages are passed to Log4j configured loggers; however, any other underlying logging framework can be used via standard SLF4j configuration.

Log4j supports many different log appenders (components responsible for writing log messages to various output locations) out-of-the-box including logging to files, sockets, databases, unix syslog, and the Windows event log. All of the appenders can be configured in the <install_dir>/conf/log4j-node.xml file.

Configuration

The AIE Log4J configuration file file (<install_dir>/conf/log4j-node.xml) contains the default configuration for all AIE node logging in the log4j xml configuration format. For example, to enable logging at the TRACE level for a custom component, add the following to the configuration file:

<install_dir>/conf/log4j-node.xml
<logger name="com.acme.MyClass">
  <level value="TRACE"/>
</logger>

Since Log4j controls log levels through a hierarchical configuration mechanism, the majority of the AIE logging is controlled by the single logger configuration element.

Refer to the Log4j manual for more information on how hierarchical logger configuration works.

Changing the AIE primary logger/level element value to DEBUG or TRACE will cause much more information to be logged and could lead to performance and/or disk space issues if not properly configured and managed. DEBUG or TRACE level logging should be used with great care and only in non-production environments.

General Logging Properties

The following AIE properties can be set to control AIE logging behavior in addition to log4j-node.xml configuration.

The sample below is from the default attivio.properties file; so the values listed represent the default values for AIE. To change these properties, add the properties with the desired value to an AIE project's <project_dir>/conf/properties/core-app/attivio.core-app.properties file.

<install_dir>/conf/attivio.properties
################################################################# logging
log.directory=${attivio.project}/logs
# whether or not to print stack traces on the console
log.printStackTraces=false
# the base level for all attivio classes
log.level=INFO
# set to DEBUG to see all documents as they flow through the ingest system
log.level.ingest=WARN
# set to TRACE to see low-level tracing of message flow
log.level.messaging=INFO
# maximum length for all log messages
# log.platformMessage.length.max=200
org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger

Runtime Logging Level Changes

The AIE Administrator Logging Level Settings page supports changing the logging level for a single class with changes taking effect immediately.

In addition, a

Log4jService

supports global changes to the log4j-node.xml file. If the configureAndWatch property of the Log4jService is set to true, then any changes to the log4j-node.xml file will be picked every 60 seconds and made live.

Primary Default Log Files

By default, AIE is configured to support two primary output log files. All AIE log files are organized into directories labeled by node-names ("local" below) in order to make it clear from which AIE node the files came. Log files can be found in the <data-agent>/projects/<project-name>/<environment-name>/logs/logs-<node-name>/ directories.  The <data-agent> files will be found either in the location where you ran AIE Agent, or in the location you specified when you did so. 

Log file

Contents

Configuration

aie-node.log

All log messages at an info level, by default this should be the same as the console output.

<install-dir>/conf/log4j-node.xml
<appender name="filesync" class="org.apache.log4j.DailyRollingFileAppender">
    <param name="Encoding" value="UTF-8"/>
    <param name="Threshold" value="TRACE"/>
    <param name="DatePattern" value="'.'yyyy-MM-dd"/>
    <param name="File" value="${aie.logfile}"/>
    <param name="Append" value="true"/>
    <layout class="com.attivio.platform.util.NoStackTracePatternLayout">
      <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/>
    </layout>
  </appender>

aie-node.error.log

Contains all log messages at a warning or error level. In addition, includes the full stack traces for any exceptions for further debugging

<install-dir>/conf/log4j-node.xml
  <appender name="errorfilesync" class="org.apache.log4j.DailyRollingFileAppender">
    <param name="Encoding" value="UTF-8"/>
    <param name="Threshold" value="WARN"/>
    <param name="DatePattern" value="'.'yyyy-MM-dd"/>
    <param name="File" value="${attivio.log.directory}/aie-node.error.log"/>
    <param name="Append" value="true"/>
    <layout class="org.apache.log4j.PatternLayout">
      <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/>
    </layout>
  </appender>

Query Log

By default AIE provides a query log using the following log4j-node.xml configuration. The resulting file is <data-agent>/projects/<project-name>/<environment-name>/logs/logs-<node-name>/querylogs/queries.log.

<install-dir>/conf/log4j-node.xml
  <appender name="queryresponselogfilesync" class="org.apache.log4j.DailyRollingFileAppender">
    <param name="Encoding" value="UTF-8"/>
    <param name="Threshold" value="INFO"/>
    <param name="File" value="${attivio.log.directory}/querylogs/queries.log"/>
    <param name="Append" value="true"/>
    <param name="DatePattern" value="'.'yyyy-MM-dd-HH"/>  
    <layout class="org.apache.log4j.PatternLayout">
      <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/>
    </layout>

By default, the query log is in the following format:

workflow clientid ip [date] "request" error_code rows workflow_time search_time facet_time highlight_time result_time

Column

Description

workflow

Workflow to which each query was sent.

clientid

ID of the query client connected to AIE.

ip

IP address from which the query request was issued.

[date]

Date of the query request

request

The full CGI serialized query string for each query. It is often very lengthy.

error_code

Will be 'OK' for successful queries or will contain the corresponding error code for unsuccessful queries.

rows

The total number of rows (documents) matched by the query . The query hits parameter limits the number of rows returned in the query response object.

workflow_time

Total time the query spends in all the workflow stages. (See diagram below).

search_time

The time (in ms) to execute the search and return a list of matching documents.

facet_time

The time (in ms) required to compute facets if any were requested.

highlight_time

How much time is spent highlighting results.

result_time

The time required to fetch the document results from disk.

The relationships between the timing values is depicted below where the child values are included in parent values.

QueryLogTimes

This format is similar to a web server request log file format in which workflow maps to host, rows to bytes, and additional columns for search/facet/result time are present. This format allows the query logs to be parsed and analyzed by a variety of web server traffic analysis tools.

Which fields are logged, as well as their their order, delimiter, and enclosing character may be customized in the

QueryLogService

service configuration.

 

Common Logging Questions

How do I change the default level for all Attivio log messages?

The <install_dir>/conf/log4j-node.xml file can be edited or copied to a <project_dir>/resources directory and edited in order to change the logging configuration. The default com.attivio.* logging levels are controlled via an attivio property attivio.log.level which can be set in a project's <project_name>.properties file.

How do I configure the log files to roll over on a different frequency?

Log4j support rolling files over based on two different types of logic.

Class

Description

Sample configuration

RollingFileAppender

Rolls the underlying file at a user chosen file size based frequency.

<install-dir>/conf/log4j-node.xml
<appender name="filesync" class="org.apache.log4j.RollingFileAppender">
  <param name="Encoding" value="UTF-8"/>
  <param name="Threshold" value="TRACE"/>
  *<param name="MaximumFileSize" value="50MB"/>*
  <param name="File" value="myLogFile.log"/>
  <param name="Append" value="true"/>
  <layout class="com.attivio.util.AttivioLogger$NoStackTracePatternLayout">
    <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/>
  </layout>
</appender>

DailyRollingFileAppender

Rolls the underlying file at a user chosen time based frequency.

<install-dir>/conf/log4j-node.xml
<appender name="filesync" class="org.apache.log4j.DailyRollingFileAppender">
  <param name="Encoding" value="UTF-8"/>
  <param name="Threshold" value="TRACE"/>
  *<param name="DatePattern" value="'.'yyyy-MM-dd"/>*
  <param name="File" value="myLogFile.log"/>
  <param name="Append" value="true"/>
  <layout class="com.attivio.util.AttivioLogger$NoStackTracePatternLayout">
    <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/>
  </layout>
</appender>

The AIE primary and query logs use the DailyRollingFileAppender by default.

  • No labels