Overview
All logging inside the Attivio Intelligence Engine (AIE) is managed through the
com.attivio.util.AttivioLoggerclass. The AttivioLogger provides methods to log information, warning and error messages, and requires all warnings and errors to have proper error codes.
AIE Logging Configuration
- Attivio Logger (this page): How to configure Appenders in the log4j configuration files.
- Configure System Logging: Enable the "configure and watch" logging feature.
- Configure Query Logging Services: Maintain a log of incoming queries.
- Logging: AIE Administrator logging features.
- Logging Level Settings: Temporaily reconfigure logging settings for individual components in the AIE Administrator.
- Log Viewer: View recent log entries from the AIE Administrator.
- Creating FATAL, WARN, and INFO System Events: Issuing System Events from custom java code.
View incoming links.
Logging Architecture
Once the Attivio Logger has finished preparing a message for logging, the message is sent to the underlying logging stack which is managed by SLF4j. For example, for some log messages, the Attivio Logger will merge error or warning message parameters into standard error or warning message templates before the resulting final message is passed to the underlying logging infrastructure.
By default, AIE ships with the log4j SLF4J adapter installed such that all messages are passed to Log4j configured loggers; however, any other underlying logging framework can be used via standard SLF4j configuration.
Log4j supports many different log appenders (components responsible for writing log messages to various output locations) out-of-the-box including logging to files, sockets, databases, unix syslog, and the Windows event log. All of the appenders can be configured in the <install_dir>/conf/log4j-node.xml file.
Configuration
The AIE Log4J configuration file file (<install_dir>/conf/log4j-node.xml) contains the default configuration for all AIE node logging in the log4j xml configuration format. For example, to enable logging at the TRACE level for a custom component, add the following to the configuration file:
<logger name="com.acme.MyClass"> <level value="TRACE"/> </logger>
Since Log4j controls log levels through a hierarchical configuration mechanism, the majority of the AIE logging is controlled by the single logger
configuration element.
Refer to the Log4j manual for more information on how hierarchical logger configuration works.
Changing the AIE primary logger
/level
element value to DEBUG or TRACE will cause much more information to be logged and could lead to performance and/or disk space issues if not properly configured and managed. DEBUG or TRACE level logging should be used with great care and only in non-production environments.
General Logging Properties
The following AIE properties can be set to control AIE logging behavior in addition to log4j-node.xml configuration.
The sample below is from the default attivio.properties file; so the values listed represent the default values for AIE. To change these properties, add the properties with the desired value to an AIE project's <project_dir>/conf/properties/core-app/attivio.core-app.properties file.
################################################################# logging log.directory=${attivio.project}/logs # whether or not to print stack traces on the console log.printStackTraces=false # the base level for all attivio classes log.level=INFO # set to DEBUG to see all documents as they flow through the ingest system log.level.ingest=WARN # set to TRACE to see low-level tracing of message flow log.level.messaging=INFO # maximum length for all log messages # log.platformMessage.length.max=200 org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger
Runtime Logging Level Changes
The AIE Administrator Logging Level Settings page supports changing the logging level for a single class with changes taking effect immediately.
In addition, a
Log4jServicesupports global changes to the log4j-node.xml file. If the configureAndWatch property of the Log4jService is set to true, then any changes to the log4j-node.xml file will be picked every 60 seconds and made live.
Primary Default Log Files
By default, AIE is configured to support two primary output log files. All AIE log files are organized into directories labeled by node-names ("local" below) in order to make it clear from which AIE node the files came. Log files can be found in the <data-agent>/projects/<project-name>/<environment-name>/logs/logs-<node-name>/ directories. The <data-agent> files will be found either in the location where you ran AIE Agent, or in the location you specified when you did so.
Log file | Contents | Configuration |
---|---|---|
aie-node.log | All log messages at an info level, by default this should be the same as the console output. | <install-dir>/conf/log4j-node.xml <appender name="filesync" class="org.apache.log4j.DailyRollingFileAppender"> <param name="Encoding" value="UTF-8"/> <param name="Threshold" value="TRACE"/> <param name="DatePattern" value="'.'yyyy-MM-dd"/> <param name="File" value="${aie.logfile}"/> <param name="Append" value="true"/> <layout class="com.attivio.platform.util.NoStackTracePatternLayout"> <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/> </layout> </appender> |
aie-node.error.log | Contains all log messages at a warning or error level. In addition, includes the full stack traces for any exceptions for further debugging | <install-dir>/conf/log4j-node.xml <appender name="errorfilesync" class="org.apache.log4j.DailyRollingFileAppender"> <param name="Encoding" value="UTF-8"/> <param name="Threshold" value="WARN"/> <param name="DatePattern" value="'.'yyyy-MM-dd"/> <param name="File" value="${attivio.log.directory}/aie-node.error.log"/> <param name="Append" value="true"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/> </layout> </appender> |
Query Log
By default AIE provides a query log using the following log4j-node.xml configuration. The resulting file is <data-agent>/projects/<project-name>/<environment-name>/logs/logs-<node-name>/querylogs/queries.log.
<appender name="queryresponselogfilesync" class="org.apache.log4j.DailyRollingFileAppender"> <param name="Encoding" value="UTF-8"/> <param name="Threshold" value="INFO"/> <param name="File" value="${attivio.log.directory}/querylogs/queries.log"/> <param name="Append" value="true"/> <param name="DatePattern" value="'.'yyyy-MM-dd-HH"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/> </layout>
By default, the query log is in the following format:
workflow clientid ip [date] "request" error_code rows workflow_time search_time facet_time highlight_time result_time
Column | Description |
---|---|
workflow | Workflow to which each query was sent. |
clientid | ID of the query client connected to AIE. |
ip | IP address from which the query request was issued. |
[date] | Date of the query request |
request | The full CGI serialized query string for each query. It is often very lengthy. |
error_code | Will be 'OK' for successful queries or will contain the corresponding error code for unsuccessful queries. |
rows | The total number of rows (documents) matched by the query . The query hits parameter limits the number of rows returned in the query response object. |
workflow_time | Total time the query spends in all the workflow stages. (See diagram below). |
search_time | The time (in ms) to execute the search and return a list of matching documents. |
facet_time | The time (in ms) required to compute facets if any were requested. |
highlight_time | How much time is spent highlighting results. |
result_time | The time required to fetch the document results from disk. |
The relationships between the timing values is depicted below where the child values are included in parent values.
This format is similar to a web server request log file format in which workflow
maps to host
, rows
to bytes
, and additional columns for search/facet/result time are present. This format allows the query logs to be parsed and analyzed by a variety of web server traffic analysis tools.
Which fields are logged, as well as their their order, delimiter, and enclosing character may be customized in the
QueryLogServiceservice configuration.
Common Logging Questions
How do I change the default level for all Attivio log messages?
The <install_dir>/conf/log4j-node.xml file can be edited or copied to a <project_dir>/resources directory and edited in order to change the logging configuration. The default com.attivio.* logging levels are controlled via an attivio property attivio.log.level which can be set in a project's <project_name>.properties file.
How do I configure the log files to roll over on a different frequency?
Log4j support rolling files over based on two different types of logic.
Class | Description | Sample configuration |
---|---|---|
Rolls the underlying file at a user chosen file size based frequency. | <install-dir>/conf/log4j-node.xml <appender name="filesync" class="org.apache.log4j.RollingFileAppender"> <param name="Encoding" value="UTF-8"/> <param name="Threshold" value="TRACE"/> *<param name="MaximumFileSize" value="50MB"/>* <param name="File" value="myLogFile.log"/> <param name="Append" value="true"/> <layout class="com.attivio.util.AttivioLogger$NoStackTracePatternLayout"> <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/> </layout> </appender> | |
Rolls the underlying file at a user chosen time based frequency. | <install-dir>/conf/log4j-node.xml <appender name="filesync" class="org.apache.log4j.DailyRollingFileAppender"> <param name="Encoding" value="UTF-8"/> <param name="Threshold" value="TRACE"/> *<param name="DatePattern" value="'.'yyyy-MM-dd"/>* <param name="File" value="myLogFile.log"/> <param name="Append" value="true"/> <layout class="com.attivio.util.AttivioLogger$NoStackTracePatternLayout"> <param name="ConversionPattern" value="%d %-5p %c{1} [%t] - %m%n"/> </layout> </appender> |
The AIE primary and query logs use the DailyRollingFileAppender by default.