Overview

The evolution of an AIE project from initial experimentation to final deployment often involves an escalating series of topologies as the default, single-node topology adds partitions, nodes and servers to scale up for production.  The AIE environment feature lets us set up one project (with a single set of files for configuration and Java customization) that can be run using any number of distinct topologies.  We can switch from one environment to another transparently at any time. 

Each "environment" consists of:

• Node, nodeset, configuration server, store, and performance monitor configurations in a dedicated topology-nodes.xml file.
• There can be environment-specific indexWriter, indexObserver, and indexShare rows in the Index.index.xml file.
• Optionally, there can be environment-specific properties in an environment.properties file.
• The optional properties may be referenced in the Index.index.xml file and in the project's configuration.xml file.

View incoming links.

• Attivio Platform ,
• Developing in AIE - Concepts and Tools ,
• Multi-Node Topologies

Hardware / Environments

A typical project will implement several (if not all) of the following environments.  You should consider some similar set of environments in your development plan.

Test Procedure

Given that you have development, QA, UAT, and production environments set up, you can establish a testing/validation routine that certifies candidate builds along the following path:

1. Deploy builds to the development system on a daily or on-demand basis.
2. Perform functional testing on the development system (automated and/or manual).
3. If the build passed the development tests, you can deploy it to the QA system.  Otherwise wait for the next build and start over.
4. Perform functional and integration testing on the QA system.
5. If QA testing is successful, you can deploy the build to the UAT system. 
6. Perform User Acceptance Testing and scale/load testing on the UAT system.
7. If UAT system testing is successful, deploy the build in the production environment.

Example Multi-Environment Project

The next few sections demonstrate how to set up a project with four different topologies:

• Default (single-node) topology (similar to the Quick Start Tutorial).
• Four-node topology with index replication (from Multi-Node Topologies).
• Six-node topology with replication and index sharing (from Configure the Index).
• Six-node topology with six partitions spread across six nodes. 

These environments have no great virtue of themselves, but in combination they pose an appropriate multi-environment configuration challenge. The configuration files are in conf.zip, which is attached to this page.

Configure the Index

The demonstration involves setting up a project with four different topologies and index configurations.  The most challenging part of the project is setting up a single Index.index.xml feature file to describe four very different index configurations simultaneously.

When setting up Index.index.xml, keep these points in mind:

• Each indexWriter, indexObserver, and indexShare element must be assigned to a specific environment (or set of environments) using an environment property.
• List all of the indexWriters first, followed by all of the indexObservers, and then by all of the indexShares. 
• Every indexWriter must have a name.  The Index.index.xml file for the default environment uses a single anonymous indexWriter declaration.  This syntactic shortcut cannot be used in a more complex index file.

How Many Partitions?

When setting up environments that experiment with different numbers of partitions and different index node configurations, it is important to understand the relationship between partitions and nodes.  See Configure the Index.

The number of partitions is set (for the project, that is, for all environments) in the partitionSet element of the Index.index.xml file:

  <f:index compatibleNames="false" enabled="true" name="index" nio="false">
    <f:partitionSet size="2">

It is possible to vary the number of partitions in your environments by setting the size attribute to an environment property.  Each environment has its own environment.properties file.  Set the variable in Index.index.xml like this:

  <f:index compatibleNames="false" enabled="true" name="index" nio="false">
    <f:partitionSet size="${partitions}">

In each environment, set the property like this:

   partitions=2

This mechanism lets you vary the number of partitions among your environments, even through all environments must use the same Index.index.xml file.

Remember that you cannot casually change the number of partitions in an index after performing any ingestion.  However, you can delete the environment's data (including its current index), change the partitions property, and redeploy the environment to build an all-new index. First, stop all servers for the project.  Stop the AIE Agent.  Delete all contents of the <data-agent>\projects\<project-name>\<environment-name> directory.  Start AIE Agent. Start all servers, passing in the appropriate environment using the -e parameter.

Configure the IndexWriters

The indexWriter rows for all environments must be configured together near the top of the partitionset element of Index.index.xml.  Each indexWriter must be uniquely named, and each must have an environments property, or AIE will refuse to start.

    <f:partitionSet size="${partitions}">
      <f:indexWriter name="default-indexer" nodeset="indexers">
        <f:environments>default</f:environments>
      </f:indexWriter>
      <f:indexWriter name="four-node-indexer" nodeset="indexers">
        <f:environments>four-node-system</f:environments>
      </f:indexWriter>
      <f:indexWriter name="six-node-indexer-1" nodeset="node1" search="false">
        <f:environments>six-node-system</f:environments>
      </f:indexWriter>
      <f:indexWriter name="six-node-indexer-2" nodeset="node2" search="false">
        <f:environments>six-node-system</f:environments>
      </f:indexWriter>
      <f:indexWriter name="distributed-partitions" nodeset="indexers">
        <f:environments>distributed-partitions</f:environments>
      </f:indexWriter>

The indexWriters of the various environments have no interaction with one another, except that they must all be listed together before any indexObservers or indexShares are listed.

Configure the IndexObservers

After the indexWriters are declared, the indexObservers for all environments must be declared.  IndexObservers all must identify a source, which the the name of an indexWriter in the same environment.

      <f:indexObserver nodeset="observers" source="four-node-indexer">
        <f:environments>four-node-system</f:environments>
      </f:indexObserver>
      <f:indexObserver nodeset="node5" source="six-node-indexer-1">
        <f:environments>six-node-system</f:environments>
      </f:indexObserver>
      <f:indexObserver nodeset="node6" source="six-node-indexer-2">
        <f:environments>six-node-system</f:environments>
      </f:indexObserver>

Configure the IndexShares

Once all the indexObservers have been declared, we finish up by declaring all of the indexShares from all environments.  That completes the partitionSet.

      <f:indexShare nodeset="node3" source="six-node-indexer-1">
        <f:environments>six-node-system</f:environments>
      </f:indexShare>
      <f:indexShare nodeset="node4" source="six-node-indexer-2">
        <f:environments>six-node-system</f:environments>
      </f:indexShare>
    </f:partitionSet>

Configure Configuration.xml

The project's configuration file is <project-dir>\conf\configuration.xml. It is outside the environment mechanism, so the same file is used by all environments.

It is not necessary to make any alterations to this file, but there are situations where adding environment properties to the file can enhance the environments.  This is a hypothetical example:

  <configuration projectName="Factbook">
    <haft faultTolerant="${fault.tolerant}"/>
    <message-queue size="3"/>
    <default-performance maxIngestInstances="0" maxQueryInstances="0"/>
    <security keyStoreFileName="${key.filename}"
       trustStoreFileName="${trust.filename}"
       keyPassword="${key.password}"
       storePassword="${store.password}"
       trustPassword="${trust.password}"
       useSSL="${security.useSSL}">
    <default-authentication-provider authentication-provider-ref="${auth.provider}"/>
    <administrators>
      <principal realm="aie" principal="aieadmin" name="aieadmin" type="USER" 
         password="${admin.password}"/>
    </administrators>

The corresponding environment.properties file might resemble this one:

fault.tolerant=false
key.filename=attivio.ks
trust.filename=attivio.ks
keyPassword=*{QeDBUhv9roY=}
store.password=*{QeDBUhv9roY=}
trust.password=*{QeDBUhv9roY=}
security.useSSL=*{QeDBUhv9roY=}
auth.provider=myAuthProviderBeanName
admin.password=*{QeDBUhv9roY=}

Note that the passwords can be lightly encrypted to avoid posting them in the clear in a configuration file.  The *{} syntax forces an evaluation of the encrypted string when the property is loaded. See Encrypting User Passwords.

Deploying the Environments

Any time you edit AIE project source files, you must re-deploy the project to the configuration servers before you will see any different at runtime. You can do this using the deploy command in either AIE-CLI.  Note that if you have also created dynamic changes, by editing connectors or components in the AIE Administration editors, you will need to update the project and resolve all conflicts before deploying.

Source Control Integration

The project directory tree is designed to let you check in the entire project into source control. Projects can be checked out and run as is.

Before backing up or committing project files to source control, you must capture Dynamic Configuration changes by using the Update Local Configuration feature of AIE-CLI. This copies dynamic changes from the configuration server(s) into XML configuration files in the project tree.