

Setting up EI Analytics for Observability¶

Follow the instructions given below to enable observability for your Micro Integrator deployment using EI Analytics.

EI-Analytics Observability

EI Analytics consists of two components: Server and Portal. The server processes the data streams that are sent from the Micro Integrator and publishes the statistics to a database. The portal reads the statistics published by the worker and displays the statistics. The server and portal are connected through the database.

This solution is more suitable if you already have an observability stack such as ELK, or if you want more business analytics and less operational observability. To select the most appropriate observability solution for your deployment, see Observability Deployment Strategy.

System requirements¶

You will be running three servers (EI Analytics server, EI Analytics portal, and the Micro Integrator) for this solution. Be sure that you have the required system specifications to run each server.

For the Analytics Server:

Memory

~ 4 GB per worker node
It is recommended to allocate 4 cores.
~ 2 GB is the initial heap (-Xms) required for the server startup. The maximum heap size is 4 GB (-Xmx)

Disk

~ 480 MB, excluding space allocated for log files and databases.

For the Analytics Portal:

Memory

~ 2 GB minimum, 4 GB Maximum
2 CPU cores minimum. It is recommended to allocate 4 cores.
~ 512 MB heap size. This is generally sufficient to process typical SOAP messages but the requirements vary with larger message sizes and the number of messages processed concurrently.

Disk

~ 480 MB, excluding space allocated for log files and databases.

For the Micro Integrator, see the installation prerequsites.

Download the servers¶

To download EI Analytics:
1. Go to the WSO2 Enterprise Integrator product page, click Download, and then go to the Other Resources section.
2. Click Integration Analytics to download the distribution.
Info

The location of your Analytics installation will be referred to as <EI_ANALYTICS_HOME>.
Download and install the Micro Integrator of EI 7.1.

Configuring the Integrator Analytics¶

EI Analytics contains two runtimes, namely server and portal. The server is responsible for the summarization of the collected data and the portal is responsible to represent the summarised data in the dashboards. Therefore, two separate JVMs are required. As a best practice, the server and portal runtime can have the same analytics binary. This helps when managing the deployment and when applying updates. However, it is up to the dev-ops engineer to decide whether to use the same binary (pack) or two binaries for the two runtimes.

Configure the Analytics Server¶

Open the /conf/server/deployment.yaml file. Edit the EI_ANALYTICS datasource under dataSources section.

- name: EI_ANALYTICS
      description: "The datasource used for EI Analytics dashboard feature"
      jndiConfig:
        name: jdbc/EI_ANALYTICS
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:mysql://127.0.0.1:3306/EI_ANALYTICS'
          username: root
          password: root
          driverClassName: com.mysql.cj.jdbc.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          validationTimeout: 30000
          isAutoCommit: false

Copy the driver to the /lib folder

Configure the Analytics Portal¶

Open the /conf/portal/deployment.yaml file. Edit the EI_ANALYTICS datasource under dataSources section.

- name: EI_ANALYTICS
      description: "The datasource used for EI Analytics dashboard feature"
      jndiConfig:
        name: jdbc/EI_ANALYTICS
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:mysql://127.0.0.1:3306/EI_ANALYTICS'
          username: root
          password: root
          driverClassName: com.mysql.cj.jdbc.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          validationTimeout: 30000
          isAutoCommit: false

Copy the driver to the /lib folder

Configuring the Micro Integrator¶

Enabling statistics monitoring¶

To enable statistics monitoring for the Micro Integrator, add the following parameters in the deployment.toml file of your Micro Integrator. This file is stored in the MI_HOME/conf.

[mediation]
flow.statistics.enable=true
stat.tracer.collect_payloads=true
stat.tracer.collect_mediation_properties=true

Configuring the Micro Integrator to publish data to EI Analytics¶

Analytics publishing can be configured in the [monitoring] section of the <MI_HOME>/conf/deployment.toml file as shown below.

Note

By default, the Micro Integrator is internally configured (with the following) to connect with an EI Analytics server running on the same VM. To change the default setup, you need to add the following to the deployment.toml file and update the values.

[monitoring]
ei_analytics.server_url = "tcp://localhost:7612"
ei_analytics.auth_server_url = "ssl://localhost:7712"
ei_analytics.username = "admin"
ei_analytics.password = "admin"

If the Analytics nodes run in cluster mode or in different VMs, you can configure the ei_analytics.server_url and the ei-analytics.auth_server_url parameters in a load balancing manner. For more information, see, Load balancing among multiple Analytics servers.

Enabling statistics for ALL artifacts¶

If you want to collect statistics for all your integration artifacts, be sure to add the following parameter under the [mediation] header in the deployment.toml file in addition the parameters explained above:

flow.statistics.capture_all=true

Alternatively, you can enable statistics for selected artifacts as explained below.

Enabling statistics for specific artifacts¶

Let's use the integration artifacts from the service chaining tutorial.

Warning

It is not recommended to enable tracing in production environments as it generates a large number of events that reduces the performance of the analytics profile. Therefore, tracing should only be enabled in development environments.

If you did not try the service chaining tutorial yet:

Download the pre-packaged project for the service chaining use case.
Open WSO2 Integration Studio and import the pre-packaged project.

Follow the steps below to enable statistics and tracing for the REST API artifact:

Select HealthcareAPI in the canvas of WSO2 Integration Studio to open the Properties tab.
Select Statistics Enabled and (if required) Trace Enabled as shown below.

Follow the steps below to enable statistics for the endpoint artifacts:

Select the required endpoint artifacts from the project explorer.
Select Statistics Enabled and (if required) Trace Enabled as shown below.

Load balancing among multiple Analytics servers¶

You can send events to multiple Analytics servers either by sending the same event to many Analytics servers or by load balancing events among a set of servers. This handles the failover problem. When events are load balanced within a set of servers and if one receiver cannot be reached, events are automatically sent to the other available and active Analytics servers. The following scenarios are covered in this section.

Load balancing across a group of servers¶

To configure this setup, configure the Analytics receiver URL specified in the Micro Integrator as a comma-separated list of Analytics servers.

The format of the receiver URL should be as follows: tcp://<Analytics-1>:<port>,tcp://<Analytics-2>:<port>,tcp://<Analytics-3>:<port>

Example configuration in the deployment.toml file of the Micro Integrator:

[monitoring]
ei_analytics.server_url = "tcp://10.100.2.32:7611, tcp://10.100.2.33:7611, tcp://10.100.2.34:7611"
ei_analytics.auth_server_url = "tcp://10.100.2.32:7612, tcp://10.100.2.33:7612, tcp://10.100.2.34:7612"
ei_analytics.username = "admin"
ei_analytics.password = "admin"

lb events to servers

This handles failover as follows: If Analytics Receiver-1 is marked as down, then the Micro Integrator will send the data only to Analytics Receiver-2 and Analytics Receiver-3 in a round robin manner. When the Analytics Receiver-1 becomes active after some time, the Micro Integrator automatically detects it, adds it to the operation, and again starts to load balance between all three receivers. This functionality significantly reduces the loss of data and provides more concurrency.

Load balancing across multiple groups of servers¶

In this setup, there are two sets of servers that are referred to as set-A and set-B. You can send events to both the sets. You can also carry out load balancing for both sets as mentioned in Load balancing across a group of servers. This scenario is a combination of load balancing between a set of servers and sending an event to several receivers. An event is sent to both set-A and set-B. Within set-A, it is sent either to Analytics A1 or Analytics A2. Similarly within set-B, it is sent either to Analytics B1 or Analytics B2. In the setup, you can have any number of sets and any number of servers as required. lb events to set of servers

Similar to the other scenarios, you need to describe the server URLs as the receiver URL in the Micro Integrator configuration. The sets should be specified within curly braces separated by commas. Furthermore, each receiver that belongs to the set should be within the curly braces and with the receiver URLs in a comma-separated format.

The format of the receiver URL should be as follows: {tcp://Analytics-A1:port, tcp://Analytics-A2:port},{tcp://Analytics-B1:port, tcp://Analytics-B2:port}

Example configuration in the deployment.toml file of the Micro Integrator:

[monitoring]
ei_analytics.server_url = "{tcp://10.100.2.32:7611, tcp://10.100.2.33:7611}, {tcp://10.100.2.34:7611, tcp://10.100.2.35:7611}"
ei_analytics.auth_server_url = "{tcp://10.100.2.32:7612, tcp://10.100.2.33:7612}, {tcp://10.100.2.34:7612, tcp://10.100.2.35:7612}"
ei_analytics.username = "admin"
ei_analytics.password = "admin"

Sending all events to several analytics servers¶

This setup involves sending all the events to more than one Analytics server. This approach is useful when you want to have multiple Analytics servers to analyze the same events simultaneously. For example, as shown below, you can configure the Micro Integrator to publish the same event to both Analytics servers at the same time.

all events to all servers

The Analytics receiver URL should be configured with the following format in the Micro Integrator: {tcp://Analytics-1>:<port>}, {tcp://Analytics-2>:<port>}, {tcp://<Analytics-3>:<port>}

Example configuration in the deployment.toml file of the Micro Integrator:

[monitoring]
ei_analytics.server_url = "{tcp://10.100.2.32:7611},{ tcp://10.100.2.33:7611}, {tcp://10.100.2.34:7611}"
ei_analytics.auth_server_url = "{tcp://10.100.2.32:7612},{ tcp://10.100.2.33:7612}, {tcp://10.100.2.34:7612}"
ei_analytics.username = "admin"
ei_analytics.password = "admin"

Failover configuration¶

When using the failover configuration in publishing events to Analytics, events are sent to multiple Analytics servers in a sequential order based on priority. You can specify multiple Analytics servers so that events can be sent to the next server in the specified sequence (in a situation where they were not successfully sent to the first server). In the scenario depicted in the above image, the events are first sent to Analytics-1. If it is unavailable, then events are sent to Analytics-2. If Analytics-2 is also unavailable, then the events are sent to Analytics-3.

fail over

The Analytics receiver URL should be configured with the following format in the Micro Integrator: tcp://<Analytics-1>:<port>|tcp://<Analytics-2>:<port>|tcp://<Analytics-3>:<port>

[monitoring]
ei_analytics.server_url = "tcp://10.100.2.32:7611|tcp://10.100.2.33:7611|tcp://10.100.2.34:7611"
ei_analytics.auth_server_url = "tcp://10.100.2.32:7612|tcp://10.100.2.33:7612|tcp://10.100.2.34:7612"
ei_analytics.username = "admin"
ei_analytics.password = "admin"

What's Next?¶

If you have successfully set up your anlaytics deployment, see the instructions on using the analytics portal.

Top