Skip to content

Monitoring ETL Flows with Grafana

Before you begin:

To enable WSO2 Streaming Integrator to publish statistics in Grafana, follow the instructions in Setting up Grafana to Display WSO2 SI Statistics.

Setting Up Grafana to Monitor WSO2 Streaming Integrator

For the purpose of monitoring ETL (Extract, Transform, Load) statistics WSO2 provides nine pre-configured dashboards. To view them in Grafana, follow the steps below:

  1. Download the following dashboards (i.e., the JSON file with the dashboard configuration) from here.

    Directory Dashboard
    overview-statistics - WSO2 Streaming Integrator - Overall Statistics.json
    - WSO2 Streaming Integrator - App Statistics.json
    file-statistics - WSO2 Streaming Integrator - File Statistics.json
    - WSO2 Streaming Integrator - File Source Statistics.json
    - WSO2 Streaming Integrator - File Sink Statistics.json
    rdbms-statistics - WSO2 Streaming Integration - RDBMS Statistics.json
    - WSO2 Streaming Integration - RDBMS Table Statistics.json
    cdc-statistics - WSO2 Streaming Integrator - CDC Statistics.json
    - WSO2 Streaming Integrator - CDC Streaming Statistics.json
    - WSO2 Streaming Integrator - CDC Scheduled Statistics.json
    kafka-statistics - WSO2 Streaming Integrator - Kafka Statistics.json
    - WSO2 Streaming Integrator - Kafka Source Statistics.json
    - WSO2 Streaming Integrator - Kafka Sink Statistics.json
  2. Start and access Grafana. Then import the nine dashboards you downloaded in the previous step. For more information, see Managing Grafana Dashboards - Importing Dashboards.

  3. In the Dashboards/Manage tab, create four new folders named overview-statistics, file-statistics, rdbms-statistics, cdc-statistics, and kafka-statistics . Then organize the dashboards you imported in the new folders you created as shown in the image below. For instructions, see Managing Grafana Dashboards - Organizing Dashboards in Folders.

    Info

    The following is a suggested method to organize your dashboards, and the following instructions are provided based on this structure.

    Organize Dashboards

Accessing Grafana Dashboards for Monitoring

To navigate through the Grafana dashboards you set up for monitoring WSO2 Streaming Integrator and to analyze statistics, follow the procedure below:

  1. To start and access the dashboards in Grafana, follow the steps below:

    1. Start the Prometheus server by issung the following command from the <PROMETHEUS_HOME> directory.

      ./prometheus

    2. Start Grafana.

      Info

      The procedure to start Grafana depends on your operating system and the installation process. e.g., If your operating system is Mac OS and you have installed Grafana via Homebrew, you start Grafana by issuing the brew services start grafana command.

    3. Start the WSO2 Streaming Integrator server by navigating to the <SI_HOME>/bindirectory, and issuing the appropriate command out of the following based on your operating system.

      • For Linux: ./server.sh
      • For Windows: server.bat

    Access Grafana via http://localhost:3000/.

  2. In the left pane, click the Dashboards icon, and then click Manage to open the Dashboards page.

  3. In the Dashboards page, click WSO2 Streaming Integrator Overall Statistics.

    Open Overall Statistics

    The overall statistics are displayed as shown in the following example.

    Overall Statistics

    This provides an overview on the overall performance of the WSO2 Streaming Integrator instance by displaying the total number of input events consumed and the total number of output events. It also provides the breakdown of the total inputs and outputs by Siddhi application. The (Consume/Publish)/Sec graph indicates the rate at which the consumed input events are published per second.

  4. To view the statistics specific to a Siddhi application, click on the relevant Siddhi application.

    Select Siddhi Application

    The WSO2 Streaming Integrator App Statistics dashboard opens. This allows you to access any statistics relating to change data capture, file processing and data store integration activities carried out via the selected Siddhi application.

    Application Statistics

Viewing CDC statistics**

CDC related statistics are displayed in the Sources section of the WSO2 Streaming Integrator App Statistics dashboard. To view the detailed statistics for a specific table, click on that table.

As a result, the CDC Statistics dashboard opens with change data statistice specific to that table.

CDC Statistics Dashboard

If you click on a table for which change data is captured in a streaming manner (i.e., a table in the CDC Streaming Statistics section), the CDC Streaming Statistics dashboard opens with change data related statistics specific to the selected table as shown in the example below.

CDC Streaming Statistics Dashboard

If you click on a table for which change data is captured in a scheduled manner by polling the table (i.e., a table in the CDC Scheduled Statistics section), the CDC Scheduled Statistics dashboard opens with change data related statistics specific to the selected table as shown in the example below.

CDC Scheduled Statistics Dashboard

Viewing File statistics**

  • When the content of a file is used as input data, the file is displayed in the Sources section as shown in the example below.

File Source Statistics

To view statistics for a specific file, click on it. As a result, the WSO2 Streaming Integrator - File Statistics dashboard opens as shown in the example below.

File Statistics Dashboard

When you click on the same file again, the WSO2 Streaming Integrator - File Source Statistics dashboard opens with more detailed information about this source file as shown in the example below.

File Source Statistics Dashboard

  • When the content of a file is generated as an output of the Siddhi application, the file is displayed in the Destinations section as shown in the example below.

File Sink Statistics

To view statistics for a specific file, click on it. As a result, the WSO2 Streaming Integrator - File Statistics dashboard opens as shown in the example below.

File Statistics Dashboard

When you click on the same file again, the WSO2 Streaming Integrator - File Sink Statistics dashboard opens with more detailed information about this sink file as shown in the example below.

File Sink Statistics Dashboard

Viewing RDBMS statistics**

When the records of an RDBMS database is used as input data by the selected Siddhi application, the database is displayed in the Sources section. When any output generated by the selected Siddhi application is saved in a database, that database is displayed under Destinations.

In each section, all the databases of the relevant category is listed in a table as shown in the example below.

RDBMS Statistics

To view detailed statistics of a specific database, click on that database.

Select RDBMS

The RDBMS Statistics dashboard opens with RDBMS statistics specific to the selected database.

Database Statistics

To view statistics relating to a specific table, click on the relevant table name in the RDBMS Table table.

Select RDBMS Table

The RDBMS Table Statistics dashboard opens with statistics specific to the selected table.

RDBMS Table Statistics

Viewing Kafka statistics

Kafka related statistics are displayed in the Sources and Destinations sections of the WSO2 Streaming Integrator App Statistics dashboard.

To view the detailed overall Kafka Statistics for a specific app, click on any stream link on the table. As a result, you will be taken to WSO2 Streaming Integrator Kafka Statistics dashboard.

This page has two sections: Source and Sink.

  • The Source section shows the overall Kafka Source metrics such as total reads, input event rate etc. It also contains a table that lists all the streams that are associated to a Kafka source. If you click on a stream, it will take you to WSO2 Streaming Integrator Kafka Source Statistics dashboard.

    This dashboard shows the Kafka Source statistics related to the selected stream. A stream is associated to one or more topics. Furthermore, each topic is associated to a partition. You can use this dashboard to view statistics for each separate Topic-Partition combination to which the selected stream is subscribed.

  • The Sink section shows the overall Kafka Sink metrics such as total writes, output event rate etc. It also contains a table that lists all the streams that are associated to a Kafka source. If you click on a stream, it will take you to WSO2 Streaming Integrator Kafka Sink Statistics dashboard.

    This dashboard shows the Kafka Sink statistics related to the selected stream. Here, a stream publishes to a specific Kafka Topic and to one or more partitions. You can use this dashboard to view statistics for each separate Topic-Partition combination to which the selected stream publishes messages.

Top