Skip to content

Monitoring ETL Flows with Grafana

Before you begin:

To enable WSO2 Streaming Integrator to publish statistics in Grafana, follow the instructions in Setting up Grafana to Display WSO2 SI Statistics.

Setting Up Grafana to Monitor WSO2 Streaming Integrator

For the purpose of monitoring ETL (Extract, Transform, Load) statistics WSO2 provides nine pre-configured dashboards. To view them in Grafana, follow the steps below:

  1. Download the following dashboards (i.e., the JSON file with the dashboard configuration) from here.

    Directory Dashboard
    overview-statistics - WSO2 Streaming Integrator - Overall Statistics.json
    - WSO2 Streaming Integrator - App Statistics.json
    file-statistics - WSO2 Streaming Integrator - File Statistics.json
    - WSO2 Streaming Integrator - File Source Statistics.json
    - WSO2 Streaming Integrator - File Sink Statistics.json
    rdbms-statistics - WSO2 Streaming Integration - RDBMS Statistics.json
    - WSO2 Streaming Integration - RDBMS Table Statistics.json
    cdc-statistics - WSO2 Streaming Integrator - CDC Statistics.json
    - WSO2 Streaming Integrator - CDC Streaming Statistics.json
    - WSO2 Streaming Integrator - CDC Scheduled Statistics.json
  2. Start and access Grafana. Then import the nine dashboards you downloaded in the previous step. For more information, see Managing Grafana Dashboards - Importing Dashboards.

  3. In the Dashboards/Manage tab, create four new folders named overview-statistics, file-statistics, rdbms-statistics, and cdc-statistics . Then organize the dashboards you imported in the new folders you created as shown in the image below. For instructions, see Managing Grafana Dashboards - Organizing Dashboards in Folders.

    Info

    The following is a suggested method to organize your dashboards, and the following instructions are provided based on this structure.

    Organize Dashboards

Accessing Grafana Dashboards for Monitoring

To navigate through the Grafana dashboards you set up for monitoring WSO2 Streaming Integrator and to analyze statistics, follow the procedure below:

  1. To start and access the dashboards in Grafana, follow the steps below:

    1. Start the Prometheus server by issung the following command from the <PROMETHEUS_HOME> directory.

      ./prometheus

    2. Start Grafana.

      Info

      The procedure to start Grafana depends on your operating system and the installation process. e.g., If your operating system is Mac OS and you have installed Grafana via Homebrew, you start Grafana by issuing the brew services start grafana command.

    3. Start the WSO2 Streaming Integrator server by navigating to the <SI_HOME>/bindirectory, and issuing the appropriate command out of the following based on your operating system.

      • For Linux: ./server.sh
      • For Windows: server.bat

    Access Grafana via http://localhost:3000/.

  2. In the left pane, click the Dashboards icon, and then click Manage to open the Dashboards page.

  3. In the Dashboards page, click WSO2 Streaming Integrator Overall Statistics.

    Open Overall Statistics

    The overall statistics are displayed as shown in the following example.

    Overall Statistics

    This provides an overview on the overall performance of the WSO2 Streaming Integrator instance by displaying the total number of input events consumed and the total number of output events. It also provides the breakdown of the total inputs and outputs by Siddhi application. The (Consume/Publish)/Sec graph indicates the rate at which the consumed input events are published per second.

  4. To view the statistics specific to a Siddhi application, click on the relevant Siddhi application.

    Select Siddhi Application

    The WSO2 Streaming Integrator App Statistics dashboard opens. This allows you to access any statistics relating to change data capture, file processing and data store integration activities carried out via the selected Siddhi application.

    Application Statistics

    • Viewing CDC statistics

      CDC related statistics are displayed in the Sources section of the WSO2 Streaming Integrator App Statistics dashboard. To view the detailed statistics for a specific table, click on that table.

      As a result, the CDC Statistics dashboard opens with change data statistice specific to that table.

      CDC Statistics Dashboard

      If you click on a table for which change data is captured in a streaming manner (i.e., a table in the CDC Streaming Statistics section), the CDC Streaming Statistics dashboard opens with change data related statistics specific to the selected table as shown in the example below.

      CDC Streaming Statistics Dashboard

      If you click on a table for which change data is captured in a scheduled manner by polling the table (i.e., a table in the CDC Scheduled Statistics section), the CDC Scheduled Statistics dashboard opens with change data related statistics specific to the selected table as shown in the example below.

      CDC Scheduled Statistics Dashboard

    • Viewing File statistics

    • When the content of a file is used as input data, the file is displayed in the Sources section as shown in the example below.

      File Source Statistics

      To view statistics for a specific file, click on it. As a result, the WSO2 Streaming Integrator - File Statistics dashboard opens as shown in the example below.

      File Statistics Dashboard

      When you click on the same file again, the WSO2 Streaming Integrator - File Source Statistics dashboard opens with more detailed information about this source file as shown in the example below.

      File Source Statistics Dashboard

    • When the content of a file is generated as an output of the Siddhi application, the file is displayed in the Destinations section as shown in the example below.

      File Sink Statistics

      To view statistics for a specific file, click on it. As a result, the WSO2 Streaming Integrator - File Statistics dashboard opens as shown in the example below.

      File Statistics Dashboard

      When you click on the same file again, the WSO2 Streaming Integrator - File Sink Statistics dashboard opens with more detailed information about this sink file as shown in the example below.

      File Sink Statistics Dashboard

    • Viewing RDBMS statistics

      When the records of an RDBMS database is used as input data by the selected Siddhi application, the database is displayed in the Sources section. When any output generated by the selected Siddhi application is saved in a database, that database is displayed under Destinations.

      In each section, all the databases of the relevant category is listed in a table as shown in the example below.

      RDBMS Statistics

      To view detailed statistics of a specific database, click on that database.

      Select RDBMS

      The RDBMS Statistics dashboard opens with RDBMS statistics specific to the selected database.

      Database Statistics

      To view statistics relating to a specific table, click on the relevant table name in the RDBMS Table table.

      Select RDBMS Table

      The RDBMS Table Statistics dashboard opens with statistics specific to the selected table.

      RDBMS Table Statistics

Top