Getting started with Elementary

Elementary is by far the best open source Observability tool for DBT data engineers. It mostly translates the DBT metadata to internal teams, though it can also be used to gauge various metrics across teams.

We love the built in dashboards that elementary creates and testing toolkit is very useful. Please join us to get started with DBT in this blog post.

What is Elementary?

Elementary is very impressive open source DBT package. DBT itself provides many tests, though Elementary expands these tests to a new level. One can easily monitor DBT pipeline and it’s effects on data with the rich dashboard provided by Elementary.

On top it, Elementary also provides a seamless integration with slack to send automatically generated reports to teams and find out which Job, test has failed. Not to mention anomalies detection can be very powerful. If this sounds interesting, then please allow us to walk you through the setup and use cases.

How to install Elementary in DBT project?

Step 1: Add elementary to packages.yml

packages:
  - package: elementary-data/elementary
    version: 0.13.0

Step 2: Add elementary schema information in dbt_project.yml

models:
  elementary:
    ## elementary models will be created in the schema '<your_schema>_elementary'
    +schema: "elementary"

Step 3: Now, install Elementary by running below command and build elementary models for the first time.

dbt deps
dbt run --select elementary
dbt test  # this command will now add details in `elementary_test_results` table with every run & test

These steps are going to install elementary in your DBT project. Please follow along on how to generate dashboard.

Reference Install Elementary dbt package — Elementary

Reporting Dashboard

To generate reporting dashboard, Elementary uses a python package. This python package comes with a bunch of CLI commands which will help us generate the report and do much more!

Step 1: Provide DWH connection profile to elementary CLI. Run below command to generate the profile. Add it to your .dbt/profile.yml file.

dbt run-operation elementary.generate_elementary_cli_profile

Step 2: Install CLI python package.

pip install elementary-data
pip install 'elementary-data[databricks]'  # to support databricks DWH
# Other supported DWH are snowflake, bigquery, redshift, postgresql, mysql

Step 3: Generate the report

edr report

Reporting Dashboard shows a nice UI on current count of tests, tests coverage, how many failed and how many are passing.

Apart from the regular DBT tests, Elementary also provides Freshness, Data Volume anomalies, Column anomalies. These elementary specific tests are a great way to boost your data lineage needs into a solid tests.

One can also drill down these reports with tags and owners. Different teams can watch for their marts to accept shared responsibility model.

Image Credit: Elementary Documentation

Slack Alerts

As you can see, slack alert shows which test failed with test description, owner and subscribers. No more silently failing tests or need to go through emails. It meets where you collaborate with your team and stakeholders.

You can adjust sending your slack alerts on your need, though best practice would be to generate new monitoring alerts and reporting dashboard with every production job.

Image Credit: Elementary Documentation

Elementary also provides integrations to upload Generated report to static file servers such as AWS S3, GCP GCS, Azure Blob Storage and Slack report as attachment.

Apart from all these features, Elementary goes a step further to help you deploy OSS solution in your production environment with GitHub Actions, and GitLab CI integrations.

Elementary has set a new standard in observability for DBT data engineering solutions. We are hoping to see a wide range of such unique ideas for other Data Engineering tools & technologies

Team at Aptologics have a wide range of tools experience building scalable Data Engineering solutions. Please feel free to reach out to us on LinkedIn to transform your journey in Data Analytics.