Log The Results Of The DQ Check

[rishujuneja1298]

There should be ways to audit the DQ runs to track the history of the failed or passed records.

[mwojtyczka]

The checks are currently written as a separate reporting column (_errors, _warnings). You can track the history using that. Or are you looking for additional capabilities?
We are releasing this week a change that will greatly enhance the amount of info that is reported. You will also be able to define custom metadata.

[mwojtyczka]

DQX logs now more comprehensive info as part of the results. Please check the latest version: https://github.com/databrickslabs/dqx/releases/tag/v0.3.0

Please find it documented here: https://databrickslabs.github.io/dqx/docs/guide/#quality-check-results

[rishujuneja1298]

I'm considering saving logs for tracking purposes to analyze trends and summarize the results of the data quality (DQ) checks. This way, we can track how many failures occurred for a specific dataset on a given date, and see if the counts have increased or decreased over time. Currently, I would need to extract details from the quarantine dataframe and store them somewhere for analysis. Is that the correct approach, or does DQX have any built-in functionality for this?

[mwojtyczka]

DQX will gives you a dataframe with quarantined data and reporting columns. You can track the trends using this. Alternative, you can save the reporting columns (_warnrnings, _errors) from the dataframe to a separate table if you want.
We also provide a dashboard (if you install DQX in the workspace using Databricks CLI) that has some queries/visualizations. The dashboard is using the quarantined table as input. This is the in-build functionality that we provide. You can extract the queries from the dashboard or modify it to suite your specific needs. Basically, you can run queries on the quarantine data to do analytics on the results.

[mwojtyczka]

Here you can find documentation on how to analyse the results: https://databrickslabs.github.io/dqx/docs/guide/#quality-check-results

[mwojtyczka]

we will be providing summary: #376

[mwojtyczka]

This has been released a while ago: https://databrickslabs.github.io/dqx/docs/guide/summary_metrics/