Slack Alerts For CramMultiQC & GvcfMultiQC Findings

Aug 7, 2025 by Luna Greco 52 views

Add 'Send-to-Slack' Functionality for Findings in CramMultiQC and GvcfMultiQC Stages

Hey guys! We're diving into adding a super useful feature to our cpg_workflows: the ability to send alerts to Slack when QC findings in the CramMultiQC and GvcfMultiQC stages fall outside the acceptable thresholds. This is going to help us keep a closer eye on our data quality and react quickly to any issues. Let's break down what needs to be done.

Understanding the Context

In our cpg_workflows implementation, these stages have the capability to check QC data against predefined thresholds. You can see these thresholds in our configuration files, specifically here. We use a nifty method called check_report_job to make these checks. The goal now is to integrate Slack notifications so we're immediately aware of any QC issues.

Why Slack Notifications?

Imagine this: instead of manually checking QC reports, you get an instant notification in your Slack channel whenever something goes awry. This means faster response times, quicker troubleshooting, and ultimately, higher data quality. By integrating Slack, we're streamlining our workflow and ensuring that critical issues don't slip through the cracks. It's like having a vigilant QC watchdog that barks (or, in this case, Slacks) when something's up.

The Current Implementation

Currently, our system checks the QC data, but it doesn't actively notify us of any issues. We need to bridge this gap by adding Slack integration. This involves a few key steps:

Adding slack_sdk as a package dependency.
Adding boolean config options for "send to slack" for SomalierPedigree, CramMultiQC, and GvcfMultiQC stages.
Adding QC thresholds to the default config.toml.
Implementing the check_report code into the CramMultiQC and GvcfMultiQC stages.

Let's dive deeper into each of these steps.

Step-by-Step Implementation

1. Adding `slack_sdk` as a Package Dependency

First things first, we need to make sure our project can actually talk to Slack. The slack_sdk package is our go-to tool for this. We've already taken care of this step in #13, so we can check this one off the list! Nice and easy!

2. Adding Boolean Config Options for "Send to Slack"

Next up, we need to add some switches in our configuration to control whether Slack notifications are enabled for each stage. This gives us the flexibility to turn notifications on or off depending on our needs. We'll be adding boolean config options for SomalierPedigree, CramMultiQC, and GvcfMultiQC stages. Think of it like adding light switches for our Slack alerts – we can flip them on or off as needed.

How to Add the Config Options

We'll be adding these options in our configuration files, similar to how we've done it in other parts of the project. For example, in the cpg_workflows/stages/cram_qc.py file (as seen here), we can add a boolean option like send_to_slack. This option will determine whether notifications are sent for that specific stage.

# Example of adding a boolean config option
class CramQCStage(Stage):
    def __init__(self, config, analysis_id):
        super().__init__(config, analysis_id)
        self.send_to_slack = config.get('send_to_slack', False)  # Default to False

By setting the default value to False, we ensure that notifications are not sent unless explicitly enabled in the configuration.

3. Adding QC Thresholds to the Default `config.toml`

To make sure our Slack notifications are meaningful, we need to define the QC thresholds that trigger them. These thresholds will be added to our default config.toml file. This file acts as the central hub for our configuration settings, making it easy to manage and update our thresholds as needed. These thresholds act as our warning system – when a metric crosses the line, Slack gets the message.

Defining the Thresholds

We'll need to carefully consider what thresholds are appropriate for each QC metric. This might involve looking at historical data, consulting with experts, and running some tests to fine-tune our settings. The goal is to set thresholds that are sensitive enough to catch real issues but not so sensitive that we're flooded with false alarms.

For example, we might set thresholds for metrics like:

Mapping rate: A minimum percentage of reads that should map to the reference genome.
Coverage: The average depth of coverage across the genome.
Contamination: The level of contamination from other samples or sources.

These thresholds will be defined in the config.toml file, making them easy to adjust and update as our needs evolve.

4. Implementing the `check_report` Code

This is where the magic happens! We need to take our existing check_report code and integrate it into the CramMultiQC and GvcfMultiQC stages. This code will be responsible for comparing the QC metrics against our defined thresholds and triggering Slack notifications when necessary.

Breaking Down the Implementation

The check_report code will essentially do the following:

Fetch QC Metrics: It will retrieve the relevant QC metrics from the MultiQC reports.
Compare to Thresholds: It will compare these metrics against the thresholds defined in our config.toml file.
Send Slack Notification: If any metrics fall outside the acceptable range, it will send a notification to our designated Slack channel.

Integrating into Stages

We'll need to modify the CramMultiQC and GvcfMultiQC stages to incorporate this logic. This will likely involve adding a new step that calls the check_report function and sends the Slack notification if needed.

# Example of integrating check_report into a stage
class CramMultiQCStage(Stage):
    def run(self):
        # ... existing code ...
        self.check_qc_report()

    def check_qc_report(self):
        if self.config.get('send_to_slack', False):
            report_data = self.get_report_data()  # Placeholder for fetching report data
            issues = check_report(report_data, self.config.get('qc_thresholds'))  # Placeholder for check_report function
            if issues:
                send_slack_notification(issues)  # Placeholder for sending Slack notification

This is a simplified example, but it gives you an idea of how we'll integrate the check_report code into our stages.

Diving Deeper into Key Concepts

Let's zoom in on some of the core components we'll be working with.

`check_report_job` Method

The check_report_job method is the heart of our QC checking process. It takes the QC report data and compares it against the defined thresholds. If any metrics fall outside the acceptable range, this method will flag them as issues.

How It Works

The check_report_job method typically involves the following steps:

Data Extraction: It extracts the relevant QC metrics from the MultiQC report.
Threshold Comparison: It compares these metrics against the thresholds defined in the config.toml file.
Issue Identification: It identifies any metrics that fall outside the acceptable range.
Reporting: It generates a report of the identified issues.

This method is designed to be flexible and configurable, allowing us to easily adapt it to different QC metrics and thresholds.

Slack Integration

Sending notifications to Slack involves using the slack_sdk library to communicate with the Slack API. This library provides a convenient way to send messages to Slack channels, making it easy to integrate Slack notifications into our workflow.

Key Steps for Slack Integration

Install slack_sdk: We've already taken care of this step by adding it as a package dependency.
Set Up Slack App: We'll need to create a Slack app and obtain the necessary credentials (e.g., a Slack token) to authenticate with the Slack API.
Send Messages: We'll use the slack_sdk library to send messages to our designated Slack channel, including details about any QC issues that have been identified.

Configuration with `config.toml`

The config.toml file plays a crucial role in our system by providing a centralized location for all our configuration settings. This includes QC thresholds, Slack notification settings, and other parameters that control the behavior of our workflows.

Benefits of Using `config.toml`

Centralized Configuration: All our settings are in one place, making it easy to manage and update them.
Flexibility: We can easily adjust settings without having to modify our code.
Reproducibility: We can ensure that our workflows are reproducible by using the same configuration settings across different runs.

Next Steps and Collaboration

So, what's next? We've laid out the plan, and now it's time to put it into action. Here's a quick recap of the remaining tasks:

[ ] Add in boolean config options for "send to slack" for SomalierPedigree, CramMultiQC, and GvcfMultiQC stages.
[ ] Add QC thresholds to default config.toml.
[ ] Add the check_report code and implement it into the CramMultiQC, and GvcfMultiQC stages.

Let's collaborate to get this done! If you have any questions, ideas, or suggestions, please don't hesitate to chime in. Together, we can make our QC workflow even more robust and efficient. Let's keep the conversation flowing and make this happen! 🚀