Alerts
Alerts are useful to receive notifications when a metric has reached a defined threshold (e.g. when the response time of your model takes more than 100ms).
Triggered alerts
An overview of triggered alerts is available on the Alerts page. Use the alert status to let others know if an alert has been acknowledged or resolved.
Manage alert rules
Click Manage alert rules on the Alerts page.
Add alert rules
Click Add alert rule on the Manage alert rules page. Configure the following parameters:
- Name: displayed in notifications for this alert rule
- Metric: set up alert rules for any metric (including custom metrics) except disagreement per class.
Removing a custom metric automatically deletes alert rules that are configured for that custom metric.
- Severity: set to Warning by default. You can set up multiple alert rules for the same metric, each with a different severity level to reflect varying thresholds.
- Webhook: select a webhook to which data will be posted when the alert rule is triggered. For example, if you want to receive information about an alert in a messaging service.
The Slack notifications integration applies to all notifications and is not available in the webhooks list by default. You can set up a Slack as an alert webhook to receive just alerts.
The Slack notifications integration applies to all notifications, not just alerts. If you want to send a Slack message for a specific alert rule, add an alert webhook and select it when creating the rule.
The following parameters may be configurable depending on the selected metric:
- Feature: for feature-level metrics; select the feature to set up the alert.
- Guardrail: for the Triggered guardrails metric; select All to count trigger events across all deployment's available guardrails, or select one specific guardrail. If you want to monitor multiple specific guardrails, create separate alert rules for each.
Removing a guardrail from a deployment automatically deletes alert rules that are configured for that specific guardrail.
- Threshold: the options for thresholds depend on the selected metric:
- Errors: the threshold can’t be customized because error-based alert rules automatically include detailed information about each error.
- Input validation: define a minimum and maximum threshold. If your Deployment metadata includes an observed minimum and maximum for the selected feature, they’ll appear here to help guide your selection.
- Other: define a minimum or a maximum threshold. If you want to have multiple thresholds, set up separate alert rules for each one.
- Custom metrics: define a minimum or a maximum threshold. If you want to have multiple thresholds, set up separate alert rules for each one. Behavior depends on the chart type of the custom metric:
- Line/bar chart: an alert is raised for every x-y value pair where the y value crosses the defined threshold.
- Timeseries: similar to other metrics; an alert triggers when the y value crosses the defined minimum or maximum threshold.
Upon uploading timeseries custom metric data, all data points (newly submitted and existing) whose xValue timestamp falls within the rolling window [now − window, now] are evaluated against the threshold.
Custom metric alert rules are triggered based on submitted custom metric data and are not necessarily related to inference activity.
- Cooldown: set a cooldown to avoid numerous triggered alerts in a short time. Select how long the alert should wait before triggering again after it’s been triggered once.
- Window: the alert rule is triggered based on the value of the metric in a (rolling) time window.
- Minimum sample size: set to 30 by default. Alerts won’t trigger if there are fewer predictions in the window.
This table includes some example alert rule configurations:
| Parameter | Example 1 | Example 2 | Example 3 | Example 4 |
|---|---|---|---|---|
| Name | Age of applicant | Performance | Limit predictions per team | Validate agent relevancy score |
| Metric | Input validation | Accuracy | Predictions per team (Custom bar) | Relevancy score (Custom timeseries) |
| Feature | Age | - | - | - |
| Severity | Warning | Critical | Warning | Critical |
| Threshold | Min. 18 - Max. 70 | Min. 80 | Max. 500 | Min. 0.75 |
| Cooldown | 12 hrs | 1 hr | 6 hrs | 1 hr |
| Window | - | 1 hr | - | 1 week |
| Min. Sample size | - | 100 | - | 50 |
Edit alert rules
Click the action button and select Edit. You can make changes to the elements described in the previous section.
Disable and enable alert rules
Click the action button and select Disable or Enable. Disabled alert rules remain visible in the overview (marked with an icon) but won’t trigger any alerts.
Delete alert rules
Click the action button and select Delete.
View alert rules in monitoring graphs
To view an alert rule in a monitoring graph, navigate to the graph and set the bin size to match the window of the alert rule.
Notifications
Deployment owners receive emails and in-product notifications for all alert rules in the Deployment by default. To change this, update your email and notification preferences on the Preferences page under your account.