Custom Metrics

Custom metrics allow you to visualize your own metrics on a Deployment. You define the metric's name, axis labels, and chart type, then push data points whenever you have new values to record.

Creating a Custom Metric

Use create_custom_metric to register a new metric on a Deployment. You must supply a GraphType (line, bar, or timeseries) and axis labels.

For line and bar graphs, provide title_x_axis and optionally unit_x_axis. For timeseries graphs, the x-axis is the data point's submission timestamp — do not provide title_x_axis or unit_x_axis. You can optionally set an aggregation_method (average, sum, min, or max) to control how data points are binned; it defaults to average.

from deeploy.models import CreateCustomMetric
from deeploy.enums import GraphType

# Line or bar graph — x-axis is defined by you
metric = client.create_custom_metric(
    deployment_id,
    CreateCustomMetric(
        name='Daily Token Usage',
        graph_type=GraphType.LINE,
        title_x_axis='Date',
        title_y_axis='Total tokens',
        unit_x_axis='day',
    )
)

print(metric.id)  # save this to add data points later

from deeploy.models import CreateCustomMetric
from deeploy.enums import AggregationMethod, GraphType

# Timeseries graph — x-axis is the submission timestamp; do not set title_x_axis or unit_x_axis
metric = client.create_custom_metric(
    deployment_id,
    CreateCustomMetric(
        name='Token Usage',
        graph_type=GraphType.TIMESERIES,
        title_y_axis='Total tokens',
        aggregation_method=AggregationMethod.SUM,
    )
)

print(metric.id)  # save this to add data points later

Adding Data Points

Use create_custom_metric_data_points to push one or more data points to an existing metric. Each data point has an x_value and a y_value (a float). You can optionally associate a list of prediction_log_ids with a data point for traceability.

• For line and bar graphs, x_value can be any string or number.
• For timeseries graphs, x_value must be a valid ISO 8601 datetime string (e.g. "2024-01-15T10:30:00Z").

from deeploy.models import CreateCustomMetricDataPoint

custom_metric_id = metric.id  # from the create step above

# Line / bar: x_value is a label or number
data_points = [
    CreateCustomMetricDataPoint(x_value='2026-03-06', y_value=312.4),
    CreateCustomMetricDataPoint(x_value='2026-03-07', y_value=289.1),
]

# Timeseries: x_value must be an ISO 8601 datetime string
data_points = [
    CreateCustomMetricDataPoint(x_value='2026-03-06T00:00:00Z', y_value=312.4),
    CreateCustomMetricDataPoint(x_value='2026-03-07T00:00:00Z', y_value=289.1),
]

client.create_custom_metric_data_points(deployment_id, custom_metric_id, data_points)

Example: LLM Token Usage per Day

The following example fetches request logs for the past 7 days from an OpenAI schema compatible external Deployment, extracts the total_tokens field from each LLM response body, computes the daily total, and submits those totals as data points to a custom metric.

Request logs must be fetched with include_raw_body=True so that the raw LLM response — including token usage — is returned.

Example implementation

from datetime import datetime, timedelta, timezone

from deeploy import Client
from deeploy.models import CreateCustomMetric, CreateCustomMetricDataPoint, GetRequestLogsOptions
from deeploy.enums import AggregationMethod, GraphType

deployment_id = "..."
workspace_id = "..."

client = Client(
    host="app.deeploy.ml",
    access_key="",
    secret_key="",
    workspace_id=workspace_id
)

# --- 1. Create the custom metric ---
metric = client.create_custom_metric(
    deployment_id,
    CreateCustomMetric(
        name='Token Usage',
        graph_type=GraphType.TIMESERIES,
        title_y_axis='Total tokens',
        aggregation_method=AggregationMethod.SUM
    ),
)

# --- 2. Determine the time window: last 7 complete days ---
now = datetime.now(tz=timezone.utc)
end_of_today = now.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(days=1)
start_of_window = end_of_today - timedelta(days=7)

start_ms = int(start_of_window.timestamp() * 1000)
end_ms = int(end_of_today.timestamp() * 1000)

# --- 3. Page through all request logs in the window ---
PAGE_SIZE = 100
offset = 0
all_logs = []

while True:
    page = client.get_request_logs(
        deployment_id,
        params=GetRequestLogsOptions(
            start=start_ms,
            end=end_ms,
            limit=PAGE_SIZE,
            offset=offset,
        ),
        include_raw_body=True,
    )
    all_logs.extend(page)
    if len(page) < PAGE_SIZE:
        break
    offset += PAGE_SIZE

# --- 4. Build one data point per request log ---
# This example assumes LLM deployments that return an OpenAI-compatible response body with a
# top-level "usage" object: { "prompt_tokens": int, "completion_tokens": int, "total_tokens": int }
data_points = []
for log in all_logs:
    if log.raw_response_body is None:
        continue
    usage = log.raw_response_body.get('usage')
    if not usage:
        continue
    total_tokens = usage.get('total_tokens')
    if total_tokens is None:
        continue

    data_points.append(
        CreateCustomMetricDataPoint(
            x_value=log.created_at,
            y_value=float(total_tokens),
        )
    )

# --- 5. Submit the data points ---
client.create_custom_metric_data_points(deployment_id, metric['id'], data_points, timezone='UTC')

print(f"Submitted {len(data_points)} data points to metric '{metric['name']}' ({metric['id']})")