Inference

Use the Python client to inference your Deployment.

Predict

Make a call to the /predict endpoint of the Deployment.

request_body = {
    "instances": [
        [39, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9],
        [25, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9]
    ]
}

# Single request with full batch
prediction = client.predict(deployment_id, request_body)

# With parallel batch processing (see below for details)
from deeploy.models import ClientOptions
options = ClientOptions(nr_requests=2)
results = client.predict(deployment_id, request_body, client_options=options)

Explain

Make a call to the /explain endpoint of the Deployment.

request_body = {
    "instances": [
        [39, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9],
        [25, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9]
    ]
}

# Single request with full batch
explanation = client.explain(deployment_id, request_body)

# With parallel batch processing
from deeploy.models import ClientOptions
options = ClientOptions(nr_requests=2)
results = client.explain(deployment_id, request_body, client_options=options)

Completions

Make a call to the /completions endpoint of the Deployment. Only available for generative Hugging Face models

request_body = {
    "prompt": [
        "Tell me a joke",
        "Give a random fact"
    ],
    "logprobs": true,
    "max_tokens": 40
}

completions = client.completions(deployment_id, request_body)

# with explanation — only available for generative Hugging Face Deployments with a standard explainer
from deeploy.models.inference_options import CompletionsInferenceOptions
completions = client.completions(deployment_id, request_body, completions_options=CompletionsInferenceOptions(explain=True))

# with streaming
from deeploy.models import ClientOptions
stream = client.completions(deployment_id, request_body, client_options=ClientOptions(stream=True))
for chunk in stream:
    print(chunk)

Chat completions

Make a call to the /chat/completions endpoint of the Deployment. Only available for generative Hugging Face models.

request_body = {
    "logprobs": true,
    "messages": [
        {
          "role": "system",
          "content": "You are a helpful assistant that gives specific answers."
        },
        {
          "role": "user",
          "content": "what is 1 + 1?"
        }
    ],
    "model": "model", # only when inferencing an external Deployment
    "max_tokens": 50
}

chat_completions = client.chat_completions(deployment_id, request_body)

# with streaming
from deeploy.models import ClientOptions
stream = client.chat_completions(deployment_id, request_body, client_options=ClientOptions(stream=True))
for chunk in stream:
    print(chunk)

Embeddings

Make a call to the /embeddings endpoint of the Deployment. Only available for embedded Hugging Face models.

request_body = {
    "input": [
        "Tell me a joke", 
        "Wonderful world"
    ],
    "logprobs": true,
    "model": "model", # only when inferencing an external Deployment
    "max_tokens": 40
}

embeddings = client.embeddings(deployment_id, request_body)

Client options

All inference methods accept an optional ClientOptions object that controls request behaviour.

Option	Type	Default	Description
nr_requests	int	None	Number of parallel requests to split the batch into. See Parallel batch processing.
nr_retries	int	None	Number of times to retry a failed request. Uses exponential back-off (1 s, 2 s, 4 s, …).
progress	bool	True	Show a progress bar when processing batches in parallel.
stream	bool	None	Stream the response. Incompatible with nr_requests > 1.

from deeploy.models import ClientOptions

options = ClientOptions(nr_requests=4, nr_retries=2, progress=True)

Inference options

Each inference method also accepts an endpoint-specific options object. The parameter name and accepted class differ per method:

Method	Parameter	Options class	Available options
predict	inference_options	BaseInferenceOptions	skip_log
explain	explain_options	ExplainInferenceOptions	skip_log, image
completions	completions_options	CompletionsInferenceOptions	skip_log, explain
chat_completions	inference_options	BaseInferenceOptions	skip_log
embeddings	inference_options	BaseInferenceOptions	skip_log

BaseInferenceOptions

Option	Type	Default	Description
skip_log	bool	None	Skip logging this request in Deeploy.

ExplainInferenceOptions (extends BaseInferenceOptions)

Option	Type	Default	Description
skip_log	bool	None	Skip logging this request in Deeploy.
image	bool	None	Return the explanation as an image instead of JSON.

CompletionsInferenceOptions (extends BaseInferenceOptions)

Option	Type	Default	Description
skip_log	bool	None	Skip logging this request in Deeploy.
explain	bool	False	Return an explanation alongside the completion. Only available for Hugging Face Deployments with a standard explainer.

from deeploy.models.inference_options import BaseInferenceOptions, ExplainInferenceOptions, CompletionsInferenceOptions

# Skip logging a predict request
prediction = client.predict(deployment_id, request_body, inference_options=BaseInferenceOptions(skip_log=True))

# Return explanation if there is an image model with explainer
explanation = client.explain(deployment_id, request_body, explain_options=ExplainInferenceOptions(image=True))

# Return completion with explanation (if huggingface model deployed with explainer)
completions = client.completions(deployment_id, request_body, completions_options=CompletionsInferenceOptions(explain=True))

# Skip logging a chat completions or embeddings request
chat_completions = client.chat_completions(deployment_id, request_body, inference_options=BaseInferenceOptions(skip_log=True))
embeddings = client.embeddings(deployment_id, request_body, inference_options=BaseInferenceOptions(skip_log=True))

Parallel batch processing

When you have a large batch of instances, use nr_requests to split it into smaller sub-batches and send them concurrently. This can significantly reduce total latency.

Requirements

• The request body must contain multiple instances. Supported formats:
  • {"instances": [...]} — standard predict/explain format
  • {"inputs": [...]} — alternative predict format
  • A top-level list [...]
• nr_requests must be > 1.
• Streaming (stream=True) is not compatible with parallel processing.

note

Parallel processing is only supported for predict and explain. The following endpoints use request body keys that are not recognised by the batch splitter and will raise a ValueError if nr_requests > 1:

• completions — uses {"prompt": [...]}
• chat_completions — uses {"messages": [...]}
• embeddings — uses {"input": [...]} (singular)

Return value

When nr_requests > 1, the method returns a list of response dicts, one per sub-batch, in completion order (not necessarily the original order). Merge the results yourself if you need a single combined response.

from deeploy.models import ClientOptions

request_body = {
    "instances": [
        [39, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9],
        [52, 4, 0, 0, 0, 0, 2, 1, 0, 0, 40, 9],
        # ... many more instances
    ]
}

# Split into 4 parallel requests
options = ClientOptions(nr_requests=4, nr_retries=2, progress=True)

results = client.predict(deployment_id, request_body, client_options=options)
# results is a list of dicts, one per sub-batch:
# [{"predictions": [...]}, {"predictions": [...]}, ...]

# Merge predictions from all batches
all_predictions = [pred for batch in results for pred in batch["predictions"]]

If the batch cannot be split (e.g. only one instance), a ValueError is raised. If nr_requests exceeds the number of instances the batch is split into as many sub-batches as there are instances.