Inference
Serve popular open-source models behind an OpenAI-compatible API that scales with your traffic — no servers to manage, low latency worldwide, pay for what you serve.
Overview
Inference turns open models (Llama, Mistral, Qwen, DeepSeek, Gemma) into production endpoints in one step. Drop-in OpenAI-compatible API means existing clients just change a base URL. Autoscaling follows demand to zero and back; zero egress keeps costs predictable.
Loading live data…
Why teams choose Inference
Built on PodStack's proprietary PodVirt layer — not vanilla Kubernetes or OpenStack — so you get fractional GPUs, sub-second scaling, and one bill.
Low-latency endpoints for open-source models
OpenAI-compatible API - point your existing client at it
Autoscaling to match traffic
Fractional NVIDIA GPUs from 12.5% to 100% via PodVirt
Per-minute billing with zero egress fees
Runs on operator-owned, ISO 27001 certified hardware
Drop-in OpenAI-compatible API
Point your existing OpenAI client at PodStack Inference and serve open-source models behind autoscaling GPU endpoints. Scale to zero when traffic drops, scale up when it spikes — and pay per minute for the GPU fraction you use.
- ✓OpenAI-compatible chat/completions endpoint
- ✓Autoscaling on fractional NVIDIA GPUs
- ✓Low-latency serving from ISO 27001 data centres
- ✓Zero egress fees and per-minute billing
from openai import OpenAI
client = OpenAI(
base_url="https://api.podstack.ai/v1",
api_key="$PODSTACK_API_KEY"
)
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-70B-Instruct",
messages=[{"role":"user","content":"Hello"}]
)Use cases
Drop-in OpenAI replacement
Swap the base URL, keep your app.
Spiky production traffic
Autoscaling absorbs bursts, scales to zero when idle.
Data-sensitive serving
Run open models on isolated infra you can also self-host.
When to use Inference
- ✓You want to serve, not operate, model infra
- ✓You need an OpenAI-compatible API for open models
- ✓Your traffic is variable and you want usage-based cost
How to use
- 1
Pick a model
From the catalog.
- 2
Get your endpoint + key
- 3
Call it
inferencecurl https://api.podstack.ai/v1/chat/completions \ -H "Authorization: Bearer $PODSTACK_KEY" \ -d '{"model":"llama-3.1-70b","messages":[{"role":"user","content":"Hi"}]}'
Frequently asked questions
What is PodStack Inference?
Inference is PodStack’s product for serving open-source models as low-latency endpoints. It exposes an OpenAI-compatible API and autoscales on NVIDIA GPUs so you can ship models to production instantly.
Is the API OpenAI-compatible?
Yes. Inference endpoints speak the OpenAI-compatible API, so you can point an existing OpenAI client or SDK at your PodStack endpoint without rewriting your integration.
Which models can I serve?
Inference is built for open-source models. You bring the open model you want to serve and PodStack hosts it behind a low-latency, autoscaling endpoint.
How does autoscaling and billing work?
Endpoints autoscale to match traffic, and you are billed per minute for the GPU fraction in use with zero egress fees. Pricing is sales-led - contact sales@podstack.ai for a quote.
What GPUs power Inference?
Inference runs on NVIDIA A10G, L40S, A100 40/80GB and H100 80GB, all supporting fractional allocation from 12.5% to 100% via PodVirt, so you can right-size the GPU to the model.
Where does inference run and is it secure?
Endpoints run on PodStack’s operator-owned hardware in ISO 27001 certified data centres - no peer marketplace hosts - with one operator, one SLA and one bill.