Rafay Adds Token Router Capabilities to Token Factory, Putting Frontier and Open-source Models Under One Spending Policy with Granular Token Budgets

New routing layer directs inference requests based on optimal model, priority, health and cost, providing a path to predictable AI spend

SUNNYVALE, Calif., Nov. 5, 2026 /PRNewswire/ -- Rafay Systems, the infrastructure orchestration and monetization platform for neoclouds, telcos and enterprise AI, today announced new capabilities in Rafay Token Factory that give operators control over where every inference request runs and what it costs. Rafay's Token Router, a new routing layer, sends each request to in-cluster model deployments or to external providers such as OpenAI, Anthropic and any OpenAI-compatible endpoint, based on priority, health and cost. Token Budgets let administrators set daily or monthly spending limits for an organization, a project or an individual user, and enforce them before a request is admitted.

The timing reflects a shift in how AI is consumed. Agentic workflows use more tokens per task, so total AI bills keep rising even as the price per token falls, and teams are moving high-volume work to open-source models to keep spending in check. Operators now need to serve frontier and open-source models side by side and control what each one costs.

Used together, the Token Factory and Token Router let operators offer frontier models and open-source models behind a single endpoint and shift traffic between them automatically. An enterprise can give its teams a frontier model while their daily budget lasts, then move them to an open-source model running on its own GPUs for the rest of the day, with no change to the applications making the calls.

The new capabilities are built for two groups: neoclouds delivering open-source models as a service, and enterprises operating internal token factories for their own developers and applications.

One endpoint for every model

Token Router sits between applications and every place inference can run. Operators manage it through one API, give consumers one set of credentials, and see token spend in one view. Behind that stable endpoint, operators can change where models come from without asking users to change their code.

• External models: Token Factory can now serve models from OpenAI, Anthropic and any OpenAI-compatible endpoint alongside in-cluster model deployments, with the same API surface, cost tracking and failover controls.
• Priority routing: Each backend holds ordered groups of providers. Traffic goes to the highest-priority group first and fails over to the next when the primary is unavailable. Common patterns include self-hosted first with burst to cloud, frontier first with an in-cluster fallback, and multi-vendor resilience across providers.
• Health-aware routing and load balancing: Requests are spread evenly across all healthy providers in a group, including a mix of in-cluster deployments and external APIs. Providers that fail health checks leave rotation automatically and return when they recover.
• Cost-aware metering: Operators set per-group rates for input, output, cached input and cache write tokens. Each request is metered at the rate of the group that served it, keeping showback, chargeback and billing accurate even when traffic reroutes.

Budgets that steer traffic, not only stop it

Request rate limits do not make AI spending predictable, because a call to a small model can cost a fraction of a cent while a long reasoning request to a premium model costs far more. Token Budgets set the limit in money instead:

• Three levels: Administrators set budgets for the organization, for each project and for each user, daily or monthly, in the currency the service provider supports. A user's budget follows them across projects.
• Enforcement before admission: Every request is checked against all applicable budgets before it is admitted. When a budget is exhausted, the operator's policy decides what happens next: deny further requests, keep serving, or fall back to a lower-cost model. Other teams keep working within their own limits.
• Defaults and overrides: New projects and users inherit default budgets, and administrators can raise them for heavier workloads without changing applications or endpoints.
• Alerts and self-service visibility: Threshold alerts notify admins before a limit is reached. Developers see their own spend and remaining budget in the Rafay Developer Hub, and budgets reset automatically each day or month.

With budget-based routing, Token Router can also act on those budgets. When a tenant, team or API key nears or reaches its budget, Token Router moves its traffic to a lower-cost group, such as a lower-priced or self-hosted model, instead of rejecting requests. Users keep getting answers, and spend stays within the limit the administrator set.

Built for neoclouds and enterprise token factories

Neoclouds delivering open-source models as a service can now add frontier models to their catalog on day one, through the same API and credentials their customers already use. A provider can sell each customer a daily allowance of frontier model usage and route the rest of the day's traffic to the open-source models it hosts on its own GPUs. Customers get the model they want when it matters most, the provider keeps its capacity busy, and every token is billed at the rate of the model that served it. Tenant administrators manage budgets for their own projects and users, so the provider does not have to govern every developer.

Enterprises operating internal token factories can give teams access to frontier models without an open-ended bill. A platform team can set a daily frontier quota for each team or application. Developers work with the frontier model until that quota is spent, then Token Router moves their requests to an open-source model running on the company's own GPUs for the rest of the day. Applications keep calling the same endpoint, and the quota resets the next day. A token usage dashboard with per-user filtering and CSV export supports internal showback and chargeback.

"Every operator we work with wants to offer the best models without handing its margins, or its budget, to someone else," said Mohan Atreya, chief product officer at Rafay Systems. "Token Router and Token Budgets turn that tradeoff into policy. Operators decide which model serves each request and what it costs, and applications never have to change. Neoclouds get a way to sell frontier access while keeping their own capacity busy, and enterprises get AI spending they can predict."

Also new in Rafay Token Factory

• Guardrail providers: Any guardrail service that implements the LiteLLM Generic Guardrail API can inspect prompts and responses on routed traffic, with the same policies applied to in-cluster and external models.
• Cached input token pricing: Operators can price tokens served from the prefix cache separately, passing prompt-caching savings on to agentic AI and retrieval workloads.
• API key rotation policies: Operations teams can enforce key expiration, automatic rotation with an overlap window, and alerts before keys expire.

Availability

Token Router, Token Budgets and the capabilities described above are available today to Rafay Token Factory customers. To learn more or request a demo, visit https://rafay.co/request-demo.

About Rafay Systems

Rafay Systems is a leading software provider powering operators building the AI cloud, including neoclouds, telecommunications providers, enterprises and sovereign AI operators. The Rafay Platform enables organizations to operationalize GPU and compute infrastructure through self-service automation, governance and multi-tenancy. Its capabilities span bare-metal provisioning, infrastructure lifecycle management, virtual machines, Kubernetes, GPU PaaS, AI development environments and Token Factory for publishing AI models as token-metered inference services. By simplifying orchestration and operations across this stack, Rafay helps operators increase GPU utilization and turn raw infrastructure into monetizable, production-ready AI services, all while maintaining security, consistency, and control. Rafay is a validated partner in the NVIDIA AI Cloud Ready initiative. For more information, visit rafay.co.

Media contact

Angela Shugarts | Rafay Systems | angela@rafay.co | 303-868-6294

OpenAI and Anthropic are trademarks of their respective owners. All other trademarks are the property of their respective owners.

SOURCE Rafay Systems