improve model provider selection algorithm

This commit is contained in:
miloschwartz
2026-08-11 16:10:34 -04:00
parent 75c6af3b11
commit c42df737b0
11 changed files with 408 additions and 200 deletions
+285
View File
@@ -0,0 +1,285 @@
# AI Gateway Provider Selection
How the AI gateway picks which attached provider handles a request when an
inference resource has more than one AI provider.
**Code:**
- Route → capability binding: `server/routers/aiGateway/createAiGatewayRouter.ts`
- Request pipeline: `server/routers/aiGateway/pipeline.ts` (`selectProvider`)
- Tie-break scoring: `server/lib/aiProviderSelection.ts`
- Allow/block matching: `server/lib/aiModelKeyMatch.ts`
- Model catalog: `server/lib/aiModelCatalog.ts`
- Default capabilities per provider type: `server/lib/aiProviderDefaults.ts`
Overlapping model allows are permitted at save time. Selection happens at
request time. If the algorithm cannot confidently pick one provider, the
gateway returns `403` with an ambiguous-provider error.
## Selection Pipeline
Every gateway request runs through these steps in order. Each step narrows
the candidate set. Later steps only run when more than one provider remains.
```
1. Capability filter
2. Allow / block lists
3. Most specific allow pattern
4. Catalog ownership
5. Provider class preference
6. Ambiguous → error
```
### 1. Capability Filter
The incoming path selects a capability before any provider logic runs.
| Path | Capability |
|------|------------|
| `POST /v1/chat/completions` | `openai_chat` |
| `POST /v1/responses` | `openai_responses` |
| `POST /v1/messages` | `anthropic_messages` |
| Gemini / Vertex / Bedrock routes | their respective capability ids |
Only attached providers that advertise that capability stay in the candidate
set. Default capabilities do not overlap for native OpenAI vs Anthropic:
| Provider type | Default capabilities |
|---------------|----------------------|
| `openai` | `openai_chat`, `openai_responses` |
| `anthropic` | `anthropic_messages` |
| `openRouter` | `openai_chat` |
| `vercelAiGateway` | `openai_chat`, `openai_responses` |
| `microsoftFoundry` | `openai_chat`, `openai_responses`, `anthropic_messages` |
| `custom` | whatever was configured |
### 2. Allow / Block Lists
For each remaining provider, the gateway resolves the effective allow and
block patterns:
- **`inherit`**: use the provider's own model lists
- **`select`**: use the resource-selected subset of those lists
A candidate is kept only if `isAllowedByLists(requestedModel, allows, blocks)`
passes:
1. At least one allow pattern must match
2. No block pattern may match
Patterns support `*` and `?` globs (`gpt-*`, `claude-3-5-sonnet-?`).
### 3. Most Specific Allow Pattern
Among providers that allow the model, keep those whose matching allow
pattern is most specific:
1. Exact keys beat patterns
2. Fewer wildcard characters win
3. Longer literal length wins
Example: `gpt-4o` beats `gpt-*` beats `*`.
### 4. Catalog Ownership
When specificity is tied (common with multiple `*` allows), score each
provider against the known model catalog:
| Score | Meaning |
|------:|---------|
| 2 | Typed provider whose catalog contains the model (`openai` → openai catalog, `anthropic` → anthropic, etc.) |
| 1 | Aggregator or custom (`openRouter`, `vercelAiGateway`, `custom`) and the model exists somewhere in the catalog |
| 0 | No ownership signal (typed catalog miss, or unknown model on aggregator/custom) |
Model id lookup tries the raw id, then a stripped `vendor/model` form
(e.g. `openai/gpt-4o` → also try `gpt-4o`).
Typed providers map to catalog providers as:
| Provider type | Catalog |
|---------------|---------|
| `openai` | `openai` |
| `anthropic` | `anthropic` |
| `googleGemini` | `gemini` |
| `vertexAi` | `vertex` |
| `bedrock` | `bedrock` |
| `microsoftFoundry` | `azure` |
| `openRouter` / `vercelAiGateway` / `custom` | none (aggregator/custom path) |
### 5. Provider Class Preference
If catalog ownership is still tied, prefer:
| Rank | Class |
|-----:|-------|
| 2 | Native typed provider (`openai`, `anthropic`, `googleGemini`, …) |
| 1 | Aggregator (`openRouter`, `vercelAiGateway`) |
| 0 | `custom` |
### 6. Ambiguous Error
If more than one distinct provider remains after all steps, the gateway
rejects the request:
```
Model "<id>" is ambiguous across multiple AI providers on this resource
```
Typical remaining ties: two OpenAI-type providers both with `*`, or two
customs advertising the same capability for an unknown model.
## Examples
Assume each provider below is attached and enabled on the same inference
resource.
### Example A: OpenAI + Anthropic, Both `*`
| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenAI | `*` | `openai_chat`, `openai_responses` |
| Anthropic | `*` | `anthropic_messages` |
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
1. Capability → only OpenAI remains
2. Allow → OpenAI matches `*`
3. Result → **OpenAI**
Anthropic never reaches pattern or catalog scoring. Capability alone decides.
**Request:** `POST /v1/messages` with `model: "claude-3-5-sonnet-latest"`
1. Capability → only Anthropic remains
2. Result → **Anthropic**
### Example B: OpenAI + OpenRouter, Both `*`
| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenAI | `*` | `openai_chat`, … |
| OpenRouter | `*` | `openai_chat` |
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
1. Capability → both remain (`openai_chat`)
2. Allow → both match `*`
3. Specificity → tie (`*` vs `*`)
4. Catalog → OpenAI scores `2` (owns `gpt-4o`); OpenRouter scores `1`
5. Result → **OpenAI**
### Example C: OpenRouter Only Serving a Claude Model Over OpenAI Chat
| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenRouter | `*` | `openai_chat` |
**Request:** `POST /v1/chat/completions` with `model: "anthropic/claude-3.5-sonnet"`
1. Capability → OpenRouter remains
2. Only one candidate → **OpenRouter**
No tie-breaking needed.
### Example D: OpenAI (`gpt-*`) + OpenRouter (`*`)
| Provider | Allow |
|----------|-------|
| OpenAI | `gpt-*` |
| OpenRouter | `*` |
**Request:** `model: "gpt-4o"` on `openai_chat`
1. Capability → both
2. Allow → both match
3. Specificity → OpenAI's `gpt-*` beats OpenRouter's `*`
4. Result → **OpenAI**
Catalog scoring is not needed because specificity already unique'd the set.
### Example E: OpenAI + Anthropic With Overlapping Custom Capabilities
Someone grants Anthropic `openai_chat` as well (non-default).
| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenAI | `*` | `openai_chat`, … |
| Anthropic | `*` | `anthropic_messages`, `openai_chat` |
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
1. Capability → both remain
2. Allow → both match `*`
3. Specificity → tie
4. Catalog → OpenAI `2`, Anthropic `0` (`gpt-4o` is not in the anthropic catalog)
5. Result → **OpenAI**
### Example F: Two Aggregators, Known Model
| Provider | Allow |
|----------|-------|
| OpenRouter | `*` |
| Vercel AI Gateway | `*` |
**Request:** `model: "gpt-4o"` on `openai_chat`
1. Capability → both
2. Allow / specificity → tie
3. Catalog → both score `1` (known model, no typed owner in the set)
4. Class → both aggregators (rank `1`) → still tied
5. Result → **ambiguous error**
Attach a native OpenAI provider (or narrow one aggregator's allow list) to
make this determinable.
### Example G: Two OpenAI Providers, Both `*`
| Provider | Type | Allow |
|----------|------|-------|
| OpenAI Prod | `openai` | `*` |
| OpenAI Staging | `openai` | `*` |
**Request:** `model: "gpt-4o"`
15 all leave both candidates (same capability, same specificity, same
catalog ownership, same class).
Result → **ambiguous error**
Disambiguate with different allow patterns, disable one attachment, or
split across resources.
### Example H: Unknown Model Across Native + Aggregator
| Provider | Allow |
|----------|-------|
| OpenAI | `*` |
| OpenRouter | `*` |
**Request:** `model: "my-fine-tune-v3"` (not in catalog)
1. Capability → both
2. Allow / specificity → tie
3. Catalog → both score `0` (typed miss + unknown aggregator model)
4. Class → OpenAI (`2`) beats OpenRouter (`1`)
5. Result → **OpenAI**
## Practical Guidance
- Native OpenAI + Anthropic with `*` is safe. Different default APIs never
collide.
- OpenAI + OpenRouter with `*` is usually fine for catalog-known OpenAI
models. Native wins.
- Prefer specific allow patterns (`gpt-4o`, `gpt-*`) when two providers share
a capability.
- Two providers of the same type both using `*` will stay ambiguous. Narrow
at least one allow list.
- Custom providers only win ties when no stronger native/aggregator signal
remains.
## Related Behavior
- **Saving providers on a resource does not reject overlapping allows.**
Collisions are resolved (or rejected) per request.
- Budgets, auth, and upstream URL / target routing run after a single
provider has been selected.
-161
View File
@@ -10,7 +10,6 @@ import {
type Transaction type Transaction
} from "@server/db"; } from "@server/db";
import { z } from "zod"; import { z } from "zod";
import { modelKeysConflict } from "@server/lib/aiModelKeyMatch";
type DbOrTrx = Transaction | typeof db; type DbOrTrx = Transaction | typeof db;
@@ -100,142 +99,6 @@ function normalizeAttachments(
); );
} }
type EffectiveAllowRow = {
providerId: number;
modelKey: string;
};
/**
* Ensure effective allow modelKeys do not conflict across attached providers.
* inherit uses provider allows; select uses resource-selected allows (or the
* optional override map). Block patterns are ignored for overlap checks.
*/
export async function assertNoOverlappingModelKeys(
attachments: ResourceAiProviderAttachment[],
options: {
trx?: DbOrTrx;
resourceId?: number;
siteResourceId?: number;
selectedAllowsByProvider?: Map<number, string[]>;
} = {}
): Promise<InferenceFieldsError | null> {
const trx = options.trx ?? db;
const activeAttachments = attachments.filter((a) => a.enabled);
if (activeAttachments.length < 2) {
return null;
}
const inheritProviderIds = activeAttachments
.filter((a) => a.accessMode === "inherit")
.map((a) => a.providerId);
const selectProviderIds = activeAttachments
.filter((a) => a.accessMode === "select")
.map((a) => a.providerId);
const effectiveAllows: EffectiveAllowRow[] = [];
if (inheritProviderIds.length > 0) {
const providerAllows = await trx
.select({
providerId: aiModels.providerId,
modelKey: aiModels.modelKey
})
.from(aiModels)
.where(
and(
inArray(aiModels.providerId, inheritProviderIds),
eq(aiModels.enabled, true),
eq(aiModels.listType, "allow")
)
);
effectiveAllows.push(...providerAllows);
}
if (selectProviderIds.length > 0) {
if (options.selectedAllowsByProvider) {
for (const providerId of selectProviderIds) {
const keys =
options.selectedAllowsByProvider.get(providerId) ?? [];
for (const modelKey of keys) {
effectiveAllows.push({ providerId, modelKey });
}
}
} else if (options.resourceId !== undefined) {
const rows = await trx
.select({
providerId: aiModels.providerId,
modelKey: aiModels.modelKey
})
.from(resourceAiModels)
.innerJoin(
aiModels,
eq(resourceAiModels.modelId, aiModels.modelId)
)
.where(
and(
eq(resourceAiModels.resourceId, options.resourceId),
inArray(aiModels.providerId, selectProviderIds),
eq(resourceAiModels.listType, "allow"),
eq(aiModels.enabled, true)
)
);
effectiveAllows.push(...rows);
} else if (options.siteResourceId !== undefined) {
const rows = await trx
.select({
providerId: aiModels.providerId,
modelKey: aiModels.modelKey
})
.from(siteResourceAiModels)
.innerJoin(
aiModels,
eq(siteResourceAiModels.modelId, aiModels.modelId)
)
.where(
and(
eq(
siteResourceAiModels.siteResourceId,
options.siteResourceId
),
inArray(aiModels.providerId, selectProviderIds),
eq(siteResourceAiModels.listType, "allow"),
eq(aiModels.enabled, true)
)
);
effectiveAllows.push(...rows);
}
}
const conflictPairs: string[] = [];
for (let i = 0; i < effectiveAllows.length; i++) {
for (let j = i + 1; j < effectiveAllows.length; j++) {
const left = effectiveAllows[i];
const right = effectiveAllows[j];
if (left.providerId === right.providerId) {
continue;
}
if (!modelKeysConflict(left.modelKey, right.modelKey)) {
continue;
}
const pair = [left.modelKey, right.modelKey].sort().join(" vs ");
if (!conflictPairs.includes(pair)) {
conflictPairs.push(pair);
}
}
}
if (conflictPairs.length === 0) {
return null;
}
conflictPairs.sort();
return {
error: `Model keys must be unique across providers on a resource. Overlapping keys: ${conflictPairs.join(", ")}`
};
}
/** /**
* Validate provider attachments for an org. * Validate provider attachments for an org.
*/ */
@@ -243,8 +106,6 @@ export async function resolveProviderAttachments(input: {
orgId: string; orgId: string;
attachments: ResourceAiProviderInput[]; attachments: ResourceAiProviderInput[];
requireAtLeastOne: boolean; requireAtLeastOne: boolean;
resourceId?: number;
siteResourceId?: number;
}): Promise<ResourceAiProviderAttachment[] | InferenceFieldsError> { }): Promise<ResourceAiProviderAttachment[] | InferenceFieldsError> {
const attachments = normalizeAttachments(input.attachments); const attachments = normalizeAttachments(input.attachments);
@@ -286,14 +147,6 @@ export async function resolveProviderAttachments(input: {
}; };
} }
const overlapError = await assertNoOverlappingModelKeys(attachments, {
resourceId: input.resourceId,
siteResourceId: input.siteResourceId
});
if (overlapError) {
return overlapError;
}
return attachments; return attachments;
} }
@@ -830,7 +683,6 @@ async function assertModelEntriesValid(input: {
const catalogRows = await db const catalogRows = await db
.select({ .select({
modelId: aiModels.modelId, modelId: aiModels.modelId,
modelKey: aiModels.modelKey,
listType: aiModels.listType, listType: aiModels.listType,
providerId: aiModels.providerId, providerId: aiModels.providerId,
enabled: aiModels.enabled enabled: aiModels.enabled
@@ -850,7 +702,6 @@ async function assertModelEntriesValid(input: {
} }
const catalogById = new Map(catalogRows.map((row) => [row.modelId, row])); const catalogById = new Map(catalogRows.map((row) => [row.modelId, row]));
const selectedAllowsByProvider = new Map<number, string[]>();
for (const entry of input.modelEntries) { for (const entry of input.modelEntries) {
const catalog = catalogById.get(entry.modelId); const catalog = catalogById.get(entry.modelId);
if (!catalog) { if (!catalog) {
@@ -862,18 +713,6 @@ async function assertModelEntriesValid(input: {
if (!catalog.enabled) { if (!catalog.enabled) {
return `Model ${entry.modelId} is disabled on its provider`; return `Model ${entry.modelId} is disabled on its provider`;
} }
if (entry.listType === "allow") {
const keys = selectedAllowsByProvider.get(catalog.providerId) ?? [];
keys.push(catalog.modelKey);
selectedAllowsByProvider.set(catalog.providerId, keys);
}
}
const overlapError = await assertNoOverlappingModelKeys(input.attachments, {
selectedAllowsByProvider
});
if (overlapError) {
return overlapError.error;
} }
return null; return null;
-23
View File
@@ -61,29 +61,6 @@ export function compareModelKeySpecificity(a: string, b: string): number {
return literalLength(b) - literalLength(a); return literalLength(b) - literalLength(a);
} }
/**
* Attach-time conflict check. Detects identical keys and exact-vs-pattern
* matches. Does not attempt full glob intersection.
*/
export function modelKeysConflict(a: string, b: string): boolean {
if (a === b) {
return true;
}
const aIsPattern = isModelKeyPattern(a);
const bIsPattern = isModelKeyPattern(b);
if (aIsPattern === bIsPattern) {
return false;
}
if (aIsPattern) {
return modelKeyMatches(a, b);
}
return modelKeyMatches(b, a);
}
/** /**
* Provider-layer policy: empty allowlist denies all. Blocklist only applies * Provider-layer policy: empty allowlist denies all. Blocklist only applies
* after an allow match. * after an allow match.
+97
View File
@@ -0,0 +1,97 @@
import {
aiModelCatalog,
getCatalogProviderForType,
type CatalogProvider
} from "@server/lib/aiModelCatalog";
import type { AiProviderType } from "@server/lib/aiProviderDefaults";
function stripVendorPrefix(modelId: string): string | null {
const idx = modelId.indexOf("/");
if (idx === -1 || idx === modelId.length - 1) {
return null;
}
return modelId.slice(idx + 1);
}
function modelKeysToTry(modelId: string): string[] {
const keys = [modelId];
const stripped = stripVendorPrefix(modelId);
if (stripped) {
keys.push(stripped);
}
return keys;
}
function catalogOwnsModel(
catalogProvider: CatalogProvider,
modelId: string
): boolean {
for (const key of modelKeysToTry(modelId)) {
if (aiModelCatalog.get(catalogProvider, key)) {
return true;
}
}
return false;
}
function modelKnownInAnyCatalog(modelId: string): boolean {
for (const key of modelKeysToTry(modelId)) {
if (aiModelCatalog.listByKey(key).length > 0) {
return true;
}
}
return false;
}
/**
* How strongly a provider "owns" a requested model id via the known catalog.
*
* 2 - Typed provider whose catalog contains the model
* 1 - Aggregator/custom that can proxy a catalog-known model
* 0 - No ownership signal (typed miss, or unknown model on aggregator/custom)
*/
export function catalogOwnershipScore(
type: AiProviderType,
modelId: string
): number {
const catalogProvider = getCatalogProviderForType(type);
if (catalogProvider != null) {
return catalogOwnsModel(catalogProvider, modelId) ? 2 : 0;
}
return modelKnownInAnyCatalog(modelId) ? 1 : 0;
}
/**
* Prefer native vendor providers over aggregators over custom when catalog
* ownership is tied.
*
* 2 - Native typed provider (openai, anthropic, gemini, ...)
* 1 - Aggregator gateway (openRouter, vercelAiGateway)
* 0 - Custom
*/
export function providerClassRank(type: AiProviderType): number {
if (type === "custom") {
return 0;
}
if (type === "openRouter" || type === "vercelAiGateway") {
return 1;
}
return 2;
}
export function keepBestScored<T>(
items: T[],
scoreFn: (item: T) => number
): T[] {
if (items.length <= 1) {
return items;
}
let best = Number.NEGATIVE_INFINITY;
for (const item of items) {
const score = scoreFn(item);
if (score > best) {
best = score;
}
}
return items.filter((item) => scoreFn(item) === best);
}
+20 -4
View File
@@ -50,6 +50,11 @@ import {
isAllowedByLists, isAllowedByLists,
mostSpecificMatchingAllow mostSpecificMatchingAllow
} from "@server/lib/aiModelKeyMatch"; } from "@server/lib/aiModelKeyMatch";
import {
catalogOwnershipScore,
keepBestScored,
providerClassRank
} from "@server/lib/aiProviderSelection";
import { aiGatewayUpstreamFetch } from "@server/lib/aiGatewayUpstreamFetch"; import { aiGatewayUpstreamFetch } from "@server/lib/aiGatewayUpstreamFetch";
import { getModelPricing, calculateAiCost } from "@server/lib/aiModelPricing"; import { getModelPricing, calculateAiCost } from "@server/lib/aiModelPricing";
import { import {
@@ -502,17 +507,28 @@ async function selectProvider(
}; };
} }
// 1) Prefer the most specific allow pattern that matched the request.
candidates.sort((a, b) => candidates.sort((a, b) =>
compareModelKeySpecificity(a.modelKey, b.modelKey) compareModelKeySpecificity(a.modelKey, b.modelKey)
); );
const bestSpecificity = candidates[0].modelKey; const bestSpecificity = candidates[0].modelKey;
const topCandidates = candidates.filter( let remaining = candidates.filter(
(c) => compareModelKeySpecificity(c.modelKey, bestSpecificity) === 0 (c) => compareModelKeySpecificity(c.modelKey, bestSpecificity) === 0
); );
// 2) Prefer providers whose catalog owns this model id. Aggregators only
// score when the model is known somewhere in the catalog.
remaining = keepBestScored(remaining, (c) =>
catalogOwnershipScore(c.provider.type as AiProviderType, requestedModel)
);
// 3) Prefer native typed providers over aggregators over custom.
remaining = keepBestScored(remaining, (c) =>
providerClassRank(c.provider.type as AiProviderType)
);
const uniqueProviders = new Map<number, AiProvider>(); const uniqueProviders = new Map<number, AiProvider>();
for (const candidate of topCandidates) { for (const candidate of remaining) {
uniqueProviders.set(candidate.provider.providerId, candidate.provider); uniqueProviders.set(candidate.provider.providerId, candidate.provider);
} }
@@ -523,7 +539,7 @@ async function selectProvider(
return { return {
ok: false, ok: false,
status: HttpCode.FORBIDDEN, status: HttpCode.FORBIDDEN,
message: `Model "${requestedModel}" is ambiguous across multiple AI providers on this resource` message: `Model "${requestedModel}" is ambiguous across multiple AI providers on this resource. Ask your administrator to configure a more specific allow pattern for this model.`
}; };
} }
@@ -131,8 +131,7 @@ export async function addAiProviderToResource(
const attachments = await resolveProviderAttachments({ const attachments = await resolveProviderAttachments({
orgId: resource.orgId, orgId: resource.orgId,
attachments: nextAttachments, attachments: nextAttachments,
requireAtLeastOne: true, requireAtLeastOne: true
resourceId
}); });
if (isInferenceFieldsError(attachments)) { if (isInferenceFieldsError(attachments)) {
return next( return next(
@@ -134,8 +134,7 @@ export async function removeAiProviderFromResource(
const attachments = await resolveProviderAttachments({ const attachments = await resolveProviderAttachments({
orgId: resource.orgId, orgId: resource.orgId,
attachments: remaining, attachments: remaining,
requireAtLeastOne: false, requireAtLeastOne: false
resourceId
}); });
if (isInferenceFieldsError(attachments)) { if (isInferenceFieldsError(attachments)) {
return next( return next(
@@ -113,8 +113,7 @@ export async function setResourceAiProviders(
const attachments = await resolveProviderAttachments({ const attachments = await resolveProviderAttachments({
orgId: resource.orgId, orgId: resource.orgId,
attachments: providers, attachments: providers,
requireAtLeastOne: false, requireAtLeastOne: false
resourceId
}); });
if (isInferenceFieldsError(attachments)) { if (isInferenceFieldsError(attachments)) {
return next( return next(
@@ -131,8 +131,7 @@ export async function addAiProviderToSiteResource(
const attachments = await resolveProviderAttachments({ const attachments = await resolveProviderAttachments({
orgId: siteResource.orgId, orgId: siteResource.orgId,
attachments: nextAttachments, attachments: nextAttachments,
requireAtLeastOne: true, requireAtLeastOne: true
siteResourceId
}); });
if (isInferenceFieldsError(attachments)) { if (isInferenceFieldsError(attachments)) {
return next( return next(
@@ -133,8 +133,7 @@ export async function removeAiProviderFromSiteResource(
const attachments = await resolveProviderAttachments({ const attachments = await resolveProviderAttachments({
orgId: siteResource.orgId, orgId: siteResource.orgId,
attachments: remaining, attachments: remaining,
requireAtLeastOne: false, requireAtLeastOne: false
siteResourceId
}); });
if (isInferenceFieldsError(attachments)) { if (isInferenceFieldsError(attachments)) {
return next( return next(
@@ -115,8 +115,7 @@ export async function setSiteResourceAiProviders(
const attachments = await resolveProviderAttachments({ const attachments = await resolveProviderAttachments({
orgId: siteResource.orgId, orgId: siteResource.orgId,
attachments: providers, attachments: providers,
requireAtLeastOne: false, requireAtLeastOne: false
siteResourceId
}); });
if (isInferenceFieldsError(attachments)) { if (isInferenceFieldsError(attachments)) {
return next( return next(