mirror of
https://github.com/fosrl/pangolin.git
synced 2026-08-12 15:30:53 +02:00
improve model provider selection algorithm
This commit is contained in:
@@ -0,0 +1,285 @@
|
|||||||
|
# AI Gateway Provider Selection
|
||||||
|
|
||||||
|
How the AI gateway picks which attached provider handles a request when an
|
||||||
|
inference resource has more than one AI provider.
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
|
||||||
|
- Route → capability binding: `server/routers/aiGateway/createAiGatewayRouter.ts`
|
||||||
|
- Request pipeline: `server/routers/aiGateway/pipeline.ts` (`selectProvider`)
|
||||||
|
- Tie-break scoring: `server/lib/aiProviderSelection.ts`
|
||||||
|
- Allow/block matching: `server/lib/aiModelKeyMatch.ts`
|
||||||
|
- Model catalog: `server/lib/aiModelCatalog.ts`
|
||||||
|
- Default capabilities per provider type: `server/lib/aiProviderDefaults.ts`
|
||||||
|
|
||||||
|
Overlapping model allows are permitted at save time. Selection happens at
|
||||||
|
request time. If the algorithm cannot confidently pick one provider, the
|
||||||
|
gateway returns `403` with an ambiguous-provider error.
|
||||||
|
|
||||||
|
## Selection Pipeline
|
||||||
|
|
||||||
|
Every gateway request runs through these steps in order. Each step narrows
|
||||||
|
the candidate set. Later steps only run when more than one provider remains.
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Capability filter
|
||||||
|
2. Allow / block lists
|
||||||
|
3. Most specific allow pattern
|
||||||
|
4. Catalog ownership
|
||||||
|
5. Provider class preference
|
||||||
|
6. Ambiguous → error
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1. Capability Filter
|
||||||
|
|
||||||
|
The incoming path selects a capability before any provider logic runs.
|
||||||
|
|
||||||
|
| Path | Capability |
|
||||||
|
|------|------------|
|
||||||
|
| `POST /v1/chat/completions` | `openai_chat` |
|
||||||
|
| `POST /v1/responses` | `openai_responses` |
|
||||||
|
| `POST /v1/messages` | `anthropic_messages` |
|
||||||
|
| Gemini / Vertex / Bedrock routes | their respective capability ids |
|
||||||
|
|
||||||
|
Only attached providers that advertise that capability stay in the candidate
|
||||||
|
set. Default capabilities do not overlap for native OpenAI vs Anthropic:
|
||||||
|
|
||||||
|
| Provider type | Default capabilities |
|
||||||
|
|---------------|----------------------|
|
||||||
|
| `openai` | `openai_chat`, `openai_responses` |
|
||||||
|
| `anthropic` | `anthropic_messages` |
|
||||||
|
| `openRouter` | `openai_chat` |
|
||||||
|
| `vercelAiGateway` | `openai_chat`, `openai_responses` |
|
||||||
|
| `microsoftFoundry` | `openai_chat`, `openai_responses`, `anthropic_messages` |
|
||||||
|
| `custom` | whatever was configured |
|
||||||
|
|
||||||
|
### 2. Allow / Block Lists
|
||||||
|
|
||||||
|
For each remaining provider, the gateway resolves the effective allow and
|
||||||
|
block patterns:
|
||||||
|
|
||||||
|
- **`inherit`**: use the provider's own model lists
|
||||||
|
- **`select`**: use the resource-selected subset of those lists
|
||||||
|
|
||||||
|
A candidate is kept only if `isAllowedByLists(requestedModel, allows, blocks)`
|
||||||
|
passes:
|
||||||
|
|
||||||
|
1. At least one allow pattern must match
|
||||||
|
2. No block pattern may match
|
||||||
|
|
||||||
|
Patterns support `*` and `?` globs (`gpt-*`, `claude-3-5-sonnet-?`).
|
||||||
|
|
||||||
|
### 3. Most Specific Allow Pattern
|
||||||
|
|
||||||
|
Among providers that allow the model, keep those whose matching allow
|
||||||
|
pattern is most specific:
|
||||||
|
|
||||||
|
1. Exact keys beat patterns
|
||||||
|
2. Fewer wildcard characters win
|
||||||
|
3. Longer literal length wins
|
||||||
|
|
||||||
|
Example: `gpt-4o` beats `gpt-*` beats `*`.
|
||||||
|
|
||||||
|
### 4. Catalog Ownership
|
||||||
|
|
||||||
|
When specificity is tied (common with multiple `*` allows), score each
|
||||||
|
provider against the known model catalog:
|
||||||
|
|
||||||
|
| Score | Meaning |
|
||||||
|
|------:|---------|
|
||||||
|
| 2 | Typed provider whose catalog contains the model (`openai` → openai catalog, `anthropic` → anthropic, etc.) |
|
||||||
|
| 1 | Aggregator or custom (`openRouter`, `vercelAiGateway`, `custom`) and the model exists somewhere in the catalog |
|
||||||
|
| 0 | No ownership signal (typed catalog miss, or unknown model on aggregator/custom) |
|
||||||
|
|
||||||
|
Model id lookup tries the raw id, then a stripped `vendor/model` form
|
||||||
|
(e.g. `openai/gpt-4o` → also try `gpt-4o`).
|
||||||
|
|
||||||
|
Typed providers map to catalog providers as:
|
||||||
|
|
||||||
|
| Provider type | Catalog |
|
||||||
|
|---------------|---------|
|
||||||
|
| `openai` | `openai` |
|
||||||
|
| `anthropic` | `anthropic` |
|
||||||
|
| `googleGemini` | `gemini` |
|
||||||
|
| `vertexAi` | `vertex` |
|
||||||
|
| `bedrock` | `bedrock` |
|
||||||
|
| `microsoftFoundry` | `azure` |
|
||||||
|
| `openRouter` / `vercelAiGateway` / `custom` | none (aggregator/custom path) |
|
||||||
|
|
||||||
|
### 5. Provider Class Preference
|
||||||
|
|
||||||
|
If catalog ownership is still tied, prefer:
|
||||||
|
|
||||||
|
| Rank | Class |
|
||||||
|
|-----:|-------|
|
||||||
|
| 2 | Native typed provider (`openai`, `anthropic`, `googleGemini`, …) |
|
||||||
|
| 1 | Aggregator (`openRouter`, `vercelAiGateway`) |
|
||||||
|
| 0 | `custom` |
|
||||||
|
|
||||||
|
### 6. Ambiguous Error
|
||||||
|
|
||||||
|
If more than one distinct provider remains after all steps, the gateway
|
||||||
|
rejects the request:
|
||||||
|
|
||||||
|
```
|
||||||
|
Model "<id>" is ambiguous across multiple AI providers on this resource
|
||||||
|
```
|
||||||
|
|
||||||
|
Typical remaining ties: two OpenAI-type providers both with `*`, or two
|
||||||
|
customs advertising the same capability for an unknown model.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
Assume each provider below is attached and enabled on the same inference
|
||||||
|
resource.
|
||||||
|
|
||||||
|
### Example A: OpenAI + Anthropic, Both `*`
|
||||||
|
|
||||||
|
| Provider | Allow | Capabilities |
|
||||||
|
|----------|-------|--------------|
|
||||||
|
| OpenAI | `*` | `openai_chat`, `openai_responses` |
|
||||||
|
| Anthropic | `*` | `anthropic_messages` |
|
||||||
|
|
||||||
|
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
|
||||||
|
|
||||||
|
1. Capability → only OpenAI remains
|
||||||
|
2. Allow → OpenAI matches `*`
|
||||||
|
3. Result → **OpenAI**
|
||||||
|
|
||||||
|
Anthropic never reaches pattern or catalog scoring. Capability alone decides.
|
||||||
|
|
||||||
|
**Request:** `POST /v1/messages` with `model: "claude-3-5-sonnet-latest"`
|
||||||
|
|
||||||
|
1. Capability → only Anthropic remains
|
||||||
|
2. Result → **Anthropic**
|
||||||
|
|
||||||
|
### Example B: OpenAI + OpenRouter, Both `*`
|
||||||
|
|
||||||
|
| Provider | Allow | Capabilities |
|
||||||
|
|----------|-------|--------------|
|
||||||
|
| OpenAI | `*` | `openai_chat`, … |
|
||||||
|
| OpenRouter | `*` | `openai_chat` |
|
||||||
|
|
||||||
|
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
|
||||||
|
|
||||||
|
1. Capability → both remain (`openai_chat`)
|
||||||
|
2. Allow → both match `*`
|
||||||
|
3. Specificity → tie (`*` vs `*`)
|
||||||
|
4. Catalog → OpenAI scores `2` (owns `gpt-4o`); OpenRouter scores `1`
|
||||||
|
5. Result → **OpenAI**
|
||||||
|
|
||||||
|
### Example C: OpenRouter Only Serving a Claude Model Over OpenAI Chat
|
||||||
|
|
||||||
|
| Provider | Allow | Capabilities |
|
||||||
|
|----------|-------|--------------|
|
||||||
|
| OpenRouter | `*` | `openai_chat` |
|
||||||
|
|
||||||
|
**Request:** `POST /v1/chat/completions` with `model: "anthropic/claude-3.5-sonnet"`
|
||||||
|
|
||||||
|
1. Capability → OpenRouter remains
|
||||||
|
2. Only one candidate → **OpenRouter**
|
||||||
|
|
||||||
|
No tie-breaking needed.
|
||||||
|
|
||||||
|
### Example D: OpenAI (`gpt-*`) + OpenRouter (`*`)
|
||||||
|
|
||||||
|
| Provider | Allow |
|
||||||
|
|----------|-------|
|
||||||
|
| OpenAI | `gpt-*` |
|
||||||
|
| OpenRouter | `*` |
|
||||||
|
|
||||||
|
**Request:** `model: "gpt-4o"` on `openai_chat`
|
||||||
|
|
||||||
|
1. Capability → both
|
||||||
|
2. Allow → both match
|
||||||
|
3. Specificity → OpenAI's `gpt-*` beats OpenRouter's `*`
|
||||||
|
4. Result → **OpenAI**
|
||||||
|
|
||||||
|
Catalog scoring is not needed because specificity already unique'd the set.
|
||||||
|
|
||||||
|
### Example E: OpenAI + Anthropic With Overlapping Custom Capabilities
|
||||||
|
|
||||||
|
Someone grants Anthropic `openai_chat` as well (non-default).
|
||||||
|
|
||||||
|
| Provider | Allow | Capabilities |
|
||||||
|
|----------|-------|--------------|
|
||||||
|
| OpenAI | `*` | `openai_chat`, … |
|
||||||
|
| Anthropic | `*` | `anthropic_messages`, `openai_chat` |
|
||||||
|
|
||||||
|
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
|
||||||
|
|
||||||
|
1. Capability → both remain
|
||||||
|
2. Allow → both match `*`
|
||||||
|
3. Specificity → tie
|
||||||
|
4. Catalog → OpenAI `2`, Anthropic `0` (`gpt-4o` is not in the anthropic catalog)
|
||||||
|
5. Result → **OpenAI**
|
||||||
|
|
||||||
|
### Example F: Two Aggregators, Known Model
|
||||||
|
|
||||||
|
| Provider | Allow |
|
||||||
|
|----------|-------|
|
||||||
|
| OpenRouter | `*` |
|
||||||
|
| Vercel AI Gateway | `*` |
|
||||||
|
|
||||||
|
**Request:** `model: "gpt-4o"` on `openai_chat`
|
||||||
|
|
||||||
|
1. Capability → both
|
||||||
|
2. Allow / specificity → tie
|
||||||
|
3. Catalog → both score `1` (known model, no typed owner in the set)
|
||||||
|
4. Class → both aggregators (rank `1`) → still tied
|
||||||
|
5. Result → **ambiguous error**
|
||||||
|
|
||||||
|
Attach a native OpenAI provider (or narrow one aggregator's allow list) to
|
||||||
|
make this determinable.
|
||||||
|
|
||||||
|
### Example G: Two OpenAI Providers, Both `*`
|
||||||
|
|
||||||
|
| Provider | Type | Allow |
|
||||||
|
|----------|------|-------|
|
||||||
|
| OpenAI Prod | `openai` | `*` |
|
||||||
|
| OpenAI Staging | `openai` | `*` |
|
||||||
|
|
||||||
|
**Request:** `model: "gpt-4o"`
|
||||||
|
|
||||||
|
1–5 all leave both candidates (same capability, same specificity, same
|
||||||
|
catalog ownership, same class).
|
||||||
|
|
||||||
|
Result → **ambiguous error**
|
||||||
|
|
||||||
|
Disambiguate with different allow patterns, disable one attachment, or
|
||||||
|
split across resources.
|
||||||
|
|
||||||
|
### Example H: Unknown Model Across Native + Aggregator
|
||||||
|
|
||||||
|
| Provider | Allow |
|
||||||
|
|----------|-------|
|
||||||
|
| OpenAI | `*` |
|
||||||
|
| OpenRouter | `*` |
|
||||||
|
|
||||||
|
**Request:** `model: "my-fine-tune-v3"` (not in catalog)
|
||||||
|
|
||||||
|
1. Capability → both
|
||||||
|
2. Allow / specificity → tie
|
||||||
|
3. Catalog → both score `0` (typed miss + unknown aggregator model)
|
||||||
|
4. Class → OpenAI (`2`) beats OpenRouter (`1`)
|
||||||
|
5. Result → **OpenAI**
|
||||||
|
|
||||||
|
## Practical Guidance
|
||||||
|
|
||||||
|
- Native OpenAI + Anthropic with `*` is safe. Different default APIs never
|
||||||
|
collide.
|
||||||
|
- OpenAI + OpenRouter with `*` is usually fine for catalog-known OpenAI
|
||||||
|
models. Native wins.
|
||||||
|
- Prefer specific allow patterns (`gpt-4o`, `gpt-*`) when two providers share
|
||||||
|
a capability.
|
||||||
|
- Two providers of the same type both using `*` will stay ambiguous. Narrow
|
||||||
|
at least one allow list.
|
||||||
|
- Custom providers only win ties when no stronger native/aggregator signal
|
||||||
|
remains.
|
||||||
|
|
||||||
|
## Related Behavior
|
||||||
|
|
||||||
|
- **Saving providers on a resource does not reject overlapping allows.**
|
||||||
|
Collisions are resolved (or rejected) per request.
|
||||||
|
- Budgets, auth, and upstream URL / target routing run after a single
|
||||||
|
provider has been selected.
|
||||||
@@ -10,7 +10,6 @@ import {
|
|||||||
type Transaction
|
type Transaction
|
||||||
} from "@server/db";
|
} from "@server/db";
|
||||||
import { z } from "zod";
|
import { z } from "zod";
|
||||||
import { modelKeysConflict } from "@server/lib/aiModelKeyMatch";
|
|
||||||
|
|
||||||
type DbOrTrx = Transaction | typeof db;
|
type DbOrTrx = Transaction | typeof db;
|
||||||
|
|
||||||
@@ -100,142 +99,6 @@ function normalizeAttachments(
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
type EffectiveAllowRow = {
|
|
||||||
providerId: number;
|
|
||||||
modelKey: string;
|
|
||||||
};
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Ensure effective allow modelKeys do not conflict across attached providers.
|
|
||||||
* inherit uses provider allows; select uses resource-selected allows (or the
|
|
||||||
* optional override map). Block patterns are ignored for overlap checks.
|
|
||||||
*/
|
|
||||||
export async function assertNoOverlappingModelKeys(
|
|
||||||
attachments: ResourceAiProviderAttachment[],
|
|
||||||
options: {
|
|
||||||
trx?: DbOrTrx;
|
|
||||||
resourceId?: number;
|
|
||||||
siteResourceId?: number;
|
|
||||||
selectedAllowsByProvider?: Map<number, string[]>;
|
|
||||||
} = {}
|
|
||||||
): Promise<InferenceFieldsError | null> {
|
|
||||||
const trx = options.trx ?? db;
|
|
||||||
|
|
||||||
const activeAttachments = attachments.filter((a) => a.enabled);
|
|
||||||
|
|
||||||
if (activeAttachments.length < 2) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
const inheritProviderIds = activeAttachments
|
|
||||||
.filter((a) => a.accessMode === "inherit")
|
|
||||||
.map((a) => a.providerId);
|
|
||||||
const selectProviderIds = activeAttachments
|
|
||||||
.filter((a) => a.accessMode === "select")
|
|
||||||
.map((a) => a.providerId);
|
|
||||||
|
|
||||||
const effectiveAllows: EffectiveAllowRow[] = [];
|
|
||||||
|
|
||||||
if (inheritProviderIds.length > 0) {
|
|
||||||
const providerAllows = await trx
|
|
||||||
.select({
|
|
||||||
providerId: aiModels.providerId,
|
|
||||||
modelKey: aiModels.modelKey
|
|
||||||
})
|
|
||||||
.from(aiModels)
|
|
||||||
.where(
|
|
||||||
and(
|
|
||||||
inArray(aiModels.providerId, inheritProviderIds),
|
|
||||||
eq(aiModels.enabled, true),
|
|
||||||
eq(aiModels.listType, "allow")
|
|
||||||
)
|
|
||||||
);
|
|
||||||
effectiveAllows.push(...providerAllows);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (selectProviderIds.length > 0) {
|
|
||||||
if (options.selectedAllowsByProvider) {
|
|
||||||
for (const providerId of selectProviderIds) {
|
|
||||||
const keys =
|
|
||||||
options.selectedAllowsByProvider.get(providerId) ?? [];
|
|
||||||
for (const modelKey of keys) {
|
|
||||||
effectiveAllows.push({ providerId, modelKey });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if (options.resourceId !== undefined) {
|
|
||||||
const rows = await trx
|
|
||||||
.select({
|
|
||||||
providerId: aiModels.providerId,
|
|
||||||
modelKey: aiModels.modelKey
|
|
||||||
})
|
|
||||||
.from(resourceAiModels)
|
|
||||||
.innerJoin(
|
|
||||||
aiModels,
|
|
||||||
eq(resourceAiModels.modelId, aiModels.modelId)
|
|
||||||
)
|
|
||||||
.where(
|
|
||||||
and(
|
|
||||||
eq(resourceAiModels.resourceId, options.resourceId),
|
|
||||||
inArray(aiModels.providerId, selectProviderIds),
|
|
||||||
eq(resourceAiModels.listType, "allow"),
|
|
||||||
eq(aiModels.enabled, true)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
effectiveAllows.push(...rows);
|
|
||||||
} else if (options.siteResourceId !== undefined) {
|
|
||||||
const rows = await trx
|
|
||||||
.select({
|
|
||||||
providerId: aiModels.providerId,
|
|
||||||
modelKey: aiModels.modelKey
|
|
||||||
})
|
|
||||||
.from(siteResourceAiModels)
|
|
||||||
.innerJoin(
|
|
||||||
aiModels,
|
|
||||||
eq(siteResourceAiModels.modelId, aiModels.modelId)
|
|
||||||
)
|
|
||||||
.where(
|
|
||||||
and(
|
|
||||||
eq(
|
|
||||||
siteResourceAiModels.siteResourceId,
|
|
||||||
options.siteResourceId
|
|
||||||
),
|
|
||||||
inArray(aiModels.providerId, selectProviderIds),
|
|
||||||
eq(siteResourceAiModels.listType, "allow"),
|
|
||||||
eq(aiModels.enabled, true)
|
|
||||||
)
|
|
||||||
);
|
|
||||||
effectiveAllows.push(...rows);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const conflictPairs: string[] = [];
|
|
||||||
for (let i = 0; i < effectiveAllows.length; i++) {
|
|
||||||
for (let j = i + 1; j < effectiveAllows.length; j++) {
|
|
||||||
const left = effectiveAllows[i];
|
|
||||||
const right = effectiveAllows[j];
|
|
||||||
if (left.providerId === right.providerId) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (!modelKeysConflict(left.modelKey, right.modelKey)) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
const pair = [left.modelKey, right.modelKey].sort().join(" vs ");
|
|
||||||
if (!conflictPairs.includes(pair)) {
|
|
||||||
conflictPairs.push(pair);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (conflictPairs.length === 0) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
conflictPairs.sort();
|
|
||||||
return {
|
|
||||||
error: `Model keys must be unique across providers on a resource. Overlapping keys: ${conflictPairs.join(", ")}`
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Validate provider attachments for an org.
|
* Validate provider attachments for an org.
|
||||||
*/
|
*/
|
||||||
@@ -243,8 +106,6 @@ export async function resolveProviderAttachments(input: {
|
|||||||
orgId: string;
|
orgId: string;
|
||||||
attachments: ResourceAiProviderInput[];
|
attachments: ResourceAiProviderInput[];
|
||||||
requireAtLeastOne: boolean;
|
requireAtLeastOne: boolean;
|
||||||
resourceId?: number;
|
|
||||||
siteResourceId?: number;
|
|
||||||
}): Promise<ResourceAiProviderAttachment[] | InferenceFieldsError> {
|
}): Promise<ResourceAiProviderAttachment[] | InferenceFieldsError> {
|
||||||
const attachments = normalizeAttachments(input.attachments);
|
const attachments = normalizeAttachments(input.attachments);
|
||||||
|
|
||||||
@@ -286,14 +147,6 @@ export async function resolveProviderAttachments(input: {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
const overlapError = await assertNoOverlappingModelKeys(attachments, {
|
|
||||||
resourceId: input.resourceId,
|
|
||||||
siteResourceId: input.siteResourceId
|
|
||||||
});
|
|
||||||
if (overlapError) {
|
|
||||||
return overlapError;
|
|
||||||
}
|
|
||||||
|
|
||||||
return attachments;
|
return attachments;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -830,7 +683,6 @@ async function assertModelEntriesValid(input: {
|
|||||||
const catalogRows = await db
|
const catalogRows = await db
|
||||||
.select({
|
.select({
|
||||||
modelId: aiModels.modelId,
|
modelId: aiModels.modelId,
|
||||||
modelKey: aiModels.modelKey,
|
|
||||||
listType: aiModels.listType,
|
listType: aiModels.listType,
|
||||||
providerId: aiModels.providerId,
|
providerId: aiModels.providerId,
|
||||||
enabled: aiModels.enabled
|
enabled: aiModels.enabled
|
||||||
@@ -850,7 +702,6 @@ async function assertModelEntriesValid(input: {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const catalogById = new Map(catalogRows.map((row) => [row.modelId, row]));
|
const catalogById = new Map(catalogRows.map((row) => [row.modelId, row]));
|
||||||
const selectedAllowsByProvider = new Map<number, string[]>();
|
|
||||||
for (const entry of input.modelEntries) {
|
for (const entry of input.modelEntries) {
|
||||||
const catalog = catalogById.get(entry.modelId);
|
const catalog = catalogById.get(entry.modelId);
|
||||||
if (!catalog) {
|
if (!catalog) {
|
||||||
@@ -862,18 +713,6 @@ async function assertModelEntriesValid(input: {
|
|||||||
if (!catalog.enabled) {
|
if (!catalog.enabled) {
|
||||||
return `Model ${entry.modelId} is disabled on its provider`;
|
return `Model ${entry.modelId} is disabled on its provider`;
|
||||||
}
|
}
|
||||||
if (entry.listType === "allow") {
|
|
||||||
const keys = selectedAllowsByProvider.get(catalog.providerId) ?? [];
|
|
||||||
keys.push(catalog.modelKey);
|
|
||||||
selectedAllowsByProvider.set(catalog.providerId, keys);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const overlapError = await assertNoOverlappingModelKeys(input.attachments, {
|
|
||||||
selectedAllowsByProvider
|
|
||||||
});
|
|
||||||
if (overlapError) {
|
|
||||||
return overlapError.error;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return null;
|
return null;
|
||||||
|
|||||||
@@ -61,29 +61,6 @@ export function compareModelKeySpecificity(a: string, b: string): number {
|
|||||||
return literalLength(b) - literalLength(a);
|
return literalLength(b) - literalLength(a);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Attach-time conflict check. Detects identical keys and exact-vs-pattern
|
|
||||||
* matches. Does not attempt full glob intersection.
|
|
||||||
*/
|
|
||||||
export function modelKeysConflict(a: string, b: string): boolean {
|
|
||||||
if (a === b) {
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
const aIsPattern = isModelKeyPattern(a);
|
|
||||||
const bIsPattern = isModelKeyPattern(b);
|
|
||||||
|
|
||||||
if (aIsPattern === bIsPattern) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (aIsPattern) {
|
|
||||||
return modelKeyMatches(a, b);
|
|
||||||
}
|
|
||||||
|
|
||||||
return modelKeyMatches(b, a);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Provider-layer policy: empty allowlist denies all. Blocklist only applies
|
* Provider-layer policy: empty allowlist denies all. Blocklist only applies
|
||||||
* after an allow match.
|
* after an allow match.
|
||||||
|
|||||||
@@ -0,0 +1,97 @@
|
|||||||
|
import {
|
||||||
|
aiModelCatalog,
|
||||||
|
getCatalogProviderForType,
|
||||||
|
type CatalogProvider
|
||||||
|
} from "@server/lib/aiModelCatalog";
|
||||||
|
import type { AiProviderType } from "@server/lib/aiProviderDefaults";
|
||||||
|
|
||||||
|
function stripVendorPrefix(modelId: string): string | null {
|
||||||
|
const idx = modelId.indexOf("/");
|
||||||
|
if (idx === -1 || idx === modelId.length - 1) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return modelId.slice(idx + 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
function modelKeysToTry(modelId: string): string[] {
|
||||||
|
const keys = [modelId];
|
||||||
|
const stripped = stripVendorPrefix(modelId);
|
||||||
|
if (stripped) {
|
||||||
|
keys.push(stripped);
|
||||||
|
}
|
||||||
|
return keys;
|
||||||
|
}
|
||||||
|
|
||||||
|
function catalogOwnsModel(
|
||||||
|
catalogProvider: CatalogProvider,
|
||||||
|
modelId: string
|
||||||
|
): boolean {
|
||||||
|
for (const key of modelKeysToTry(modelId)) {
|
||||||
|
if (aiModelCatalog.get(catalogProvider, key)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function modelKnownInAnyCatalog(modelId: string): boolean {
|
||||||
|
for (const key of modelKeysToTry(modelId)) {
|
||||||
|
if (aiModelCatalog.listByKey(key).length > 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How strongly a provider "owns" a requested model id via the known catalog.
|
||||||
|
*
|
||||||
|
* 2 - Typed provider whose catalog contains the model
|
||||||
|
* 1 - Aggregator/custom that can proxy a catalog-known model
|
||||||
|
* 0 - No ownership signal (typed miss, or unknown model on aggregator/custom)
|
||||||
|
*/
|
||||||
|
export function catalogOwnershipScore(
|
||||||
|
type: AiProviderType,
|
||||||
|
modelId: string
|
||||||
|
): number {
|
||||||
|
const catalogProvider = getCatalogProviderForType(type);
|
||||||
|
if (catalogProvider != null) {
|
||||||
|
return catalogOwnsModel(catalogProvider, modelId) ? 2 : 0;
|
||||||
|
}
|
||||||
|
return modelKnownInAnyCatalog(modelId) ? 1 : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Prefer native vendor providers over aggregators over custom when catalog
|
||||||
|
* ownership is tied.
|
||||||
|
*
|
||||||
|
* 2 - Native typed provider (openai, anthropic, gemini, ...)
|
||||||
|
* 1 - Aggregator gateway (openRouter, vercelAiGateway)
|
||||||
|
* 0 - Custom
|
||||||
|
*/
|
||||||
|
export function providerClassRank(type: AiProviderType): number {
|
||||||
|
if (type === "custom") {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
if (type === "openRouter" || type === "vercelAiGateway") {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function keepBestScored<T>(
|
||||||
|
items: T[],
|
||||||
|
scoreFn: (item: T) => number
|
||||||
|
): T[] {
|
||||||
|
if (items.length <= 1) {
|
||||||
|
return items;
|
||||||
|
}
|
||||||
|
let best = Number.NEGATIVE_INFINITY;
|
||||||
|
for (const item of items) {
|
||||||
|
const score = scoreFn(item);
|
||||||
|
if (score > best) {
|
||||||
|
best = score;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return items.filter((item) => scoreFn(item) === best);
|
||||||
|
}
|
||||||
@@ -50,6 +50,11 @@ import {
|
|||||||
isAllowedByLists,
|
isAllowedByLists,
|
||||||
mostSpecificMatchingAllow
|
mostSpecificMatchingAllow
|
||||||
} from "@server/lib/aiModelKeyMatch";
|
} from "@server/lib/aiModelKeyMatch";
|
||||||
|
import {
|
||||||
|
catalogOwnershipScore,
|
||||||
|
keepBestScored,
|
||||||
|
providerClassRank
|
||||||
|
} from "@server/lib/aiProviderSelection";
|
||||||
import { aiGatewayUpstreamFetch } from "@server/lib/aiGatewayUpstreamFetch";
|
import { aiGatewayUpstreamFetch } from "@server/lib/aiGatewayUpstreamFetch";
|
||||||
import { getModelPricing, calculateAiCost } from "@server/lib/aiModelPricing";
|
import { getModelPricing, calculateAiCost } from "@server/lib/aiModelPricing";
|
||||||
import {
|
import {
|
||||||
@@ -502,17 +507,28 @@ async function selectProvider(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 1) Prefer the most specific allow pattern that matched the request.
|
||||||
candidates.sort((a, b) =>
|
candidates.sort((a, b) =>
|
||||||
compareModelKeySpecificity(a.modelKey, b.modelKey)
|
compareModelKeySpecificity(a.modelKey, b.modelKey)
|
||||||
);
|
);
|
||||||
|
|
||||||
const bestSpecificity = candidates[0].modelKey;
|
const bestSpecificity = candidates[0].modelKey;
|
||||||
const topCandidates = candidates.filter(
|
let remaining = candidates.filter(
|
||||||
(c) => compareModelKeySpecificity(c.modelKey, bestSpecificity) === 0
|
(c) => compareModelKeySpecificity(c.modelKey, bestSpecificity) === 0
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// 2) Prefer providers whose catalog owns this model id. Aggregators only
|
||||||
|
// score when the model is known somewhere in the catalog.
|
||||||
|
remaining = keepBestScored(remaining, (c) =>
|
||||||
|
catalogOwnershipScore(c.provider.type as AiProviderType, requestedModel)
|
||||||
|
);
|
||||||
|
|
||||||
|
// 3) Prefer native typed providers over aggregators over custom.
|
||||||
|
remaining = keepBestScored(remaining, (c) =>
|
||||||
|
providerClassRank(c.provider.type as AiProviderType)
|
||||||
|
);
|
||||||
|
|
||||||
const uniqueProviders = new Map<number, AiProvider>();
|
const uniqueProviders = new Map<number, AiProvider>();
|
||||||
for (const candidate of topCandidates) {
|
for (const candidate of remaining) {
|
||||||
uniqueProviders.set(candidate.provider.providerId, candidate.provider);
|
uniqueProviders.set(candidate.provider.providerId, candidate.provider);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -523,7 +539,7 @@ async function selectProvider(
|
|||||||
return {
|
return {
|
||||||
ok: false,
|
ok: false,
|
||||||
status: HttpCode.FORBIDDEN,
|
status: HttpCode.FORBIDDEN,
|
||||||
message: `Model "${requestedModel}" is ambiguous across multiple AI providers on this resource`
|
message: `Model "${requestedModel}" is ambiguous across multiple AI providers on this resource. Ask your administrator to configure a more specific allow pattern for this model.`
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -131,8 +131,7 @@ export async function addAiProviderToResource(
|
|||||||
const attachments = await resolveProviderAttachments({
|
const attachments = await resolveProviderAttachments({
|
||||||
orgId: resource.orgId,
|
orgId: resource.orgId,
|
||||||
attachments: nextAttachments,
|
attachments: nextAttachments,
|
||||||
requireAtLeastOne: true,
|
requireAtLeastOne: true
|
||||||
resourceId
|
|
||||||
});
|
});
|
||||||
if (isInferenceFieldsError(attachments)) {
|
if (isInferenceFieldsError(attachments)) {
|
||||||
return next(
|
return next(
|
||||||
|
|||||||
@@ -134,8 +134,7 @@ export async function removeAiProviderFromResource(
|
|||||||
const attachments = await resolveProviderAttachments({
|
const attachments = await resolveProviderAttachments({
|
||||||
orgId: resource.orgId,
|
orgId: resource.orgId,
|
||||||
attachments: remaining,
|
attachments: remaining,
|
||||||
requireAtLeastOne: false,
|
requireAtLeastOne: false
|
||||||
resourceId
|
|
||||||
});
|
});
|
||||||
if (isInferenceFieldsError(attachments)) {
|
if (isInferenceFieldsError(attachments)) {
|
||||||
return next(
|
return next(
|
||||||
|
|||||||
@@ -113,8 +113,7 @@ export async function setResourceAiProviders(
|
|||||||
const attachments = await resolveProviderAttachments({
|
const attachments = await resolveProviderAttachments({
|
||||||
orgId: resource.orgId,
|
orgId: resource.orgId,
|
||||||
attachments: providers,
|
attachments: providers,
|
||||||
requireAtLeastOne: false,
|
requireAtLeastOne: false
|
||||||
resourceId
|
|
||||||
});
|
});
|
||||||
if (isInferenceFieldsError(attachments)) {
|
if (isInferenceFieldsError(attachments)) {
|
||||||
return next(
|
return next(
|
||||||
|
|||||||
@@ -131,8 +131,7 @@ export async function addAiProviderToSiteResource(
|
|||||||
const attachments = await resolveProviderAttachments({
|
const attachments = await resolveProviderAttachments({
|
||||||
orgId: siteResource.orgId,
|
orgId: siteResource.orgId,
|
||||||
attachments: nextAttachments,
|
attachments: nextAttachments,
|
||||||
requireAtLeastOne: true,
|
requireAtLeastOne: true
|
||||||
siteResourceId
|
|
||||||
});
|
});
|
||||||
if (isInferenceFieldsError(attachments)) {
|
if (isInferenceFieldsError(attachments)) {
|
||||||
return next(
|
return next(
|
||||||
|
|||||||
@@ -133,8 +133,7 @@ export async function removeAiProviderFromSiteResource(
|
|||||||
const attachments = await resolveProviderAttachments({
|
const attachments = await resolveProviderAttachments({
|
||||||
orgId: siteResource.orgId,
|
orgId: siteResource.orgId,
|
||||||
attachments: remaining,
|
attachments: remaining,
|
||||||
requireAtLeastOne: false,
|
requireAtLeastOne: false
|
||||||
siteResourceId
|
|
||||||
});
|
});
|
||||||
if (isInferenceFieldsError(attachments)) {
|
if (isInferenceFieldsError(attachments)) {
|
||||||
return next(
|
return next(
|
||||||
|
|||||||
@@ -115,8 +115,7 @@ export async function setSiteResourceAiProviders(
|
|||||||
const attachments = await resolveProviderAttachments({
|
const attachments = await resolveProviderAttachments({
|
||||||
orgId: siteResource.orgId,
|
orgId: siteResource.orgId,
|
||||||
attachments: providers,
|
attachments: providers,
|
||||||
requireAtLeastOne: false,
|
requireAtLeastOne: false
|
||||||
siteResourceId
|
|
||||||
});
|
});
|
||||||
if (isInferenceFieldsError(attachments)) {
|
if (isInferenceFieldsError(attachments)) {
|
||||||
return next(
|
return next(
|
||||||
|
|||||||
Reference in New Issue
Block a user