Elgato_dark/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI synced 2026-05-24 09:28:23 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8
LocalAI/backend/python/vllm
History
Richard Palethorpe c60ed75258 feat(middleware): Model routing, PII filtering, Cloud model proxies 
Add a routing middleware stack and a cloud-proxy backend.

* cloud-proxy: a Go gRPC backend that forwards OpenAI- and
  Anthropic-shaped chat requests to upstream providers, with an
  optional translate mode (OpenAI request -> Anthropic /v1/messages
  -> OpenAI response) and full tool-calling support.

* routing: admission control, content-aware model routing
  (embedding cache + classifier + rerank + Arch-Router score),
  PII detection/redaction (regex + NER) with streaming filter and
  OpenAI/Anthropic adapters, and a per-user/per-key billing recorder
  backed by GORM or in-memory storage.

* middleware: UsageMiddleware records usage via the billing recorder,
  plus admission, route-model, usage-stamp and trace middlewares.

* observability: BackendTrace ring buffer stores full request bodies
  (capped), MITM proxy emits structured trace events, and router
  classifier decisions surface at /api/router/decide.

* gallery: Arch-Router-1.5B (Q4_K_M and Q8_0).

* UI: cloud-proxy model-editor fields, classifier system-prompt and
  score-normalization config, and a Traces page rendering request
  bodies.

Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
2026-05-24 09:42:31 +01:00
..
backend.py feat(middleware): Model routing, PII filtering, Cloud model proxies 2026-05-24 09:42:31 +01:00
install.sh fix(L4T13 backends): switch vllm/sglang/vllm-omni to PyPI aarch64+cu130 wheels (#9950) 2026-05-22 23:01:22 +02:00
Makefile feat(mlx): add mlx backend (#6049) 2025-08-22 08:42:29 +02:00
package.sh feat(vllm, distributed): tensor parallel distributed workers (#9612) 2026-05-06 00:22:50 +02:00
README.md refactor: move backends into the backends directory (#1279) 2023-11-13 22:40:16 +01:00
requirements-after.txt feat(vllm): parity with llama.cpp backend (#9328) 2026-04-13 11:00:29 +02:00
requirements-cpu-after.txt feat(vllm): parity with llama.cpp backend (#9328) 2026-04-13 11:00:29 +02:00
requirements-cpu.txt feat(vllm): parity with llama.cpp backend (#9328) 2026-04-13 11:00:29 +02:00
requirements-cublas12-after.txt fix(vllm): drop flash-attn wheel to avoid torch 2.10 ABI mismatch (#9557) 2026-04-25 15:38:13 +00:00
requirements-cublas12.txt fix(vllm): drop flash-attn wheel to avoid torch 2.10 ABI mismatch (#9557) 2026-04-25 15:38:13 +00:00
requirements-cublas13-after.txt chore: ⬆️ Update vllm-project/vllm cu130 wheel to 0.21.0 (#9846) 2026-05-15 23:45:41 +02:00
requirements-cublas13.txt feat(backends): add CUDA 13 + L4T arm64 CUDA 13 variants for vllm/vllm-omni/sglang (#9553) 2026-04-25 12:26:29 +02:00
requirements-hipblas-after.txt feat(vllm): parity with llama.cpp backend (#9328) 2026-04-13 11:00:29 +02:00
requirements-hipblas.txt feat(rocm): bump to 7.x (#9323) 2026-04-12 08:51:30 +02:00
requirements-install.txt fix(vllm): seed pybind11 for fastsafetensors build under --no-build-isolation 2026-04-28 20:08:26 +00:00
requirements-intel-after.txt feat(vllm, distributed): tensor parallel distributed workers (#9612) 2026-05-06 00:22:50 +02:00
requirements-intel.txt feat(vllm, distributed): tensor parallel distributed workers (#9612) 2026-05-06 00:22:50 +02:00
requirements-l4t13-after.txt fix(L4T13 backends): switch vllm/sglang/vllm-omni to PyPI aarch64+cu130 wheels (#9950) 2026-05-22 23:01:22 +02:00
requirements-l4t13.txt fix(L4T13 backends): switch vllm/sglang/vllm-omni to PyPI aarch64+cu130 wheels (#9950) 2026-05-22 23:01:22 +02:00
requirements.txt chore(deps): update charset-normalizer requirement from >=3.4.0 to >=3.4.7 in /backend/python/vllm (#9779) 2026-05-12 09:22:23 +02:00
run.sh fix(python-backend): make JIT subprocesses work on hosts of any size (#9679) 2026-05-06 00:28:01 +02:00
test.py feat(vllm): expose AsyncEngineArgs via generic engine_args YAML map (#9563) 2026-04-29 00:49:28 +02:00
test.sh feat: Add backend gallery (#5607) 2025-06-15 14:56:52 +02:00

README.md

Creating a separate environment for the vllm project

make vllm