285 lines
6.2 KiB
Markdown
285 lines
6.2 KiB
Markdown
# TODO
|
|
|
|
- Race condition on jwks token refresh
|
|
- Rate Limiting
|
|
|
|
git tag -d v1.0.0; git push origin :refs/tags/v1.0.0; git tag -a v1.0.0 -m "Release v1.0.0"; git push origin v1.0.0
|
|
|
|
curl https://chat.iceberg.black/api/v1/models -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJublpLek04TkZHVmpWbGFPRXZpMUtFSTVHQWRwaGlsYjh3RHRLeG5JOENZIn0.eyJleHAiOjE3NzU4MTUyODksImlhdCI6MTc3NTgxNDk4OSwianRpIjoiMTgwYzA2NDUtYzZiMC00MDRmLTgyYjEtMzA3YmY4ZmJlNmJlIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmljZWJlcmcuYmxhY2svcmVhbG1zL2ljZWJlcmciLCJzdWIiOiJmZGRiN2FjZC1kMmE5LTRmMTctOWIxNi1kZjVlN2EzNDI4YjciLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjaGF0LWFwaSIsInNjb3BlIjoiIiwiY2xpZW50SG9zdCI6Ijg2LjIxMi44NC4xOTEiLCJjbGllbnRBZGRyZXNzIjoiODYuMjEyLjg0LjE5MSIsImNsaWVudF9pZCI6ImNoYXQtYXBpIn0.abjHABcjCJNiB6vriRw60nfzabEfD7CXwyRhkahFkC8ATgfy4fn0T8PnFfsaRpsXuhamIhWwNskrA7L9V3mbWWKW-JEOvImDhTc8sIX0E5fDTbk8O5wa_2yzNdLpRxdSjLqgL544rB8I-LZ8bl5SxtdN3gHfrnWr5ef8bbLgPRzZIylT3QUpah0uywDM_cfrrve9SMHHOUUItyzmOHLw0Igit1EzyFNyjbWf6OAU6TMjOF_eFTc5sakyBwsdJnGy0nhj5R-wxLpr1ug3iEd3Y-jDzHO4m6jariXVJ7Vvbz71i7sadDEKdSyOKcCeQbU4T0Tf7IxqW4c2DNnk3BFFuw"
|
|
|
|
curl -X POST "https://auth.iceberg.black/realms/iceberg/protocol/openid-connect/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials" -d "client_id=chat-api" -d "client_secret=5fHUp8Z5GoNM70MVOGuQTfKFkaAE48Za"
|
|
|
|
curl -s -X POST https://chat.iceberg.black/api/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJublpLek04TkZHVmpWbGFPRXZpMUtFSTVHQWRwaGlsYjh3RHRLeG5JOENZIn0.eyJleHAiOjE3NzU4MTU0NTEsImlhdCI6MTc3NTgxNTE1MSwianRpIjoiOGNjMmM5MjUtZmMzNy00MDFhLWE1ZjUtODFlOGJhNjcxNzcwIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmljZWJlcmcuYmxhY2svcmVhbG1zL2ljZWJlcmciLCJzdWIiOiJmZGRiN2FjZC1kMmE5LTRmMTctOWIxNi1kZjVlN2EzNDI4YjciLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjaGF0LWFwaSIsInNjb3BlIjoiIiwiY2xpZW50SG9zdCI6Ijg2LjIxMi44NC4xOTEiLCJjbGllbnRBZGRyZXNzIjoiODYuMjEyLjg0LjE5MSIsImNsaWVudF9pZCI6ImNoYXQtYXBpIn0.Exv34aoAqwzSqQziZH3zScAnK_xiNCXqpOQl54x7NFMdO0gVgacJgMxoW82Ym8aCNfBRMFtWclZJ9RFh1b9uSEUgsdVtqvMX-2kCAhiMSjmIBPU-L0gr5N63c3bScOoAYa37baR4mXQ5LxHjfkLo_rHDJ74adg2JMo359Wbuu1_OR708_q8yjgO_4fQeYbIffxADwRDvPSIwQ8Y2qRjaAIOZs0xmA9p128CUxlUyvhxwfFquYKRaDs5QE9pIAtvm_KuoBydYopm8j8cbm4ixDhUwYjkzniIIapY77NxpmMosfh8BhK0W3ieK2gTKfmiFDhYgkGybU6b1BnJMy1_Kxg" \
|
|
-d '{
|
|
"model": "llama3:latest",
|
|
"messages": [
|
|
{"role": "user", "content": "What is Rust?"}
|
|
]
|
|
}' | jq .
|
|
|
|
curl -X POST http://localhost:3001/v1/completions \
|
|
-H "Content-Type: application/json" \
|
|
-H "x-api-key: UXVqi1Jazl_-A0TuBudw2Y3PeUiNwCMYayXwBWwuMf0" \
|
|
-H "Accept: text/event-stream" \
|
|
-d '{"model": "llama3:latest", "prompt": "hello", "stream": true}'
|
|
|
|
|
|
providers → LlmError ┐
|
|
├── services → ServiceError → api → HTTP
|
|
databases → DbError ┘
|
|
|
|
# 🦙 Ollama Rust API Wrapper
|
|
|
|
A high-performance Rust API wrapper around Ollama, providing an OpenAI-compatible interface, model lifecycle management, and advanced runtime features.
|
|
|
|
---
|
|
|
|
# 🚀 Features
|
|
|
|
- ✅ OpenAI-compatible API (`/v1/...`)
|
|
- ⚡ Streaming (Server-Sent Events)
|
|
- 🧠 Model lifecycle management (load/unload)
|
|
- 🔐 API key authentication (optional)
|
|
- 📊 Usage tracking & observability
|
|
- 🔀 Model routing & abstraction
|
|
- 🧩 Extensible architecture (multi-provider ready)
|
|
|
|
---
|
|
|
|
# 📡 API Endpoints
|
|
|
|
## 1. Core LLM API (OpenAI-compatible)
|
|
|
|
- [x] `POST /v1/chat/completions` + streaming
|
|
- [x] `POST /v1/completions` + streaming
|
|
- [ ] `POST /v1/embeddings`
|
|
- [x] `GET /v1/models`
|
|
|
|
## 2. Model Lifecycle Management
|
|
|
|
- [x] `POST /v1/models/{model}/load`
|
|
- [x] `POST /v1/models/{model}/unload`
|
|
|
|
## 3. Model Management
|
|
|
|
- [ ] `POST /v1/models/pull`
|
|
- [ ] `DELETE /v1/models/{model}`
|
|
|
|
## 4. Runtime & Observability
|
|
|
|
### Model Status
|
|
|
|
```
|
|
GET /v1/models/{model}/status
|
|
```
|
|
|
|
### List Loaded Models
|
|
|
|
```
|
|
GET /v1/runtime/models
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Health Checks
|
|
|
|
```
|
|
GET /health
|
|
GET /ready
|
|
```
|
|
|
|
---
|
|
|
|
# 🧠 Internal Mapping (Ollama)
|
|
|
|
| Wrapper Endpoint | Ollama Endpoint |
|
|
| ------------------------- | --------------- |
|
|
| /v1/chat/completions | /api/chat |
|
|
| /v1/completions | /api/generate |
|
|
| /v1/embeddings | /api/embeddings |
|
|
| /v1/models | /api/tags |
|
|
| /v1/models/pull | /api/pull |
|
|
| DELETE /v1/models/{model} | /api/delete |
|
|
| load/unload | /api/generate |
|
|
|
|
---
|
|
|
|
# 🔧 Advanced Features
|
|
|
|
## 🔀 Model Routing
|
|
|
|
Use abstract model names:
|
|
|
|
```json
|
|
{
|
|
"model": "fast"
|
|
}
|
|
```
|
|
|
|
Example mapping:
|
|
|
|
```
|
|
fast → llama3:8b
|
|
smart → llama3:70b
|
|
code → deepseek-coder
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Usage Tracking
|
|
|
|
```
|
|
GET /v1/usage
|
|
```
|
|
|
|
Tracks:
|
|
|
|
- request count
|
|
- latency
|
|
- per-model usage
|
|
|
|
---
|
|
|
|
## 🚦 Rate Limiting
|
|
|
|
- Requests per minute
|
|
- Tokens per minute
|
|
|
|
Returns:
|
|
|
|
```
|
|
429 Too Many Requests
|
|
```
|
|
|
|
---
|
|
|
|
## 🧠 Sessions (Context Management)
|
|
|
|
```
|
|
POST /v1/sessions
|
|
POST /v1/sessions/{id}/chat
|
|
```
|
|
|
|
Stores conversation history server-side.
|
|
|
|
---
|
|
|
|
## ⚡ Caching
|
|
|
|
- Embeddings
|
|
- Deterministic prompts (temperature = 0)
|
|
|
|
---
|
|
|
|
## 🧩 Tool / Function Calling
|
|
|
|
Supports structured tool execution:
|
|
|
|
```json
|
|
{
|
|
"tools": [
|
|
{
|
|
"name": "function_name",
|
|
"parameters": {}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 📦 Batch Requests
|
|
|
|
```
|
|
POST /v1/batch
|
|
```
|
|
|
|
---
|
|
|
|
## 🧠 Auto Eviction
|
|
|
|
```
|
|
POST /v1/runtime/evict
|
|
```
|
|
|
|
Strategies:
|
|
|
|
- LRU
|
|
- memory threshold
|
|
|
|
---
|
|
|
|
## 🧾 Logs
|
|
|
|
```
|
|
GET /v1/logs
|
|
```
|
|
|
|
## 🔔 Async Jobs / Webhooks
|
|
|
|
```
|
|
POST /v1/jobs
|
|
```
|
|
|
|
---
|
|
|
|
# 🏗️ Architecture
|
|
|
|
```
|
|
Client → Rust API → Ollama → Response
|
|
```
|
|
|
|
### Layers:
|
|
|
|
- HTTP (Axum)
|
|
- Service layer (business logic)
|
|
- Provider abstraction
|
|
- Ollama client
|
|
|
|
---
|
|
|
|
# 🔌 Provider Abstraction (Future-Proof)
|
|
|
|
```rust
|
|
trait LlmProvider {
|
|
async fn chat(...);
|
|
async fn embeddings(...);
|
|
}
|
|
```
|
|
|
|
Supports:
|
|
|
|
- Ollama (current)
|
|
- OpenAI (future)
|
|
- Others
|
|
|
|
---
|
|
|
|
# 🎯 Roadmap
|
|
|
|
- [ ] Full OpenAI compatibility
|
|
- [ ] Multi-node routing
|
|
- [ ] GPU-aware scheduling
|
|
- [ ] Web UI dashboard
|
|
- [ ] Distributed inference
|
|
|
|
---
|
|
|
|
# 🧠 Summary
|
|
|
|
This project turns Ollama into:
|
|
|
|
👉 A local OpenAI-compatible API
|
|
👉 A controllable model runtime
|
|
👉 A foundation for a full LLM gateway
|
|
|
|
# Bugs
|
|
## Ollama
|
|
- When model answer onoly with 1 tokens, answer is empty
|
|
|
|
# TODO
|
|
- open api doc for bearer token
|
|
- Unify check before sending to ollama payload
|
|
- load/unload model functions |