# TODO

- Race condition on jwks token refresh
- Rate Limiting

git tag -d v1.0.0; git push origin :refs/tags/v1.0.0; git tag -a v1.0.0 -m "Release v1.0.0"; git push origin v1.0.0

curl https://chat.iceberg.black/api/v1/models -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJublpLek04TkZHVmpWbGFPRXZpMUtFSTVHQWRwaGlsYjh3RHRLeG5JOENZIn0.eyJleHAiOjE3NzU4MTUyODksImlhdCI6MTc3NTgxNDk4OSwianRpIjoiMTgwYzA2NDUtYzZiMC00MDRmLTgyYjEtMzA3YmY4ZmJlNmJlIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmljZWJlcmcuYmxhY2svcmVhbG1zL2ljZWJlcmciLCJzdWIiOiJmZGRiN2FjZC1kMmE5LTRmMTctOWIxNi1kZjVlN2EzNDI4YjciLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjaGF0LWFwaSIsInNjb3BlIjoiIiwiY2xpZW50SG9zdCI6Ijg2LjIxMi44NC4xOTEiLCJjbGllbnRBZGRyZXNzIjoiODYuMjEyLjg0LjE5MSIsImNsaWVudF9pZCI6ImNoYXQtYXBpIn0.abjHABcjCJNiB6vriRw60nfzabEfD7CXwyRhkahFkC8ATgfy4fn0T8PnFfsaRpsXuhamIhWwNskrA7L9V3mbWWKW-JEOvImDhTc8sIX0E5fDTbk8O5wa_2yzNdLpRxdSjLqgL544rB8I-LZ8bl5SxtdN3gHfrnWr5ef8bbLgPRzZIylT3QUpah0uywDM_cfrrve9SMHHOUUItyzmOHLw0Igit1EzyFNyjbWf6OAU6TMjOF_eFTc5sakyBwsdJnGy0nhj5R-wxLpr1ug3iEd3Y-jDzHO4m6jariXVJ7Vvbz71i7sadDEKdSyOKcCeQbU4T0Tf7IxqW4c2DNnk3BFFuw"

curl -X POST "https://auth.iceberg.black/realms/iceberg/protocol/openid-connect/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials" -d "client_id=chat-api" -d "client_secret=5fHUp8Z5GoNM70MVOGuQTfKFkaAE48Za"

curl -s -X POST https://chat.iceberg.black/api/v1/chat/completions \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJublpLek04TkZHVmpWbGFPRXZpMUtFSTVHQWRwaGlsYjh3RHRLeG5JOENZIn0.eyJleHAiOjE3NzU4MTU0NTEsImlhdCI6MTc3NTgxNTE1MSwianRpIjoiOGNjMmM5MjUtZmMzNy00MDFhLWE1ZjUtODFlOGJhNjcxNzcwIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmljZWJlcmcuYmxhY2svcmVhbG1zL2ljZWJlcmciLCJzdWIiOiJmZGRiN2FjZC1kMmE5LTRmMTctOWIxNi1kZjVlN2EzNDI4YjciLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjaGF0LWFwaSIsInNjb3BlIjoiIiwiY2xpZW50SG9zdCI6Ijg2LjIxMi44NC4xOTEiLCJjbGllbnRBZGRyZXNzIjoiODYuMjEyLjg0LjE5MSIsImNsaWVudF9pZCI6ImNoYXQtYXBpIn0.Exv34aoAqwzSqQziZH3zScAnK_xiNCXqpOQl54x7NFMdO0gVgacJgMxoW82Ym8aCNfBRMFtWclZJ9RFh1b9uSEUgsdVtqvMX-2kCAhiMSjmIBPU-L0gr5N63c3bScOoAYa37baR4mXQ5LxHjfkLo_rHDJ74adg2JMo359Wbuu1_OR708_q8yjgO_4fQeYbIffxADwRDvPSIwQ8Y2qRjaAIOZs0xmA9p128CUxlUyvhxwfFquYKRaDs5QE9pIAtvm_KuoBydYopm8j8cbm4ixDhUwYjkzniIIapY77NxpmMosfh8BhK0W3ieK2gTKfmiFDhYgkGybU6b1BnJMy1_Kxg" \
 -d '{
"model": "llama3:latest",
"messages": [
{"role": "user", "content": "What is Rust?"}
]
}' | jq .

curl -X POST http://localhost:3001/v1/completions \
  -H "Content-Type: application/json" \
  -H "x-api-key: UXVqi1Jazl_-A0TuBudw2Y3PeUiNwCMYayXwBWwuMf0" \
  -H "Accept: text/event-stream" \
  -d '{"model": "llama3:latest", "prompt": "hello", "stream": true}'


providers → LlmError ┐
                     ├── services → ServiceError → api → HTTP
databases → DbError  ┘

# 🦙 Ollama Rust API Wrapper

A high-performance Rust API wrapper around Ollama, providing an OpenAI-compatible interface, model lifecycle management, and advanced runtime features.

---

# 🚀 Features

- ✅ OpenAI-compatible API (`/v1/...`)
- ⚡ Streaming (Server-Sent Events)
- 🧠 Model lifecycle management (load/unload)
- 🔐 API key authentication (optional)
- 📊 Usage tracking & observability
- 🔀 Model routing & abstraction
- 🧩 Extensible architecture (multi-provider ready)

---

# 📡 API Endpoints

## 1. Core LLM API (OpenAI-compatible)

- [x] `POST /v1/chat/completions` + streaming
- [x] `POST /v1/completions` + streaming
- [ ] `POST /v1/embeddings`
- [x] `GET /v1/models`

## 2. Model Lifecycle Management

- [x] `POST /v1/models/{model}/load`
- [x] `POST /v1/models/{model}/unload`

## 3. Model Management

- [ ] `POST /v1/models/pull`
- [ ] `DELETE /v1/models/{model}`

## 4. Runtime & Observability

### Model Status

```
GET /v1/models/{model}/status
```

### List Loaded Models

```
GET /v1/runtime/models
```

---

## 6. Health Checks

```
GET /health
GET /ready
```

---

# 🧠 Internal Mapping (Ollama)

| Wrapper Endpoint          | Ollama Endpoint |
| ------------------------- | --------------- |
| /v1/chat/completions      | /api/chat       |
| /v1/completions           | /api/generate   |
| /v1/embeddings            | /api/embeddings |
| /v1/models                | /api/tags       |
| /v1/models/pull           | /api/pull       |
| DELETE /v1/models/{model} | /api/delete     |
| load/unload               | /api/generate   |

---

# 🔧 Advanced Features

## 🔀 Model Routing

Use abstract model names:

```json
{
  "model": "fast"
}
```

Example mapping:

```
fast   → llama3:8b
smart  → llama3:70b
code   → deepseek-coder
```

---

## 📊 Usage Tracking

```
GET /v1/usage
```

Tracks:

- request count
- latency
- per-model usage

---

## 🚦 Rate Limiting

- Requests per minute
- Tokens per minute

Returns:

```
429 Too Many Requests
```

---

## 🧠 Sessions (Context Management)

```
POST /v1/sessions
POST /v1/sessions/{id}/chat
```

Stores conversation history server-side.

---

## ⚡ Caching

- Embeddings
- Deterministic prompts (temperature = 0)

---

## 🧩 Tool / Function Calling

Supports structured tool execution:

```json
{
  "tools": [
    {
      "name": "function_name",
      "parameters": {}
    }
  ]
}
```

---

## 📦 Batch Requests

```
POST /v1/batch
```

---

## 🧠 Auto Eviction

```
POST /v1/runtime/evict
```

Strategies:

- LRU
- memory threshold

---

## 🧾 Logs

```
GET /v1/logs
```

## 🔔 Async Jobs / Webhooks

```
POST /v1/jobs
```

---

# 🏗️ Architecture

```
Client → Rust API → Ollama → Response
```

### Layers:

- HTTP (Axum)
- Service layer (business logic)
- Provider abstraction
- Ollama client

---

# 🔌 Provider Abstraction (Future-Proof)

```rust
trait LlmProvider {
    async fn chat(...);
    async fn embeddings(...);
}
```

Supports:

- Ollama (current)
- OpenAI (future)
- Others

---

# 🎯 Roadmap

- [ ] Full OpenAI compatibility
- [ ] Multi-node routing
- [ ] GPU-aware scheduling
- [ ] Web UI dashboard
- [ ] Distributed inference

---

# 🧠 Summary

This project turns Ollama into:

👉 A local OpenAI-compatible API
👉 A controllable model runtime
👉 A foundation for a full LLM gateway

# TODO
- open api doc for bearer token
- Unify check before sending to ollama payload
- load/unload model functions