TODO

• Race condition on jwks token refresh
• Rate Limiting

git tag -d v1.0.0; git push origin :refs/tags/v1.0.0; git tag -a v1.0.0 -m "Release v1.0.0"; git push origin v1.0.0

curl https://chat.iceberg.black/api/v1/models -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJublpLek04TkZHVmpWbGFPRXZpMUtFSTVHQWRwaGlsYjh3RHRLeG5JOENZIn0.eyJleHAiOjE3NzU4MTUyODksImlhdCI6MTc3NTgxNDk4OSwianRpIjoiMTgwYzA2NDUtYzZiMC00MDRmLTgyYjEtMzA3YmY4ZmJlNmJlIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmljZWJlcmcuYmxhY2svcmVhbG1zL2ljZWJlcmciLCJzdWIiOiJmZGRiN2FjZC1kMmE5LTRmMTctOWIxNi1kZjVlN2EzNDI4YjciLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjaGF0LWFwaSIsInNjb3BlIjoiIiwiY2xpZW50SG9zdCI6Ijg2LjIxMi44NC4xOTEiLCJjbGllbnRBZGRyZXNzIjoiODYuMjEyLjg0LjE5MSIsImNsaWVudF9pZCI6ImNoYXQtYXBpIn0.abjHABcjCJNiB6vriRw60nfzabEfD7CXwyRhkahFkC8ATgfy4fn0T8PnFfsaRpsXuhamIhWwNskrA7L9V3mbWWKW-JEOvImDhTc8sIX0E5fDTbk8O5wa_2yzNdLpRxdSjLqgL544rB8I-LZ8bl5SxtdN3gHfrnWr5ef8bbLgPRzZIylT3QUpah0uywDM_cfrrve9SMHHOUUItyzmOHLw0Igit1EzyFNyjbWf6OAU6TMjOF_eFTc5sakyBwsdJnGy0nhj5R-wxLpr1ug3iEd3Y-jDzHO4m6jariXVJ7Vvbz71i7sadDEKdSyOKcCeQbU4T0Tf7IxqW4c2DNnk3BFFuw"

curl -X POST "https://auth.iceberg.black/realms/iceberg/protocol/openid-connect/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials" -d "client_id=chat-api" -d "client_secret=5fHUp8Z5GoNM70MVOGuQTfKFkaAE48Za"

curl -s -X POST https://chat.iceberg.black/api/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJublpLek04TkZHVmpWbGFPRXZpMUtFSTVHQWRwaGlsYjh3RHRLeG5JOENZIn0.eyJleHAiOjE3NzU4MTU0NTEsImlhdCI6MTc3NTgxNTE1MSwianRpIjoiOGNjMmM5MjUtZmMzNy00MDFhLWE1ZjUtODFlOGJhNjcxNzcwIiwiaXNzIjoiaHR0cHM6Ly9hdXRoLmljZWJlcmcuYmxhY2svcmVhbG1zL2ljZWJlcmciLCJzdWIiOiJmZGRiN2FjZC1kMmE5LTRmMTctOWIxNi1kZjVlN2EzNDI4YjciLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjaGF0LWFwaSIsInNjb3BlIjoiIiwiY2xpZW50SG9zdCI6Ijg2LjIxMi44NC4xOTEiLCJjbGllbnRBZGRyZXNzIjoiODYuMjEyLjg0LjE5MSIsImNsaWVudF9pZCI6ImNoYXQtYXBpIn0.Exv34aoAqwzSqQziZH3zScAnK_xiNCXqpOQl54x7NFMdO0gVgacJgMxoW82Ym8aCNfBRMFtWclZJ9RFh1b9uSEUgsdVtqvMX-2kCAhiMSjmIBPU-L0gr5N63c3bScOoAYa37baR4mXQ5LxHjfkLo_rHDJ74adg2JMo359Wbuu1_OR708_q8yjgO_4fQeYbIffxADwRDvPSIwQ8Y2qRjaAIOZs0xmA9p128CUxlUyvhxwfFquYKRaDs5QE9pIAtvm_KuoBydYopm8j8cbm4ixDhUwYjkzniIIapY77NxpmMosfh8BhK0W3ieK2gTKfmiFDhYgkGybU6b1BnJMy1_Kxg"
-d '{ "model": "llama3:latest", "messages": [ {"role": "user", "content": "What is Rust?"} ] }' | jq .

curl -X POST http://localhost:3001/v1/completions
-H "Content-Type: application/json"
-H "x-api-key: UXVqi1Jazl_-A0TuBudw2Y3PeUiNwCMYayXwBWwuMf0"
-H "Accept: text/event-stream"
-d '{"model": "llama3:latest", "prompt": "hello", "stream": true}'

providers → LlmError ┐ ├── services → ServiceError → api → HTTP databases → DbError ┘

🦙 Ollama Rust API Wrapper

A high-performance Rust API wrapper around Ollama, providing an OpenAI-compatible interface, model lifecycle management, and advanced runtime features.

---

🚀 Features

• ✅ OpenAI-compatible API (/v1/...)
• ⚡ Streaming (Server-Sent Events)
• 🧠 Model lifecycle management (load/unload)
• 🔐 API key authentication (optional)
• 📊 Usage tracking & observability
• 🔀 Model routing & abstraction
• 🧩 Extensible architecture (multi-provider ready)

---

📡 API Endpoints

1. Core LLM API (OpenAI-compatible)

• POST /v1/chat/completions + streaming
• POST /v1/completions + streaming
• POST /v1/embeddings
• GET /v1/models

2. Model Lifecycle Management

• POST /v1/models/{model}/load
• POST /v1/models/{model}/unload

3. Model Management

• POST /v1/models/pull
• DELETE /v1/models/{model}

4. Runtime & Observability

Model Status

GET /v1/models/{model}/status

List Loaded Models

GET /v1/runtime/models

---

6. Health Checks

GET /health
GET /ready

---

🧠 Internal Mapping (Ollama)

Wrapper Endpoint	Ollama Endpoint
/v1/chat/completions	/api/chat
/v1/completions	/api/generate
/v1/embeddings	/api/embeddings
/v1/models	/api/tags
/v1/models/pull	/api/pull
DELETE /v1/models/{model}	/api/delete
load/unload	/api/generate

---

🔧 Advanced Features

🔀 Model Routing

Use abstract model names:

{
  "model": "fast"
}

Example mapping:

fast   → llama3:8b
smart  → llama3:70b
code   → deepseek-coder

---

📊 Usage Tracking

GET /v1/usage

Tracks:

• request count
• latency
• per-model usage

---

🚦 Rate Limiting

• Requests per minute
• Tokens per minute

Returns:

429 Too Many Requests

---

🧠 Sessions (Context Management)

POST /v1/sessions
POST /v1/sessions/{id}/chat

Stores conversation history server-side.

---

⚡ Caching

• Embeddings
• Deterministic prompts (temperature = 0)

---

🧩 Tool / Function Calling

Supports structured tool execution:

{
  "tools": [
    {
      "name": "function_name",
      "parameters": {}
    }
  ]
}

---

📦 Batch Requests

POST /v1/batch

---

🧠 Auto Eviction

POST /v1/runtime/evict

Strategies:

• LRU
• memory threshold

---

🧾 Logs

GET /v1/logs

🔔 Async Jobs / Webhooks

POST /v1/jobs

---

🏗️ Architecture

Client → Rust API → Ollama → Response

Layers:

• HTTP (Axum)
• Service layer (business logic)
• Provider abstraction
• Ollama client

---

🔌 Provider Abstraction (Future-Proof)

trait LlmProvider {
    async fn chat(...);
    async fn embeddings(...);
}

Supports:

• Ollama (current)
• OpenAI (future)
• Others

---

🎯 Roadmap

• Full OpenAI compatibility
• Multi-node routing
• GPU-aware scheduling
• Web UI dashboard
• Distributed inference

---

🧠 Summary

This project turns Ollama into:

👉 A local OpenAI-compatible API 👉 A controllable model runtime 👉 A foundation for a full LLM gateway

TODO

• open api doc for bearer token
• Unify check before sending to ollama payload
• load/unload model functions