# StudioLLM Benchmark Report

**Date:** 2026-07-04 10:10 UTC  
**Endpoint:** `https://inference.studio.dripper.live/api/ollama/v1`

---

## 1. Model Availability

| # | Model | Size | Type | Warmup | Notes |
|---|-------|------|------|--------|-------|
| 1 | ornith:35b | 35B MoE | Text (reasoning) | ✅ 22.2s | Custom fine-tune |
| 2 | ornith:9b | 9B | Text (reasoning) | ✅ 8.8s | Custom fine-tune |
| 3 | llama3.2:3b | 3B | Text | ✅ 14.5s | |
| 4 | moondream:latest | ~2B | Vision | ✅ 12.8s | |
| 5 | llama3.2-vision:11b | 11B | Vision | ❌ | Server crashed on load |
| 6 | llava:7b | 7B | Vision | ✅ 37.2s | Also good at text |
| 7 | minicpm-v:latest | ~2B | Vision | ✅ 28.6s | |
| 8 | deepseek-r1:14b | 14B | Text (reasoning) | ✅ 11.0s | |
| 9 | qwen2.5:32b | 32B | Text | ✅ 23.2s | |
| 10 | deepseek-coder-v2:latest | ~16B MoE | Text (code) | ✅ 49.0s | |
| 11 | deepseek-r1:8b | 8B | Text (reasoning) | ✅ 24.5s | |
| 12 | qwen2.5-coder:7b | 7B | Text (code) | ✅ 26.5s | |
| 13 | phi3:mini | 3.8B | Text | ✅ 17.0s | |
| 14 | mistral:7b | 7B | Text | ✅ 20.8s | |
| 15 | llama3.1:8b | 8B | Text | ✅ 27.5s | |
| 16 | qwen2.5:14b | 14B | Text | ✅ 48.3s | |
| 17 | qwen2.5:3b | 3B | Text | ✅ 10.0s | |

**16/17 models warmed up.** `llama3.2-vision:11b` fails to load (server process terminated).

---

## 2. Comprehensive Benchmark Results

### 2.1 Speed Rankings

| Rank | Model | Simple (s) | 1st Token (s) | Reasoning (s) | JSON | Concurrent | Quality Score |
|------|-------|-----------|---------------|--------------|------|------------|--------------|
| 1 | **phi3:mini** | **14.2** | **0.81** | **6.8** | ✅✅ | 1.8s | ⭐⭐⭐⭐⭐ |
| 2 | **llama3.2:3b** | **10.2** | **1.81** | **12.7** | ✅✅ | 1.1s | ⭐⭐⭐⭐⭐ |
| 3 | **llama3.1:8b** | **22.7** | **2.74** | **21.5** | ✅✅ | 1.4s | ⭐⭐⭐⭐⭐ |
| 4 | **mistral:7b** | **26.0** | **2.00** | **7.4** | ✅✅ | 1.0s | ⭐⭐⭐⭐ |
| 5 | **deepseek-coder-v2** | **28.3** | **1.17** | **8.1** | ✅✅ | 2.2s | ⭐⭐⭐⭐ |
| 6 | qwen2.5:3b | 9.6 | 1.69 | 9.1 | ❌ | 0.9s | ⭐⭐⭐ |
| 7 | llava:7b | 21.9 | 1.23 | 3.6 | ✅ | 1.0s | ⭐⭐⭐ |
| 8 | qwen2.5-coder:7b | 21.0 | 2.58 | 3.8 | ❌ | 1.5s | ⭐⭐⭐ |
| 9 | qwen2.5:14b | 38.7 | 4.03 | 25.0 | ✅ | 1.8s | ⭐⭐⭐ |
| 10 | qwen2.5:32b | 114.4 | 10.48 | 73.2 | ✅ | 5.7s | ⭐⭐ |
| 11 | moondream:latest | 8.4 | — | 2.4 | ✅ | 0.5s | ⭐ (vision only) |
| 12 | minicpm-v:latest | 24.0 | 1.04 | 2.3 | ❌ | 1.6s | ⭐ (vision only) |
| 13 | deepseek-r1:14b | 12.0 | — | 36.3 | ❌ | 2.9s | ⚠️ broken |
| 14 | deepseek-r1:8b | 16.4 | — | 22.3 | ❌ | 2.0s | ⚠️ broken |
| 15 | ornith:35b | 85.7 | — | 17.1 | ✅ | 3.1s | ⚠️ broken |
| 16 | ornith:9b | 40.4 | 3.68 | 27.7 | ✅ | 3.3s | ⚠️ broken |

---

### 2.2 Detailed Model Analysis

#### 🟢 Winner: phi3:mini (3.8B)
- **Simple:** 14.2s → "4" ✅
- **Reasoning (snail problem):** 6.8s, 1663 chars with full step-by-step ✅
- **Multi-turn:** Very fast (1.3-1.5s/turn), high-quality verbose answers
- **JSON format:** YES (0.85s) ✅
- **Long context:** Works well at all sizes (1.3-17.1s)
- **Concurrent:** 1.8s wall time, all OK
- **Verdict:** Best bang for buck. Fast, excellent reasoning, JSON compliant. Smallest GPU footprint.

#### 🟢 Strong: llama3.2:3b (3B)
- **Simple:** 10.2s → "4" ✅
- **Reasoning:** 12.7s, 1519 chars with proper step-by-step
- **JSON format:** YES (1.33s) ✅
- **Concurrent:** 1.1s wall (0.95s avg), fastest concurrent of text models
- **Verdict:** Excellent for fast chat. Good reasoning, small footprint.

#### 🟢 Solid: llama3.1:8b (8B)
- **Simple:** 22.7s → "4" ✅
- **Reasoning:** 21.5s, 1571 chars, strong reasoning
- **JSON format:** YES (2.26s) ✅
- **Long context:** Consistent answers at all sizes
- **Verdict:** Reliable all-rounder. Good for agents needing consistent quality.

#### 🟢 Solid: mistral:7b (7B)
- **Simple:** 26.0s → "4" ✅
- **Reasoning:** 7.4s, 725 chars (fast, concise reasoning)
- **JSON format:** YES (1.39s) ✅
- **Concurrent:** Fastest concurrent of mid-size models (1.0s wall)
- **Verdict:** Good latency, concise outputs. Good for tool-calling agents.

#### 🟢 Code Specialist: deepseek-coder-v2:latest (~16B MoE)
- **Simple:** 28.3s (partial, truncated) ⚠️
- **Reasoning:** 8.1s, 1689 chars ✅ (most detailed output)
- **JSON format:** YES (1.03s) ✅
- **Long context:** Good contextual understanding
- **Concurrent:** 2.2s wall
- **Verdict:** Best for code generation. Longest, most detailed reasoning steps.

#### 🟡 Vision: llava:7b (7B)
- Surprisingly capable at text. Fast reasoning (3.6s), JSON compliant.
- Multi-turn coherent. Good for multimodal use cases.

#### 🟡 Larger Models: qwen2.5:14b, qwen2.5:32b
- **qwen2.5:14b** (38.7s simple): Slower but quality. JSON compliant. Good for heavy tasks that can tolerate latency.
- **qwen2.5:32b** (114.4s simple): Near timeout. 10.5s to first token. Only useful for offline/batch processing. **Not recommended for interactive use.**

#### 🔴 Broken Models (thinkingFormat issue)

The following models **appear broken** due to the `thinkingFormat: qwen-chat-template` compatibility setting:

| Model | Symptom | Root Cause |
|-------|---------|------------|
| **ornith:35b** | Simple returns empty (85.7s), reasoning 0 chars. Multi-turn & JSON work but output truncated. | qwen-chat-template stripping content from non-Qwen outputs |
| **ornith:9b** | Same pattern - simple/reasoning empty, multi-turn works | Same |
| **deepseek-r1:14b** | Simple empty, reasoning content OK (960 chars), multi-turn fails on turns 2-3, JSON NO | DeepSeek models don't use Qwen chat template |
| **deepseek-r1:8b** | Completely broken - ALL responses empty (0 chars), JSON NO | Same, worse because smaller |

**Fix:** The `thinkingFormat: qwen-chat-template` causes the Gateway to filter out `reasoning_content` fields and may also strip non-content tokens. These models need either:
1. `thinkingFormat:` set to `["qwen-chat-template", "deepseek-chat-template"]` (or separate model configs per-model)
2. No thinking format at all (models handle reasoning natively in content)

---

## 3. Recommendations

### 3.1 Primary Model Recommendations

| Use Case | Recommended Model | Why |
|----------|------------------|-----|
| **Fast Chat** (mustafa, susie) | **phi3:mini** (3.8B) | 0.81s first token, 14.2s full response, reasoning capable, JSON compliant, tiny GPU footprint |
| **Fast Chat v2** | **llama3.2:3b** (3B) | 1.81s first token, 10.2s full, good reasoning despite small size |
| **General Agent** (mustafa/nexus default) | **llama3.1:8b** (8B) | 2.74s first token, reliable reasoning, JSON compliant, most consistent across test types |
| **Tool-Calling Agent** | **mistral:7b** (7B) | Fast concurrent handling (0.82s avg), JSON compliant, concise outputs |
| **Coding Agent** | **deepseek-coder-v2:latest** (16B MoE) | Longest reasoning chains (1689 chars), JSON compliant, good context handling |
| **Heavy Reasoning** (archie) | **qwen2.5:14b** (14B) | Best quality-to-speed ratio for complex tasks. 38.7s simple, 25s reasoning |
| **Vision Tasks** | **llava:7b** (7B) | Only vision model with coherent text output |

### 3.2 Agent Reassignment Plan

| Agent | Current Model | Suggested Replacement | Reason |
|-------|--------------|----------------------|--------|
| **mustafa** | deepseek/deepseek-v4-flash (external) | **studiollm/phi3:mini** for fast tasks, **studiollm/llama3.1:8b** as fallback | Zero cost, fast, reasoning-capable |
| **nexus** | deepseek/deepseek-v4-flash (external) | **studiollm/llama3.1:8b** primary → **studiollm/qwen2.5:14b** fallback | Needs reliability + reasoning depth |
| **dave** | ornith:9b → ornith:35b → phi3:mini | **studiollm/phi3:mini** primary → **studiollm/llama3.1:8b** fallback | ornith models are broken with current thinkingFormat |
| **susie** | deepseek/deepseek-v4-flash (external) | **studiollm/phi3:mini** | Light tasks, fast response |
| **archie** | deepseek/deepseek-v4-pro (external) | **studiollm/qwen2.5:14b** primary → **studiollm/llama3.1:8b** fallback | Heavy lifting, best reasoning on StudioLLM |

### 3.3 Gateway Configuration Changes Required

#### a) Register all working models in Gateway

Currently only 2 models registered; need to add all working models:

```
studiollm/phi3:mini
studiollm/llama3.2:3b
studiollm/llama3.1:8b
studiollm/mistral:7b
studiollm/deepseek-coder-v2:latest
studiollm/qwen2.5:14b
studiollm/qwen2.5:32b
studiollm/qwen2.5:3b
studiollm/llava:7b
studiollm/deepseek-r1:14b
studiollm/deepseek-r1:8b
studiollm/qwen2.5-coder:7b
studiollm/orinith:9b     (if thinkingFormat fixed)
studiollm/orinith:35b    (if thinkingFormat fixed)
```

#### b) Fix thinkingFormat issue (HIGH PRIORITY)

Set `thinkingFormat: false` or remove it for models that don't use Qwen template:
- `ornith:9b` and `ornith:35b` — don't use qwen-chat-template
- `deepseek-r1:14b` and `deepseek-r1:8b` — DeepSeek models use different format

Either:
1. Set per-model `thinkingFormat` in model configs, or
2. Remove `thinkingFormat` entirely (models handle reasoning natively)

#### c) Verify injectNumCtxForOpenAICompat

Currently `false`. Test with a model that needs `num_ctx` override. If models work without it, keep as-is. If long context requests fail, set to `true`.

#### d) Warmup strategy

Consider pre-warming frequently-used models on GPU:
- phi3:mini (17s warmup)
- llama3.2:3b (14.5s warmup)
- llama3.1:8b (27.5s warmup)

### 3.4 Performance Summary for Decision-Making

| Model | First Token | Total Simple | Reasoning Quality | JSON | Concurrent | GPU Mem |
|-------|-----------|-------------|-------------------|------|-----------|---------|
| **phi3:mini** | **0.81s** 🏆 | 14.2s | Excellent | ✅ | 1.8s | Very low |
| llama3.2:3b | 1.81s | 10.2s | Excellent | ✅ | 1.1s 🏆 | Very low |
| llama3.1:8b | 2.74s | 22.7s | Excellent | ✅ | 1.4s | Low |
| mistral:7b | 2.00s | 26.0s | Good | ✅ | 1.0s | Low |
| deepseek-coder-v2 | 1.17s | 28.3s | Excellent (code) | ✅ | 2.2s | Medium |
| qwen2.5:14b | 4.03s | 38.7s | Excellent | ✅ | 1.8s | Medium |
| qwen2.5:32b | 10.48s | 114.4s | Excellent | ✅ | 5.7s | High |
| ornith:9b | 3.68s | 40.4s | ❌ broken | ✅ | 3.3s | Medium |
| ornith:35b | — | 85.7s | ❌ broken | ✅ | 3.1s | High |
| deepseek-r1:14b | — | 12.0s | ⚠️ partial | ❌ | 2.9s | Medium |
| deepseek-r1:8b | — | 16.4s | ❌ broken | ❌ | 2.0s | Low |

---

## 4. Key Findings

1. **phi3:mini is the surprise winner** — At 3.8B, it outperforms much larger models in reasoning quality while maintaining the fastest first-token latency (0.81s). Zero cost to operate.

2. **thinkingFormat is actively harmful** — The `qwen-chat-template` breaks 4 models (ornith:9b, ornith:35b, deepseek-r1:8b, deepseek-r1:14b) by stripping response content. This affects **4 of 16 working models**.

3. **No model matches DeepSeek V4 Flash** externally for speed (sub-second initial response). Best StudioLLM can do is phi3:mini (0.81s first token) or llama3.2:3b (1.81s). For deep reasoning tasks, quality is comparable.

4. **Stop using ornith models** until thinkingFormat is fixed — they return empty responses for simple and reasoning prompts, wasting GPU cycles.

5. **Vision models are usable** — llava:7b in particular has surprisingly good text capabilities alongside vision.

6. **Concurrent performance is excellent** — All working models handle 3 simultaneous requests without errors. Queue behavior is good.