Skip to content

Latest commit

 

History

History
89 lines (67 loc) · 2.61 KB

BENCHMARK_M4.md

File metadata and controls

89 lines (67 loc) · 2.61 KB

M4 Performance Benchmark

Measured on August 14, 2026 with scripts/benchmark_m4.py.

Environment

Item Value
Machine Mac mini (Mac16,10)
Chip Apple M4, 10-core configuration
Memory 24 GB
Architecture arm64
macOS 27.0
Python 3.12.13
MLX 0.32.0
MLX-Audio 0.4.8
NumPy 2.5.2
MLX device GPU 0
Model Qwen3-TTS 1.7B CustomVoice 8-bit
Model revision 41d3337e8b7f2843a75841595fc14e4b9a7a4b96
Model disk usage 3,080,145,143 bytes

Real-time factor is calculated conventionally:

generation_time / generated_audio_duration

Lower is better. Values below 1.0 indicate generation faster than playback.

Model Load

Metric Result
Cold model load 2.524 s
MLX peak allocation during load 3.147 GB
Process peak resident memory after load 2.169 GB

The process was new and the model files were already installed in the local filesystem cache. This measures model initialization, not network download.

Warm Generation

One compile/generation warm-up was run and excluded before these cases.

Case Voice Generated audio Generation RTF MLX peak
Short sentence Aiden 1.44 s 1.125 s 0.781 4.307 GB
Medium forecast Vivian 11.44 s 9.904 s 0.866 6.587 GB
Medium app message Serena 7.68 s 5.754 s 0.749 6.114 GB
Long paragraph Ryan 26.40 s 23.475 s 0.889 7.719 GB

All measured warm cases generated faster than real time. Memory increased with generated sequence length, and the long paragraph produced the highest observed MLX allocation.

End-To-End CLI

A separate Python process ran the packaged CLI with offline runtime flags:

utter speak "This is a flower." --voice Ryan -o <path> --json
Metric Result
Total command time 5.584 s
Generated audio 2.72 s
End-to-end RTF 2.053
Output mono, 24 kHz, PCM-16 WAV

For short one-shot requests, process startup and model loading dominate total latency. Warm generation itself is practical on this M4. These measurements do not justify adding a daemon to the MVP; a warm server can be reconsidered only if an application demonstrates a repeated-call latency requirement.

Reproduce

Install the pinned model, then run:

uv run utter models pull
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
  uv run python scripts/benchmark_m4.py

The raw JSON report is written to artifacts/benchmarks/m4-report.json and is ignored by Git. TTS sampling is stochastic, so audio duration, time, RTF, and peak memory can vary between runs.