GGUF in Transformers: a new option for local AI on Macs

A local AI pilot should begin with a useful job and a machine you already understand. A new runtime option is worth investigating when it makes that job easier to operate, integrate or evaluate.

On 22 September, Hugging Face announced efficient GGUF execution inside Transformers, initially targeting Apple Silicon and supported Qwen3.5 models. This is a development-path release, not a reason to assume every existing Transformers installation now has the feature. Hugging Face announcement

What changes for a local AI project?

GGUF is a model file format that packages tensors and metadata and supports different quantisation types. GGUF documentation

The important distinction is how those weights run. Current Transformers main documentation describes a packed execution path using compatible kernels on MPS, Apple's GPU backend. If the required kernel is unavailable, loading can fall back to dequantising the weights. Successful loading alone therefore does not prove that the memory-efficient path is active. Transformers GGUF documentation

Our view: teams already building in Python have a useful integration option to evaluate. Teams happy with their existing local runtime can wait for a clear operational benefit before changing it.

A smaller file is a measurable benefit

The chart compares four weight files for the same Qwen3.5-4B model from the Unsloth repository. Values are decimal gigabytes, rounded from publisher file metadata checked on 25 September.

Weight format	File size
BF16	8.42 GB
Q6_K	3.53 GB
Q5_K_M	3.14 GB
Q4_K_M	2.74 GB

These are model file sizes. They do not measure runtime memory, answer quality or generation speed. The separate vision projector is excluded. Pinned publisher file listing

For planning, keep three questions separate: how much you download, how much memory the working application needs, and whether its answers are good enough. This chart answers only the first. It provides no basis for declaring Q4_K_M the best choice for your work.

Why we are not publishing a speed winner

The announcement's comparison uses different measurement windows. Its llama.cpp result measures decoding without prompt processing and averages three runs. The Transformers result includes prefill and uses the best of three warmed runs. Both generate 128 tokens, but that shared token count does not make the timings equivalent. Published benchmark methodology

We have not benchmarked these runtimes ourselves. A leaderboard would imply a certainty the available comparison does not provide.

For a business pilot, our recommendation is to record time to first token, total task time, peak memory and correctness separately. A fast response that needs extensive correction can be the slower business process.

Test the workflow you intend to use

Choose a bounded exercise before downloading several competing models. For example, summarise a small set of sanitised support notes into a fixed handover template.

Use the same inputs and output requirements for each candidate. Fix the model version, quantisation, runtime version, context settings and hardware. Separate cold starts from warmed runs, repeat the work and keep failed attempts in the record.

Assess whether the summary preserves unresolved actions, distinguishes facts from assumptions and avoids inventing details. Score against a written checklist rather than an impression of eloquence. For planning tasks, include dependencies, missing information and the consequences of a wrong step.

This is a proposed evaluation method, not a claim that Mimir has run those tests. Our guide to AI agents for small businesses provides the broader workflow context.

Mimir's recommendation

Treat GGUF in Transformers as a candidate for a controlled Apple Silicon pilot. Check the documented model support and source-install requirements first; do not generalise the initial packed path to every model or operating system. Current installation and support notes

Assign an owner for updates, access controls and recovery. Confirm where the complete application sends data, including integrations and logs, before using confidential material. Budget for hardware and maintenance time; we are not claiming a monetary saving from this announcement.

The useful outcome is a repeatable local workflow with understood limitations. If the underlying process still needs work, start with the approach in our Microsoft 365 workflow automation guide before adding another runtime.