llama.cpp Checkpoint Fix, NuExtract3 VLM, & Qwen3.6 Local Inference Benchmarks

This pull request addresses a critical bug within the `llama.cpp` server implementation, specifically related to the creation of checkpoints. For users leveraging local models for demanding, long-running agentic tasks, such as extensive coding sessions or complex data processing, the ability to reliably save and restore model states via checkpoints is paramount. This fix ensures that `llama.cpp` can consistently manage these states, preventing data loss or unexpected interruptions during long inference sessions. The issue's context is highlighted by a scenario involving an agentic coding workflow: a 50k token discussion followed by a 20k token implementation phase, requiring the agent to read/write files and execute commands. In such complex, multi-turn interactions, checkpointing is essential for robustness and recovery. The resolution of this bug enhances the stability and trustworthiness of `llama.cpp` for production-like local AI deployments, making it more reliable for persistent agentic applications. This development reinforces `llama.cpp`'s role as a foundational tool for local AI, improving its core functionalities for advanced use cases.

Numind has officially released NuExtract3, a new open-weight 4B Vision-Language Model (VLM) designed for efficient Markdown, OCR, and structured data extraction. Operating under the Apache-2.0 license, NuExtract3 is based on Qwen3.5-4B, making it a highly accessible and self-hostable multimodal model. Its primary utility lies in transforming diverse image and text inputs into structured Markdown, streamlining workflows that involve document analysis and information retrieval. The model's 4-billion parameter size indicates that it is well-suited for deployment on consumer-grade GPUs, aligning perfectly with the focus on local inference and accessible hardware. NuExtract3 fills a crucial gap for developers needing to process visual and textual information locally without relying on cloud-based APIs, ensuring data privacy and reducing operational costs. Its ability to perform OCR and extract structured data directly into Markdown makes it a powerful tool for automating document processing, digitizing archives, and enhancing AI agents with robust data ingress capabilities directly from various media formats. This release marks a significant step forward for practical, self-hosted multimodal AI applications.

A recent benchmark showcases an impressive 1000 tokens per second (tps) generation rate for the Qwen3.6 27B model when deployed on NVIDIA V100 GPUs. This achievement highlights significant advancements in local inference acceleration, particularly for open-weight models. The setup achieved this high throughput under conditions of 128 concurrent requests, pushing the boundaries of what's considered achievable for self-hosted LLM inference on professional-grade, yet still localized, hardware configurations. This benchmark is critical for understanding the potential of acceleration techniques in real-world scenarios. While 128 concurrent requests might exceed typical individual user needs, this result demonstrates the sheer efficiency and scalability that can be unlocked with optimized configurations for Qwen3.6 27B. Such high tps rates are vital for applications requiring rapid responses or processing large batches of queries, such as powering local AI agents, conversational AI systems, or data analysis pipelines. This performance metric offers valuable insight for developers and researchers looking to maximize the efficiency of their local LLM deployments, underscoring the ongoing improvements in making powerful open-weight models practical for high-demand self-hosted environments.