bentoctl

SparkDiffusion 开源

多技术叠加让 DiT 视频生成提速 265 倍

SparkDiffusion: Accelerating DiT video generation by 265× via joint Sparsity, Disti···

llm-d-kv-cache 开源

把大模型KV缓存拆到多机，省显存提吞吐

Distributed KV cache scheduling & offloading libraries

BentoML 开源

模型打包一键变线上服务，上线运维全不操心

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues···

ollama-intel-arc 开源

让 Intel Arc 显卡跑起 Ollama 和 Stable Diffusion，一键部署 AI 全栈。

Make use of Intel Arc Series GPU to Run Ollama, StableDiffusion, Whisper and Open We···

vllm-ascend 开源

让 vLLM 在昇腾 NPU 上跑起来，无缝用上大模型推理

Community maintained hardware plugin for vLLM on Huawei Ascend

tiny-vllm 开源

用 C++/CUDA 手写一个能跑的迷你 vLLM

Build your own high performance LLM inference engine in C++ and CUDA - a smaller ver···

TriviumDB 开源

一个内核搞定向量、图、文档，写入即用

A lightweight embedded database engine that natively fuses vector search, graph, and···

LLMinator 开源

拖拽式界面，一键运行 Hugging Face 开源大模型

Gradio based tool to run opensource LLM models directly from Huggingface

universal 开源

用可定制数字类型，在精度与性能间自由取舍

Large collection of number systems providing custom arithmetic for mixed-precision a···