inference

artificial-intelligence chatglm deployment flan-t5 gemma ggml glm4 inference llama llama3 llamacpp llm machine-learning mistral openai-api pytorch qwen vllm whisper wizardlm

Score: 23.93254466318826

Last synced: about 8 hours ago
JSON representation

Repository metadata:

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

Host: GitHub
URL: https://github.com/xorbitsai/inference
Owner: xorbitsai
License: apache-2.0
Created: 2023-06-14T07:05:04.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2026-08-09T16:18:29.000Z (4 days ago)
Last Synced: 2026-08-09T23:07:09.123Z (3 days ago)
Topics: artificial-intelligence, chatglm, deployment, flan-t5, gemma, ggml, glm4, inference, llama, llama3, llamacpp, llm, machine-learning, mistral, openai-api, pytorch, qwen, vllm, whisper, wizardlm
Language: Python
Homepage: https://inference.readthedocs.io
Size: 74.8 MB
Stars: 9,486
Watchers: 60
Forks: 854
Open Issues: 54
Metadata Files:
- Readme: README.md
- License: LICENSE
- Security: SECURITY.md
- Agents: AGENTS.md

Owner metadata:

Name: Xorbits
Login: xorbitsai
Email:
Kind: organization
Description: Xorbits makes it easy to scale ML and DS workloads, from data loading to preprocessing, tuning, training, and model serving.
Website: xorbits.io
Location:
Twitter: Xorbitsio
Company:
Icon url: https://avatars.githubusercontent.com/u/109655068?v=4
Repositories: 1
Last Synced at: 2023-07-04T09:02:42.715Z
Profile URL: https://github.com/xorbitsai

GitHub Events

Total

Create event: 38
Delete event: 12
Fork event: 314
Issue comment event: 4978
Issues event: 1879
Member event: 4
Pull request event: 712
Pull request review comment event: 178
Pull request review event: 473
Push event: 349
Release event: 32
Watch event: 3064
Total: 12033

Last Year

Create event: 38
Delete event: 12
Fork event: 314
Issue comment event: 4978
Issues event: 1879
Member event: 4
Pull request event: 712
Pull request review comment event: 178
Pull request review event: 473
Push event: 349
Release event: 32
Watch event: 3064
Total: 12033

Committers metadata

Last synced: 1 day ago

Total Commits: 2,110
Total Committers: 154
Avg Commits per committer: 13.701
Development Distribution Score (DDS): 0.856

Commits in past year: 743
Committers in past year: 54
Avg Commits per committer in past year: 13.759
Development Distribution Score (DDS) in past year: 0.808

Name	Email	Commits
Xuye Qin	q**e@g**m	304
codingl2k1	1****1	171
Chengjie Li	1****8	167
OliverBryant	5****t	143
Uranus	1****n	136
amumu96	1****6	103
m199369309	z**i@c**m	102
aresnow1	1****1	96
xiaolei	1****s	95
llyycchhee	m**9@1**m	74
XprobeBot	1****t	70
Jun-Howie	6****e	61
Minamiyama	m**a@q**m	51
呆萌闷油瓶	2**2@q**m	43
zwt-1234	1****4	31
pangyoki	p**i@1**m	29
Bojun-Feng	1****g	29
Dreamsome	o**k@g**m	28
jiayini1119	1****9	21
Adam Ning	f**t@g**m	20
hainaweiben	1****n	19
Rui Ji	r**i@u**u	17
JW	k**3@g**m	14
Tai An	a**1@o**m	13
maoyuehui	1****i	12
souljoy	z**e@1**m	12
ZhikaiGuo960110	t**o@v**m	11
Mike Shi	s**i@h**m	9
bash000000	m**3@o**m	8
wxiwnd	4****d	8
and 124 more...

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 2,242
Total pull requests: 1,709
Average time to close issues: about 1 month
Average time to close pull requests: 6 days
Total issue authors: 1,150
Total pull request authors: 150
Average comments per issue: 3.14
Average comments per pull request: 0.58
Merged pull request: 1,325
Bot issues: 2
Bot pull requests: 2

Past year issues: 148
Past year pull requests: 178
Past year average time to close issues: 14 days
Past year average time to close pull requests: 2 days
Past year issue authors: 106
Past year pull request authors: 30
Past year average comments per issue: 3.47
Past year average comments per pull request: 0.83
Past year merged pull request: 114
Past year bot issues: 1
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/xorbitsai/inference

Top Issue Authors

UranusSeven (52)
Minamiyama (28)
leslie2046 (25)
qinxuye (23)
Songjiadong (19)
aresnow1 (18)
lordk911 (18)
moshilangzi (16)
xs818818 (14)
yidasanqian (13)
kelliaao (12)
wangyongpenga (12)
rexjm (12)
andylzming (11)
ChengjieLi28 (11)

Top Pull Request Authors

qinxuye (337)
codingl2k1 (173)
ChengjieLi28 (143)
amumu96 (138)
yiboyasss (114)
Minamiyama (97)
Jun-Howie (84)
aresnow1 (40)
OliverBryant (36)
frostyplanet (32)
m199369309 (32)
UranusSeven (28)
hainaweiben (24)
llyycchhee (21)
leslie2046 (19)

Top Issue Labels

stale (969)
gpu (741)
feature (328)
question (173)
bug (84)
enhancement (37)
good first issue (20)
help wanted (17)
new model (16)
pr welcome (12)
documentation (6)
duplicate (2)
build (2)
high priority (1)
Mend: dependency security vulnerability (1)
testing (1)

Top Pull Request Labels

feature (502)
bug (351)
enhancement (273)
documentation (143)
gpu (80)
build (60)
refactor (47)
testing (13)
new model (7)
question (3)
dependencies (2)
stale (2)
api (1)

Package metadata

Total packages: 2
Total downloads:
- pypi: 16,089 last-month
Total docker downloads: 750
Total dependent packages: 3 (may contain duplicates)
Total dependent repositories: 9 (may contain duplicates)
Total versions: 277
Total maintainers: 1

pypi.org: xinference

Model Serving Made Easy

Homepage: https://github.com/xorbitsai/inference
Documentation: https://xinference.readthedocs.io/
Licenses: Apache-2.0
Latest release: 3.1.0 (published 13 days ago)
Last Synced: 2026-08-12T00:06:30.865Z (1 day ago)
Versions: 150
Dependent Packages: 3
Dependent Repositories: 9
Downloads: 16,089 Last month
Docker Downloads: 750
Rankings:
- Stargazers count: 2.027%
- Downloads: 3.469%
- Average: 4.666%
- Dependent repos count: 4.955%
- Forks count: 5.508%
- Dependent packages count: 7.373%
Maintainers (1)
- xprobe

proxy.golang.org: github.com/xorbitsai/inference

Homepage:
Documentation: https://pkg.go.dev/github.com/xorbitsai/inference#section-documentation
Licenses: apache-2.0
Latest release: v3.1.0+incompatible (published 13 days ago)
Last Synced: 2026-08-12T00:06:31.164Z (1 day ago)
Versions: 127
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent packages count: 6.528%
- Average: 6.747%
- Dependent repos count: 6.966%

Dependencies

.github/workflows/python.yaml actions

actions/checkout v3 composite
actions/setup-python v4 composite
codecov/codecov-action v3 composite
conda-incubator/setup-miniconda v2 composite
isort/isort-action master composite
psf/black stable composite
py-actions/flake8 v2 composite

.github/workflows/release.yaml actions

actions/checkout v3 composite
actions/setup-python v4 composite
pypa/gh-action-pypi-publish v1.5.0 composite

frontend/package.json npm

@eslint/eslintrc ^3 development
@svgr/webpack ^8.1.0 development
@types/node ^20 development
@types/nunjucks ^3.2.6 development
@types/react ^19 development
@types/react-dom ^19 development
eslint ^9 development
eslint-config-next 15.2.3 development
eslint-config-prettier ^10.1.8 development
prettier ^3.8.3 development
tailwindcss ^3.4.17 development
typescript ^5 development
@radix-ui/react-alert-dialog ^1.1.15
@radix-ui/react-checkbox ^1.3.3
@radix-ui/react-collapsible ^1.1.12
@radix-ui/react-dialog ^1.1.15
@radix-ui/react-dropdown-menu ^2.1.15
@radix-ui/react-icons ^1.3.0
@radix-ui/react-label ^2.1.7
@radix-ui/react-popover ^1.1.15
@radix-ui/react-progress ^1.1.7
@radix-ui/react-radio-group ^1.3.8
@radix-ui/react-scroll-area ^1.2.10
@radix-ui/react-select ^2.2.6
@radix-ui/react-separator ^1.1.7
@radix-ui/react-slider ^1.3.6
@radix-ui/react-slot ^1.2.3
@radix-ui/react-switch ^1.2.6
@radix-ui/react-tabs ^1.1.13
@radix-ui/react-tooltip ^1.2.8
@tailwindcss/typography ^0.5.19
autoprefixer ^10.4.0
axios ^1.16.0
class-variance-authority ^0.7.1
clsx ^2.1.1
date-fns ^4.1.0
highlight.js ^11.11.1
lucide-react ^1.16.0
marked ^5.1.2
next 15.2.3
next-themes ^0.4.6
nunjucks ^3.2.4
react ^19.0.0
react-dom ^19.0.0
react-icons ^5.6.0
react-markdown ^9.0.1
recharts ^2.8.0
rehype-katex ^7.0.0
rehype-raw ^7.0.0
rehype-sanitize ^6.0.0
remark-gfm ^4.0.0
remark-math ^6.0.0
sonner ^2.0.7
tailwind-merge ^2.6.0
tailwindcss-animate ^1.0.7

xinference/deploy/docker/requirements-runtime.txt pypi

accelerate ==1.14.0
numpy ==2.3.5
pandas ==3.0.3
torch ==2.11.0
torchaudio ==2.11.0
torchcodec ==0.14.0
torchvision ==0.26.0
transformers ==5.13.1

xinference/deploy/docker/requirements_cpu/requirements_cpu-base.txt pypi

aioprometheus >=23.12.0
async-timeout *
bcrypt >=4.0.0
boto3 >=1.28.55,<1.28.65
cachetools *
click *
fastapi >=0.110.3,<0.137
gguf *
huggingface-hub >=0.19.4
imageio-ffmpeg *
loguru *
natsort *
nvidia-ml-py *
openai >1
opencv-contrib-python-headless *
opentelemetry-api >=1.20.0
opentelemetry-exporter-otlp-proto-grpc >=1.20.0
opentelemetry-exporter-otlp-proto-http >=1.20.0
opentelemetry-instrumentation-fastapi >=0.41b0
opentelemetry-instrumentation-httpx >=0.41b0
opentelemetry-sdk >=1.20.0
orjson *
ormsgpack *
pillow *
protobuf *
pydantic >2
python-jose *
python-multipart >=0.0.18
requests *
setproctitle *
sse_starlette >=1.6.5
tabulate *
tqdm >=4.27
typing_extensions *
uv *
uvicorn *
xoscar >=0.7.2

xinference/deploy/docker/requirements_cpu/requirements_cpu-ml.txt pypi

FlagEmbedding *
accelerate >=0.28.0
autoawq *
bitsandbytes *
controlnet_aux *
einops *
modelscope >=1.19.0
optimum *
peft <=0.17.1
sentence-transformers >=3.1.0
sentencepiece *
tensorizer *
tiktoken >=0.6.0
timm *
torch >=2.0.0
torchaudio *
transformers >=4.53.3
transformers_stream_generator *

.github/workflows/docker-cd.yaml actions

./.github/actions/check-release-permission * composite
actions/checkout v6 composite
actions/github-script 60a0d83039c74a4aee543508d2ffcb1c3799cdea composite
docker/login-action v4 composite

.github/workflows/assign.yaml actions

xinference/deploy/docker/requirements_cpu/requirements_cpu-models.txt pypi

ChatTTS >=0.2.1
HyperPyYAML *
WeTextProcessing <1.0.4
anyascii *
argbind *
cached_path *
cn2an *
conformer *
datamodel_code_generator *
diffusers >=0.32.0
eva-decord *
ffmpy *
flatten_dict *
fugashi *
funasr ==1.2.7
g2p_en *
gdown *
gruut *
hydra-core >=1.3.2
inflect *
jieba *
jj-pytorchvideo *
json5 *
jsonschema *
julius *
kokoro >=0.7.15
langdetect *
librosa *
lightning >=2.0.0
loralib *
matplotlib *
mecab-python3 *
misaki >=0.7.15
munch *
nemo_text_processing <1.1.0
num2words *
omegaconf *
onnxruntime ==1.16.0
onnxruntime-gpu ==1.16.0
pyarrow *
pydub *
pykakasi *
pyloudnorm *
pypinyin *
pyworld >=0.3.4
qwen-vl-utils *
qwen_omni_utils *
randomname *
silero-vad *
soundfile *
tensorboard *
tomli *
torchdiffeq *
unidic-lite *
vector-quantize-pytorch <=1.17.3,>=1.14.24
verovio >=4.3.1
vocos *
x_transformers >=1.31.14
xxhash *

xinference/deploy/docker/Dockerfile docker

node 20-slim build
nvidia/cuda 13.0.2-devel-ubuntu22.04 build

xinference/thirdparty/matcha/utils/monotonic_align/setup.py pypi

.github/actions/check-release-permission/action.yaml actions

actions/github-script 60a0d83039c74a4aee543508d2ffcb1c3799cdea composite

.github/workflows/build-pypiserver-image.yaml actions

./.github/actions/check-release-permission * composite
actions/checkout v6 composite
docker/build-push-action v7 composite
docker/login-action v4 composite
docker/setup-buildx-action v4 composite

.github/workflows/issue.yaml actions

actions/stale v9 composite

.github/workflows/pr_auto_run_gen_docs.yaml actions

actions/checkout v4 composite
actions/setup-python v5 composite

monitor/filebeat/docker-compose.yml docker

docker.elastic.co/beats/filebeat 8.13.4

frontend/package-lock.json npm

821 dependencies

xinference/deploy/docker/docker-compose-distributed.yml docker

xinference/deploy/docker/docker-compose.byo-wheels.yml docker

${XINFERENCE_BYO_PYPISERVER_IMAGE -pypiserver/pypiserver

xinference/deploy/docker/docker-compose.cpu.yml docker

${XINFERENCE_IMAGE -xprobe/xinference

xinference/deploy/docker/docker-compose.yml docker

${XINFERENCE_IMAGE -xprobe/xinference
${XINFERENCE_PYPISERVER_IMAGE -xprobe/xinference-pypiserver

xinference/deploy/docker/docker-compose.airgap.yml docker

${XINFERENCE_GATEWAY_IMAGE -alpine/socat

pyproject.toml pypi

setup.py pypi

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Summary