Collections: awesome-llama

https://github.com/nomic-ai/gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

ai-chat llm-inference

Last synced: 26 Oct 2026

https://github.com/internlm/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

codellama cuda-kernels deepspeed fastertransformer internlm llama llama2 llama3 llm llm-inference turbomind

Last synced: 26 Oct 2026

https://github.com/InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

codellama cuda-kernels deepspeed fastertransformer internlm llama llama2 llama3 llm llm-inference turbomind

Last synced: 26 Oct 2026

https://github.com/bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

bentoml fine-tuning llama llama2 llama3-1 llama3-2 llama3-2-vision llm llm-inference llm-ops llm-serving llmops mistral mlops model-inference open-source-llm openllm vicuna

Last synced: 26 Oct 2026

https://github.com/bentoml/openllm

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

bentoml fine-tuning llama llama2 llama3-1 llama3-2 llama3-2-vision llm llm-inference llm-ops llm-serving llmops mistral mlops model-inference open-source-llm openllm vicuna

Last synced: 26 Oct 2026

https://github.com/predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

fine-tuning gpt llama llm llm-inference llm-serving llmops lora model-serving pytorch transformers

Last synced: 26 Oct 2026

https://github.com/expectedparrot/edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.

anthropic data-labeling deepinfra domain-specific-language experiments llama2 llm llm-agent llm-framework llm-inference market-research mixtral open-source openai python social-science surveys synthetic-data

Last synced: 26 Oct 2026

https://github.com/Picovoice/picollm

On-device LLM Inference Powered by X-Bit Quantization

compression efficient-inference gemma generative-ai language-model language-models large-language-model llama llama2 llama3 llm llm-inference llms mistral mixtral model-compression natural-language-processing quantization self-hosted

Last synced: 27 Oct 2026

https://github.com/liltom-eth/llama2-webui

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

llama-2 llama2 llm llm-inference

Last synced: 27 Oct 2026

https://github.com/friendliai/friendli-client

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

ai generative-ai gpt gpt3 inference inference-engine inference-server llama2 llm llm-inference llm-ops llm-serving llmops llms mistral ml mlops serving stable-diffusion

Last synced: 27 Oct 2026

https://github.com/vectorch-ai/ScaleLLM

A high-performance inference system for large language models, designed for production environments.

cuda efficiency gpu inference llama llama3 llm llm-inference model performance production serving speculative transformer

Last synced: 26 Oct 2026

https://github.com/aniketmaurya/llm-inference

Large Language Model (LLM) Inference API and Chatbot

chatbot langchain llama llm llm-inference mistral

Last synced: 27 Oct 2026

https://github.com/vemonet/libre-chat

🦙 Free and Open Source Large Language Model (LLM) chatbot web UI and API. Self-hosted, offline capable and easy to setup.

chatbot chatgpt langchain large-language-models llm llm-inference openapi self-hosted

Last synced: 27 Oct 2026

https://github.com/kyegomez/Exa

Unleash the full potential of exascale LLMs on consumer-class GPUs, proven by extensive benchmarks, with no long-term adjustments and minimal learning curve.

inference-engine llama2 llama2-7b llamacpp llamas llm-inference llms opensource

Last synced: 27 Oct 2026

https://github.com/sophgo/LLM-TPU

Run generative AI models in sophgo BM1684X/BM1688

bm1684x bm1688 generative-ai internvl3 large-language-models llama3 llm llm-inference qwen2-5-vl qwen3 qwen3-5 qwen3-vl

Last synced: 26 Oct 2026

https://github.com/microsoft/sarathi-serve

A low-latency & high-throughput serving engine for LLMs

llama llm-inference pytorch transformer

Last synced: 27 Oct 2026

https://github.com/mukel/llama3.java

Llama 3+ inference in pure Java

chatgpt genai gguf huggingface java llama llama3 llamacpp llm llm-inference llms openai simd transformers

Last synced: 27 Oct 2026

https://github.com/inferflow/inferflow

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

baichuan2 bloom deepseek falcon gemma internlm llama2 llamacpp llm-inference m2m100 minicpm mistral mixtral mixture-of-experts model-quantization moe multi-gpu-inference phi-2 qwen

Last synced: 27 Oct 2026

https://github.com/morpheuslord/HackBot

AI-powered cybersecurity chatbot designed to provide helpful and accurate answers to your cybersecurity-related queries and also do code analysis and scan analysis.

ai automation chatbot cli-chat-app cybersecurity cybersecurity-education cybersecurity-tools llama-api llama2 llama2-7b llamacpp llm-inference runpod

Last synced: 27 Oct 2026

https://github.com/AI-Hypercomputer/jetstream-pytorch

PyTorch/XLA integration with JetStream (https://github.com/google/JetStream) for LLM inference"

attention batching gemma inference llama llama2 llm llm-inference model-serving pytorch tpu

Last synced: 27 Oct 2026

https://github.com/interestingLSY/swiftLLM

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

cuda gpt inference inference-engine llama llm llm-inference llm-serving llmops mlops model-serving pytorch transformer transformers

Last synced: 27 Oct 2026

https://github.com/rohan-paul/LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

gpt-3 gpt3-turbo large-language-models llama2 llm llm-finetuning llm-inference llm-serving llm-training mistral-7b open-source-llm pytorch

Last synced: 27 Oct 2026

https://github.com/chris-ch/llama2.hs

Inference Llama 2 in one file of pure Haskell (A port of llama2.c from Andrej Karpathy)

llama llm-inference neural-network

Last synced: 27 Oct 2026

https://github.com/hpcaitech/SwiftInfer

Efficient AI Inference & Serving

artificial-intelligence deep-learning gpt inference llama llama2 llm-inference llm-serving

Last synced: 27 Oct 2026

https://github.com/noco-ai/spellbook-docker

AI stack for interacting with LLMs, Stable Diffusion, Whisper, xTTS and many other AI models

automatic-speech-recognition bark llama2 llm-inference mixtral musicgeneration stable-diffusion text-to-speech whisper xttsv2

Last synced: 27 Oct 2026

https://github.com/cgbur/llama2.zig

Inference Llama 2 in one file of pure Zig

llama llama2 llm llm-inference simd zig ziglang

Last synced: 27 Oct 2026

https://github.com/Opla/opla

Empower Your Productivity with Local AI Assistants

ai-agent-front ai-assistant aiagent aiassistant generative-ai gpt llama llama2 llamacpp llm llm-inference llmops local-ai localai opla

Last synced: 26 Oct 2026

https://github.com/ictnlp/TruthX

Code for ACL 2024 paper "TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space"

baichuan chatglm chatgpt explainable-ai gpt-4 hallucination hallucinations language-model llama llama2 llama3 llm llm-inference llms mistral representation safety truthfulness

Last synced: 27 Oct 2026

https://github.com/luchangli03/export_llama_to_onnx

export llama to onnx

llama llm llm-inference onnx onnxruntime pytorch

Last synced: 27 Oct 2026

https://github.com/harleyszhang/llm_counts

llm theoretical performance analysis tools and support params, flops, memory and latency analysis.

gpu-performance llama llm llm-inference profiler python3 transformer

Last synced: 27 Oct 2026

https://github.com/edgett/PalmHill.BlazorChat

PalmHill.BlazorChat is a chat application and API built with Blazor WebAssembly, SignalR, and WebAPI, featuring real-time LLM conversations, markdown support, customizable settings, and a responsive design. This project supports Llama2 models and was tested with Orca2.

blazor-webassembly chatgpt csharp gpu llama2 llm llm-inference local-llm markdown-support net8 orca2 realtime-chat signalr webapi

Last synced: 27 Oct 2026

https://github.com/Riccorl/llama-trainer

Llama Trainer Utility

huggingface llama llm llm-inference llm-training llms transformer

Last synced: 27 Oct 2026

https://github.com/EmbeddedLLM/embeddedllm

EmbeddedLLM: API server for Embedded Device Deployment. Currently support CUDA/OpenVINO/IpexLLM/DirectML/CPU

aipc cpu directml directx-12 gemma ipexllm llama llm llm-inference llm-serving mistral model-inference npu open-source-llm openvino openvino-inference-engine phi-3 windows

Last synced: 27 Oct 2026

https://github.com/LLM-Semantic-Router/vllm-router

vLLM Router

huggingface kubernetes llama2 llm llm-inference vllm

Last synced: 27 Oct 2026

https://github.com/monk1337/auto-ollama

run ollama & gguf easily with a single command

autogguf autoollama gguf inference llama llm llm-inference lora mergelora mistral ollama openai

Last synced: 27 Oct 2026

https://github.com/1b5d/llm-api

Run any Large Language Model behind a unified API

chatgpt gptq huggingface langchain llama llamacpp llm llm-inference machine-learning python

Last synced: 27 Oct 2026

https://github.com/Ki-Seki/chat_prompt_templates

Collection of Basic Prompt Templates for Various Chat LLMs (Chat LLM 的基础提示模板集合)

ai baichuan basic chat-model chatglm internlm language-model large-language-models llama llm-inference llms phi-2 prompt prompt-engineering prompt-template prompting qwen yi-34b

Last synced: 27 Oct 2026

https://github.com/adithya-s-k/CompanionLLM

CompanionLLM - A framework to finetune LLMs to be your own sentient conversational companion

fine-tuning finetuning hacktoberfest hacktoberfest-accepted hacktoberfest2023 huggingface llama llama2 llamacpp llm llm-inference llm-training lora mit-license open-source peft

Last synced: 27 Oct 2026

https://github.com/K024/llm-sharp

Language models in C#

baichuan csharp language-model llama llm llm-inference qwen sentence-transformers torchsharp

Last synced: 27 Oct 2026

https://github.com/wizzard0/llama2.ts

Llama2 inference in one TypeScript file

llama2 llm llm-inference tinystories

Last synced: 27 Oct 2026

https://github.com/hscspring/llama.np

Inference Llama/Llama2/Llama3 Modes in NumPy

llama llama2 llm llm-inference numpy

Last synced: 27 Oct 2026

https://github.com/modelize-ai/LLM-Inference-Deployment-Tutorial

Tutorial for LLM developers about engine design, service deployment, evaluation/benchmark, etc. Provide a C/S style optimized LLM inference engine.

gpt inference llama llm llm-inference model-serving transformer tutorial

Last synced: 27 Oct 2026

https://github.com/BorjaOteroFerreira/IALab-Suite

Tool for test diferents large language models without code.

api-rest chat-application flask-api inference-api large-language-models llama-cpp-python llama2 llama2-7b llamacpp llm llm-inference local-inference mixtral-8x7b

Last synced: 27 Oct 2026

https://github.com/YAV-AI/amazon-bedrock-node-js-samples

This repository contains Node.js examples to get started with the Amazon Bedrock service.

amazon-bedrock amazon-titan aws aws-bedrock aws-lambda-node aws-sdk aws-sdk-javascript claude claude-3 claude-api cohere jurassic-ultra language-model llama2 llm-inference mistral mixtral-8x7b-instruct nodejs nodejsexamples stable-diffusion

Last synced: 27 Oct 2026

https://github.com/katha-ai/VELOCITI

VELOCITI Benchmark Evaluation and Visualisation Code

artificial-intelligence awesome-list benchmark benchmarking chain-of-thought clip dataset evaluation-metrics llama3 llm llm-inference semantic-role-labeling video video-understanding vision-language-model vlm

Last synced: 27 Oct 2026

https://github.com/danielgrittner/llama2-rs

LLaMA2 + Rust

ai deep-learning llama2 llama2c llm llm-inference rust

Last synced: 27 Oct 2026

https://github.com/hitz-zentroa/This-is-not-a-Dataset

We introduce a large semi-automatically generated dataset of ~400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms that we use to evaluate LLMs

benchmark common-sense commonsense decoder huggingface llama llama2 llm llm-inference negation scorer transformer

Last synced: 27 Oct 2026

https://github.com/kaalen/tiny-assistant

Experimenting with smaller LLMs that can run on commodity hardware like Raspberry Pi as personal assistant

llama llm-inference raspberrypi

Last synced: 27 Oct 2026

https://github.com/tipani86/MenagerAI

Test various open source language models alongside with ChatGPT and compare the differences.

chatgpt llama2 llm llm-inference opensource streamlit

Last synced: 27 Oct 2026

https://github.com/tbogdala/sentient_core

A terminal style user interface to chat with AI characters using llama LLMs for locally processed AI.

ai chat-application ggml llama llamacpp llm llm-inference rust terminal-ui

Last synced: 27 Oct 2026

https://github.com/chriamue/chat-flame-backend

ChatFlameBackend is an innovative backend solution for chat applications, leveraging the power of the Candle AI framework with a focus on the Mistral model

backend-api candle huggingface-inference-endpoint llama2 llm-inference mistral phi rust-lang

Last synced: 27 Oct 2026

https://github.com/kikirizki/miniChatbot

The minimum implementation of chatbot using popular LLM model rewrite from ground up with simplicity in mind

llama llama2 llm llm-inference llm-training mistral nlp transformers transformers-architecture

Last synced: 27 Oct 2026

https://github.com/kausthub-kannan/Drake

DrakeLLM is developed to help students to solve the issue of making notes from videos, books and others. Utilising RAG, Drake helps in making quick notes along with a Q&A bot. Books, YouTube tutorials or Videos, Drake supports all your means.

deeplake gemini-api gemini-pro huggingface llama2 llm llm-inference llmops rag

Last synced: 27 Oct 2026

https://github.com/Aadit3003/llm-rhyme-eval

English and Dutch rhyming datasets (5k word pairs each) for five types of rhymes. Three open-source LLMs (Llama2, Llama3, CrystalChat) are tested on these datasets, with prompt variation.

llama2 llama3 llm llm-evaluation llm-inference nlp orthography phonology rhyme rhyme-analysis

Last synced: 27 Oct 2026

https://github.com/niansa/libjustlm

Super easy to use library for doing LLaMA/GPT-J stuff! - Mirror of: https://gitlab.com/niansa/libjustlm

ai cpp17 cpp20 gpt-j llama llama2 llm llm-inference mpt python wrapper-library

Last synced: 27 Oct 2026

https://github.com/HROlive/Poland-End-To-End-LLM-Bootcamp

This bootcamp is designed to give NLP researchers an end-to-end overview on the fundamentals of NVIDIA NeMo framework, complete solution for building large language models. It will also have hands-on exercises complimented by tutorials, code snippets, and presentations to help researchers kick-start with NeMo LLM Service and Guardrails.

gpt llama2 llm llm-inference llm-training nemo-guardrails nvidia nvidia-nemo p-tuning prompt-tuning tensorrt triton

Last synced: 27 Oct 2026

https://github.com/niansa/discord_llama

Multi-Model and multi-tasking llama Discord Bot - Mirror of: https://gitlab.com/niansa/discord_llama

ai cpp20 discord-bot llama llama2 llamacpp llm llm-inference

Last synced: 27 Oct 2026

https://github.com/mohitpg/LLMs-from-scratch

A collection of LLMs implemented from scratch using pytorch

gpt llama llm llm-inference model pytorch

Last synced: 27 Oct 2026

https://github.com/mukeshmithrakumar/LLM-POC-2024

Popular Large Language Models from scratch - 2024

gpt llama llm llm-inference llm-training transformer

Last synced: 27 Oct 2026

https://github.com/siripragadashashank/accio

AccIo - Enterprise LLM : Unifying intelligence at your command!

chromadb llama2 llm-inference sagemaker streamlit

Last synced: 27 Oct 2026

https://github.com/tstambulsky/llm-local

This application uses Streamlab and Streamlit to create an interactive web interface for chatting with a locally hosted Llama 3 language model.

llama3 llm llm-inference local-llm openai

Last synced: 27 Oct 2026

https://github.com/youcefguichi/LLM-on-your-laptop

Deploy your favourite LLM model onto kubernetes.

llama3 llm-inference llms

Last synced: 27 Oct 2026

https://github.com/Regular-Baf/bafchat

Bringing local LLMs to a Minecraft front-end through commands.

ai api gpt llama llm llm-inference minecraft minecraft-mod redpajama

Last synced: 27 Oct 2026

https://github.com/AI-Hypercomputer/JetStream

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

gemma gpt gpu inference jax large-language-models llama llama2 llm llm-inference llmops mlops model-serving pytorch tpu transformer

Last synced: 26 Oct 2026

https://github.com/howardchiang2/easy_mode_LLM_inference

简单的LLama.cpp使用

llama2 llm llm-inference

Last synced: 27 Oct 2026

https://github.com/Karan-parmar-007/Blog-AI

Blog AI is an innovative blogging platform that combines advanced artificial intelligence and cloud technologies to revolutionize the way users create, share, and engage with content. With a range of features designed to enhance the blogging experience, Whisper AI empowers users to express themselves freely while ensuring privacy and security.

artificial-intelligence bootstrap django html-css-javascript langchain large-language-models llama2 llamacpp llm-inference python3 security translation

Last synced: 27 Oct 2026

https://github.com/Tiiny-AI/PowerInfer

High-speed Large Language Model Serving for Local Deployment

large-language-models llama llm llm-inference local-inference

Last synced: 27 Oct 2026

https://github.com/JSALT2024/LLMrephrase

Text enrichment using GPT-3.5/4o, Llama3-8B, Llama3-70B, Claude3.5, Gemini

augmentation claude-3 gemini gpt llama3 llm-inference nlp paraphrase-generation rephrase

Last synced: 27 Oct 2026

https://github.com/Mobile-Artificial-Intelligence/llama_sdk

lcpp is a dart implementation of llama.cpp used by the mobile artificial intelligence distribution (maid)

facebook flutter-ai gemma ggml gguf llama llama2 llamacpp llm llm-inference local-ai meta mistral mixtral mobile-ai

Last synced: 27 Oct 2026

https://github.com/b4rtaz/distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

distributed-computing distributed-llm llama2 llama3 llm llm-inference llms neural-network open-llm

Last synced: 26 Oct 2026

https://github.com/Oscar-Mesa/Pofi-Assistant

Este es mi primer proyecto relacionado con modelos de lenguaje, el objetivo es crear un asistente capaz de realizar búsquedas, manipular archivos y tener conversaciones simples

llama llamacpp llm-inference python tkinter

Last synced: 27 Oct 2026