Awesome LLM
by Hannibal046 · Hannibal046/Awesome-LLM
A curated list of Large Language Model papers, tools and resources.
A
A
AOpen-source CLI security scanner for agentic workflows. Scans your workflow’s source code, detects vulnerabilities, and generates an interactive visualization along with a detailed security report. Supports LangGraph, CrewAI, n8n, OpenAI Agents, and more.
AGateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
Open-source tool for ML observability that runs in your notebook environment. Monitor and fine tune LLM, CV and Tabular Models.
A paid product for detecting toxicity, hallucination, prompt injection, etc.
AOpen source AutoML tool for RAG. Optimize the RAG answer quality automatically. From generation evaluation datset to deploying optimized RAG pipeline.
DAn open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment.
D
E
EAn all-in-one LLM Agent platform with your private data and knowledge, delivers your production-ready AI Agents on Day 1.
EAn open-source framework to evaluate, test and monitor ML and LLM-powered systems.
GA Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs.
GA handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control.
Isimplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.
J
LA popular Python/JavaScript library for chaining sequences of language model prompts.
LFormerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain.
LOpen Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground.
L
LOpen-source LLM observability, prompt evaulation, and prompt optimzation platform.
LAn open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning.
L
L
L
LA programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools.
M
MOpen Source Hybrid AI Search Engine, Instantly Get Accurate Answers from the Internet, Bookmarks, Notes, and Docs. Support One-Click Deployment
MA python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM.
MLflow: An open-source framework for the end-to-end machine learning lifecycle, helping developers track experiments, evaluate models/prompts, deploy models, and add observability with tracing.
M- A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)
MA TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation).
A bilingual Chinese-English knowledge extraction model with knowledge graphs and natural language processing technologies.
OAn open-source library for evaluating task performance of language models and prompts.
OConfidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.
OA Python library that provides a domain-specific language to simplify prompting and constrain generation.
PTest your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.
P
POpen-source Python tools for testing and evaluating models, vector DBs, and prompts.
QAn interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources.
RCreate, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.
A paid product for building, comparing, and shipping language model apps.
SBuild your own conversational search engine using less than 500 lines of code by LeptonAI.
SA Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning.
Sa chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!
SInteract with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.
SComprehensive set of tools for working with local LLMs for various tasks.
TTensorZero is an open-source framework for building production-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluations, and experimentation.
WDeploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes.
WA paid product for tracking model training and prompt engineering experiments.
WA web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. We approach prompting as a new programming language rather than low/no-code blocks.
Bexplains how to code a Generative Pre-trained Transformer, or GPT, from scratch.
Explore the world of Large Language Models with over 275 custom made figures in this illustrated guide!
An Introductory LLM Textbook Based on A Survey of Large Language Models.
DFreeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
D
FA powerful tool for creating high-quality training datasets for Large Language Models
IOpen-Source Toolkit for Efficient Unstructured Data Processing with Pre-built Modules and Local to Cluster Scalability.
G
HHolistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models.
IThis repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.
a unified platform from LangChain framework for: evaluation, collaboration HITL (Human In The Loop), logging and monitoring LLM applications.
La lightweight LLM evaluation suite that Hugging Face has been using internally.
MA reliable click-and-go evaluation suite compatible with both open-source and proprietary models, supporting MixEval and other benchmarks.
O
Ra framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines.
DMII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.
DEasily deploy any LLM on a VM with minimal configuration, using Ansible.
EA more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
FA distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.
an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.
I
L
L
LA high-throughput and low-latency inference and serving framework for LLMs and VLs
MTo speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
M
OGet up and running with Llama 3, Mistral, Gemma, and other large language models.
OFine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.
PA distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices.
SSGLang is a fast serving framework for large language models and vision language models.
SRun LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.
V
AAn Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite.
A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively.
evaluates LLM's ability to call external functions/tools.
a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner.
CCompassRank is dedicated to exploring the most advanced language and visual models, offering a comprehensive, objective, and neutral evaluation reference for the industry and research.
a benchmark evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes).
a benchmark for evaluating the performance of large language models (LLMs) in various tasks related to both textual and visual imagination.
a meta-benchmark that evaluates how well factuality evaluators assess the outputs of large language models (LLMs).
a benchmark designed to evaluate large language models (LLMs) specifically in their ability to answer real-world coding-related questions.
a benchmark designed to evaluate large language models in the legal domain.
focuses on understanding how these models perform in various scenarios and analyzing results from an interpretability perspective.
a benchmark that evaluates large language models on a variety of multimodal reasoning tasks, including language, natural and social sciences, physical and social commonsense, temporal reasoning, algebra, and geometry.
a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems.
a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU).
a benchmark that evaluates large language models' ability to answer medical questions across multiple languages.
a multimodal question-answering benchmark designed to evaluate AI models' cognitive ability to understand human beliefs and goals.
a benchmark for evaluating AI models across multiple academic disciplines like math, physics, chemistry, biology, and more.
aims to track, rank, and evaluate LLMs and chatbots as they are released.
a biomedical question-answering benchmark designed for answering research-related questions using PubMed abstracts.
benchmark designed to evaluate large language models (LLMs) on solving complex, college-level scientific problems from domains like chemistry, physics, and mathematics.
a benchmark platform designed for evaluating large language models (LLMs) on a range of tasks, particularly focusing on their performance in different aspects such as natural language understanding, reasoning, and generalization.
a Swedish language understanding benchmark that evaluates natural language processing (NLP) models on various tasks such as argumentation analysis, semantic similarity, and textual entailment.
a large-scale Document Visual Question Answering (VQA) dataset designed for complex document understanding, particularly in financial reports.
a large-scale question-answering benchmark focused on real-world financial data, integrating both tabular and textual information.
a benchmark designed to assess the performance of multimodal web agents on realistic visually grounded tasks.
a benchmark that evaluates large multimodal models (LMMs) on their ability to perform human-like mathematical reasoning.
a benchmark dataset testing AI's ability to reason about visual commonsense through images that defy normal expectations.
AOpen-source framework for fine-tuning and evaluating LLMs. It simplifies the process of experimenting with different training configurations and makes it easy to reproduce and share results, supporting features like LoRA, QLoRA, DeepSpeed, PEFT, and multi-GPU setups.
B
CDDeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
GAn implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
L20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
MMDeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.
M
M
NGenerative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains.
OAn Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT).
RAn Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models.
TTA library for accelerating Transformer model training on NVIDIA GPUs.
TRL is a full stack library where we provide a set of tools to train transformer language models with Reinforcement Learning, from the Supervised Fine-tuning step (SFT), Reward Modeling step (RM) to the Proximal Policy Optimization (PPO) step.
UA framework that specializes in efficient fine-tuning. On its GitHub page, you can find ready-to-use fine-tuning templates for various LLMs, allowing you to easily train your own data for free on the Google Colab cloud.
V
F
LCourse to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.
LMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
B2022-06 · BIG-bench · Google
L2019-02 · GPT 2.0 · OpenAI
2022-12 · OPT-IML · Meta
2023-05 · Dromedary · CMU et al.
2021-01 · Switch Transformers · Google
2022-01 · Megatron-Turing NLG · Microsoft&NVIDIA
2021-12 · WebGPT · OpenAI
Aan experimental open-source application showcasing the capabilities of the GPT-4 language model.
CA Chrome extension for OpenAI's ChatGPT, enhancing user privacy by enabling easy hiding and unhiding of chat history. Ideal for privacy during screen shares.
CChatGPT Wrapper is an open-source unofficial Python API and CLI that lets you interact with ChatGPT.
Introducing Cohere Summarize Beta: A New Endpoint for Text Summarization
E
O
D
R
A
AA collection of prompt examples to be used with the ChatGPT model.
AHow to ask LLMs to produce reliable reasoning and make reason-responsive decisions.
AA curated list of awesome projects and resources related to GPT, ChatGPT, OpenAI, LLM, and more.
A
Aa collection of human preference datasets for LLM instruction tuning, RLHF and evaluation.
AA curation of awesome tools, documents and projects about LLM Security.
AA collection of papers and resources about aligning large language models (LLMs) with human.
AA Chinese collection of prompt examples to be used with the ChatGPT model.
A
A
A
A
AThis paper list focuses on the theoretical or empirical analysis of language models, e.g., the learning dynamics, expressive capacity, interpretability, generalization, and other interesting topics.
AA curated list of Multi-modal Large Language Model in 3D world, including 3D understanding, reasoning, generation, and embodied agents.
A
A
A
A
A
AA collection of open source, actively maintained web apps for LLM applications.
C
CA trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models.
IA trend starts from Natrural-Instruction (ACL 2022), FLAN (ICLR 2022) and T0 (ICLR 2022).
L
LApplying Large language models (LLMs) for diverse optimization tasks (Opt) is an emerging research area. This is a collection of references and papers of LLM4Opt.
La curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset
L
RCollection of papers and resources on Reasoning using Language Models.
R
K
T