Awesome NLP

is free and open source, web-based raw text annotation tool

Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS.

brat rapid annotation tool is an online environment for collaborative text annotation

support various NLP tasks for individual or teams, freemium based

doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence

FLAT is a web-based linguistic annotation environment based around the FoLiA format, a rich XML-based format for linguistic annotation. Free and open source.

General Architecture and Text Engineering is 15+ years old, free and open source

open source server annotation tool with GitHub version control and validation for XML data and collaborative spreadsheet grids

A semantic annotation platform offering intelligent assistance and knowledge management

team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $

open-core multi-modal labeling platform; widely used for NLP labeling.

Hosted and managed text annotation tool for teams, costs $

Free, open-source annotation tool covering 21+ task types (classification, span, coreference, entity linking, agent trace evaluation) with built-in MACE quality control, attention checks, AI-assisted labeling, and 300+ example tasks.

is an annotation tool powered by active learning, costs $

open source local or online tool for discourse tree annotations

Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. Shoonya is data agnostic, can be used by teams to annotate data with various level of verification stages at scale.

Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling, costs $

large collection of Chinese NLP tools and resources.

(Anthropic, 2024) - models strategically complying during training.

social bias measurement in masked LMs.

(2025) - finetuning on a narrow task (insecure code) unexpectedly produces broad alignment failures across unrelated domains.

bias measurement across many demographic axes.

(2026) - multilingual safety benchmark across 12 Indic languages; reveals 12.8% cross-language agreement, with over-refusal in low-resource scripts.

toxicity in LM generation.

(2025) - multilingual (Chinese/English) safety benchmark of 4000+ multi-turn dialogues across 22 scenarios and 7 jailbreak strategies.

measuring stereotypical bias in pretrained LMs.

models tailoring answers to user beliefs.

(2025) - modular jailbreak evaluation framework integrating 19 attacks, 29 defenses, and 19 evaluation methods across 14 models and 12 risk categories.

(2026) - alignment faking occurs in models as small as 7B in 37% of cases when policy conflicts with internalized values; steering-vector mitigation reduces it 94%.

(2024) - open safety moderation model and benchmark.

gender bias in coreference resolution.

by Stephan Raaijmakers

free, NLP notes by Dr. Jacob Eisenstein at GeorgiaTech

by Hobson Lane and Maria Dyshel

Brian & Delip Rao

by Masato Hagiwara

free, by Prof. Dan Jurafsy

bt Tiago MOnteiro A free FreeCodeCamp book teaching the math behind AI in plain English from an engineering point of view. It covers linear algebra, calculus, probability & statistics, and optimization theory with analogies, real-life applications, and Python code examples.

by Nicole Koenigstein

higher-order inference coreference.

foundation for modern neural coreference.

linguistically-motivated category-based coreference scoring.

prompting and fine-tuning for coreference.

(2024) - efficient coreference matching the best larger systems.

(2025) - 9 systems across 4 LLM-based and 5 traditional approaches; traditional methods still lead but LLMs are closing the gap.

span-based pretraining; strong coreference baseline.

Sentiwordnet, parallel labelled corpora, sense-annotated corpora, and Marathi polarity-labelled corpus.

Twitter and Facebook labelled sentiment samples in Hindi, Bengali, Tamil, Telugu.

Vietnamese dependency treebank.

20M words across 15 bilingual books, 100 parallel English-Vietnamese texts, 250 parallel law texts, 5K news articles, and 2K film subtitles.

text corpus with 5M words and word segmentation.

dataset of speeches by the current Prime Minister of Thailand.

Vietnamese Universal Dependency Treebank.

10K sentences for the constituency parsing task.

Vietnamese Text-to-SQL semantic parsing dataset (EMNLP-2020 Findings).

free Vietnamese speech corpus, 15 hours of recorded speech (HCMUS AILab).

1.75M news sentences.

5.5k Samples, 2 Domains, 10 aspect terms

4.3k Samples, 14 classes

A multi-representational multi-layered treebank for Hindi and Urdu

~1k Samples, 3 polarity classes

5.4k Samples, 12 Domains, 4k aspect terms, aspect and sentence level polarity in 4 classes

2k Samples, 3 polarity labels

60k Words POS Tagged, Bangla, Hindi, Marathi, Telugu

A smaller part of the above-mentioned treebank.

aggregated Arabic stopwords.

(2024) - Arabic MMLU benchmark.

250K tokens, 7,682 sentences with NER tags in IOB format.

tagged corpus suitable for Persian (Farsi) NLP research, ~2.6M manually tagged words across 40 POS tags.

chatbot data in Korean.

dataset in Korean from a major South Korean newspaper.

POS, NER, and other NLP tasks.

(2024) - 2T-token open-license multilingual corpus.

6.3T tokens across 167 languages.

(AI2, 2023-2024) - 3T-token open pretraining corpus with documented filtering pipeline.

~25M tokens, ~1M Persian sentences from Persian Wikipedia Corpus.

(2024) - 15T-token cleaned web corpus; FineWeb-Edu filters for educational quality.

data repository for pretrained NLP models and NLP corpora.

standard reliable Persian text collection used at CLEF 2008-2009.

the central index for modern NLP datasets, with versioned, streamable loaders.

10K sentences and 250K word tokens.

and Universal Dependencies-Indonesian

text summarization and classification.

corpus from the Korea Advanced Institute of Science and Technology in Korean.

NMT dataset for Korean to French and Korean to English.

Korean SQuAD dataset (v1.0 and v2.1) with Wiki HTML source.

large Arabic book reviews dataset.

LSCP: 120M sentences from 27M casual Persian tweets with dependency, POS, and sentiment annotations.

largest available multi-domain Arabic sentiment analysis resources.

large collection of NLP datasets.

39K sentences and 900K word tokens

first Persian dataset for relation extraction (translated SemEval-2010 Task 8).

29,982 annotated sentences covering most verbs of the Persian valency lexicon.

expired petition data from the Blue House National Petition Site.

(2023-2024) - reproductions of LLaMA pretraining data; V2 is 30T tokens with quality signals.

A multilingual and multimodal data hub providing standardized datasets and benchmarks for Southeast Asian NLP (EMNLP 2024).

Bilingual retrieval-evaluation benchmark for culturally grounded RAG. 400 rows, EN+UZ, MIT/CC-BY-4.0.

825 GiB diverse text corpus.

tiny NLP multi-lingual QA datasets and library to generate your own synthetic copies.

(2024) - open instruction-tuning data behind Tülu 3.

large freely available Persian corpus, 2.7M tokens annotated with 31 POS tags.

dependency-based syntactically annotated corpus.

large, free, semantic dictionary.

(Apple, 2025) - on-device 3B model using KV-cache sharing and 2-bit QAT for 37.5% cache memory reduction without accuracy loss.

activation-aware weight quantization.

and MiniLM - distilled encoders for production NLP.

(2024) - weight-decomposed low-rank adaptation.

(Google, 2025) - 1B-27B open models with high local-to-global attention ratio to keep KV-cache tractable at 128K context.

portable quantized inference.

post-training quantization for transformers.

, BGE, and Stella - compact text embedding models near the top of MTEB.

(ICML 2025) - sensitivity-aware layer-wise mixed-precision KV-cache quantization; up to 21% throughput improvement over uniform KV8.

and QLoRA - low-rank adapters and quantized fine-tuning; the standard for adapting LMs to NLP tasks on modest hardware.

HuggingFace library bundling LoRA, prefix tuning, IA3, and others.

(Microsoft, 2024) - small models trained on curated data, competitive with much larger ones on NLP benchmarks.

(Alibaba, 2025) - dense and MoE models 0.6B-235B with unified thinking/non-thinking modes; the 30B-A3B MoE matches larger dense models while activating only 3B parameters.

sentence and paragraph embeddings via Siamese BERT.

structured generation and efficient serving.

(HuggingFace, 2025) - fully open small-LM family with reproducible training data.

(HuggingFace, 2025) - 3B fully open decoder pretrained on 11.2T tokens with NoPE and YaRN for 128K context; competitive with 4B-class models.

HF production serving for LMs.

PagedAttention-based high-throughput LM serving.

heterogeneous IR benchmark for retrieval models.

200+ tasks probing language model capabilities.

human-preference ELO leaderboard for chat models.

MT evaluation across 200 languages.

(2025) - unified RAG evaluation: 824 multi-hop questions requiring factuality, retrieval accuracy, and cross-document reasoning together.

and SuperGLUE - English NLU benchmarks.

graduate-level Q&A, "Google-proof" reasoning evaluation.

holistic evaluation across NLP tasks, accuracy and beyond.

verifiable instruction-following evaluation.

(2024) - contamination-resistant benchmark with monthly refresh.

unified framework for LM benchmark evaluation.

bilingual long-context benchmark across NLP tasks.

(2025) - 503 expert-crafted multiple-choice questions spanning 8K-2M-word contexts with deep multi-hop reasoning; humans score 53.7% under time pressure.

multitask knowledge evaluation across 57 subjects.

(2024) - harder, more discriminative successor to MMLU.

(2025) - multilingual extension of MMLU-Pro to 29 typologically diverse languages; reveals up to 24.3% performance gap between high- and low-resource languages.

Massive Text Embedding Benchmark; standard for sentence/document encoders.

(2025) - multi-turn conversational benchmark exposing simultaneous instruction-following and in-context-reasoning failures; all tested frontier models score below 50%.

retrieval probe for long-context windows.

(2026) - scientific reasoning benchmark for evidence-grounded critique, overclaim detection, missing-evidence refusal, and calibration.

(2024) - synthetic long-context tasks beyond simple retrieval.

(2025) - extends needle-in-haystack with multi-needle and nested configurations; shows RAG mitigates lost-in-the-middle for smaller LLMs but degrades reasoning models.

cross-lingual natural language inference, 15 languages.

and XGLUE - cross-lingual NLU.

(2025) - claim-level calibration analysis for long-form generation; models are substantially worse-calibrated on extended outputs than on single claims.

(2024) - calibration analysis under format effects.

(2026) - trains models to reason about claim-level uncertainty before generating; large gains on biography factuality and FactBench AUROC.

fine-grained factual precision in long-form generation.

(2025) - faithfulness-aware uncertainty quantification for RAG fact-checking; formally separates faithfulness from factuality.

(2026) - hard multi-turn hallucination benchmark for citation-required responses; ~30% hallucination rates persist even with web search.

(2025) - hallucination benchmark with extrinsic/intrinsic taxonomy and dynamic test-set regeneration to resist data leakage.

(2024) - long-form factuality benchmark and search-augmented evaluator.

(2024) - attention-pattern-based hallucination detection in long-context generation.

(2025) - multilingual claim-hallucination benchmark across English, French, Spanish, German with token-level logits released for principled UQ evaluation.

reference-free evaluation for RAG and QA pipelines.

sampling-based hallucination detection.

taxonomy and mitigation strategies.

benchmark for truthfulness in question answering.

2018 essay on the rise of pretrained language models.

2017 NLG survey.

and The Illustrated BERT, ELMo, and co. - canonical visual explanations.

(2025) - decoder-only LLMs with a novel error-rate adaptation schedule set new SOTA on BEA-test grammatical error correction.

document-level relation extraction benchmark.

(2025) - generative LLMs with RAG and self-correction surpass encoder-decoder BERT-style models on SRL in English and Chinese.

schema-free open information extraction.

training LMs with AI-generated feedback against a written constitution.

simpler alternative to RLHF; widely adopted.

finetuned language models as zero-shot learners.

training LMs to follow instructions with human feedback.

"less is more for alignment"; small high-quality SFT data goes a long way.

(2024-2025) - synthesizes high-quality instruction-response pairs by prompting aligned LMs with nothing; SFT on the filtered subset matches official Llama-3-Instruct.

bootstrapping instruction data from LMs.

1600+ NLP tasks with instructions.

reference library for SFT, DPO, GRPO, and RLHF.

(AI2, 2024) - fully open post-training recipe with state-of-the-art results among open models.

and nlp-for-hindi ULMFIT style languge model

Trained on Sanskrit Wikipedia and OSCAR corpus

(archived) - An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.

open-source data annotation and feedback collection platform for LLM and NLP datasets.

Toolkit with state-of-the-art automatic term recognition methods.

a fast library for topic modelling

BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)

Python toolkit for Arabic NLP including dialect ID, morphology, NER.

A library for downloading&parsing standard NLP research datasets

Natural Language Processing in Clojure (opennlp)

Core libraries developed in the U of Illinois' Cognitive Computation Group.

C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way.

learned MT metrics, current de-facto standard.

Hierarchical Topic Modeling with Minimal Domain Knowledge

An R package for dynamic exploration of text collections

A variety of loaders for various NLP corpora

Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks.

CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.

word segmentation with deep learning in TensorFlow.

Creates d3 visualizations for browsing topic models of text in a web browser.

R package for exploring topic models of text.

A deep learning-based translation library for 50 languages, built on transformers and Facebook's mBART Large.

Epic is a high performance statistical parser written in Scala, along with a framework for building complex structured prediction models.

Facebook AI Research implementations of SOTA seq2seq models in Pytorch.

QCRI segmentation, POS tagging, and NER for Arabic.

Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings.

Python library for working with FoLiA, an XML format for linguistic annotation.

Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer.

Java library for Chinese text processing.

browser-native Korean morphological analyzer running fully client-side via WebAssembly (1MB model, offline, MIT).

Python library to conduct unsupervised semantic modelling from plain text :+1:

(archived) - A deep learning toolkit for NLP, built on MXNet/Gluon.

Converts numbers to Russian words with correct grammatical gender and noun declension.

Go package for Arabic text processing.

Go implementation of the jieba Chinese word segmentation algorithm.

multilingual NLP library with strong Chinese support.

End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more!

Persian NLP toolkit.

standardized loaders and processing for thousands of NLP datasets.

reference implementations for NLP metrics.

Rails-like inflection library for Clojure and ClojureScript

A neural network library for building instance-dependent NLP models with padding-free dynamic batching.

Python package for Chinese word segmentation.

Japanese Natural Langauge Processing Libraries, with Japanese sentiment classification

A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages.

JavaScript Arabic stemmer.

character cluster library in Java.

Evaluation of NLP model outputs offering various automated metrics.

Japanese morphological analyzer written in pure Go.

Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.

fast Korean morphological analyzer.

A Natural Language Processor in JS

Scala library for Korean NLP.

R package for Korean NLP.

Python package for Korean natural language processing.

an index-based text data generator written in Kotlin

Korean sentence splitter.

60+ test types for NLP model robustness, bias, and fairness.

A package for working with human languages

C++ library for the FoLiA format

A language detection library for Kotlin and Java, suitable for long and short text alike

HIT Language Technology Platform: segmentation, POS, NER, parsing.

MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text.

C++ library for Korean NLP.

ModErn Text Analysis: a C++ data sciences toolkit for mining big text data.

C, C++, and Python tools for named entity recognition and relation extraction

A multilingual and extensible document clustering pipeline

general natural language facilities for node

A library containing a wide variety of NLP functionality, supporting over 50 corpora.

A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU

Natural Language processing in the browser

Wiki entry for the NLP++ language

NLP++ engine to run NLP++ code on Linux including a full English parser

An NLP library for building bots

Fast and production-ready question answering w/ DistilBERT in Node.js

An efficient and flexible token-based regular expression language and engine.

Persian analyzer for Elasticsearch.

Persian language processing toolkit.

Human annotation tool for multilingual NLP tasks, such as machine translation.

Persian keyphrase extraction.

Persian stemmer, morphological analyzer, and partial POS tagger.

A web-based annotation tool for natural language processing (NLP)

A library to parse natural language in Clojure and ClojureScript

Text processing library supporting tokenization, part-of-speech tagging, and named-entity extraction.

Python library for Arabic.

Python Natural Language Processing Library. General purpose NLP library for Python, handles some specific formats like ARPA language models, Moses phrasetables, GIZA++ alignments.

Python package implementing the SS3 white-box text classifier; ships with interactive visualization tools that explain predictions.

Thai NLP in Python.

Unicode-aware regular-expression based tokenizer for various languages. Python binding to C++ library, supports FoLiA format.

NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU

Python Vietnamese core NLP toolkit.

adaptive probabilistic top-down and bottom-up parsers

A robust POS tagging toolkit available (in both Java & Python) together with pre-trained models for 40+ languages.

Extensible system for analyzing and manipulating natural language

Web-Scale Open Information Extraction

trainable segmenter for Arabic, Hebrew, and Coptic.

a DSL, loosely based on RUTA on Apache UIMA. Allows to define language patterns (rule-based NLP) which are then translated into spaCy, or if you prefer less features and lightweight - regex patterns.

R package to interface with the Java machine learning tool MALLET

Text processing tools and wrappers (e.g. Vowpal Wabbit)

Ready-to-use NLP pipelines and Transformer-based models

reproducible BLEU/chrF/TER scoring for machine translation.

Library for developing NLP systems, including built in modules like SRL, POS, etc.

Python library to produce d3 visualizations of how language differs between corpora

sentence/document embeddings, semantic search, and re-ranking; current standard for retrieval-style NLP.

Sentiment models for spacy using onnx

Sentiment Classification using Word Sense Disambiguation and WordNet Reader

A production ready library for intent parsing

(archived — Snips was discontinued) - A production ready library for intent parsing

Python package for Chinese NLP.

Neural Machine Translation (NMT) toolkit that powers Amazon Translate.

Industrial strength NLP with Python and Cython :+1:

R wrapper to spaCy NLP

Python library to detect, censor, and clean profanity, hate speech, and bullying in Spanish, with data from 21 Spanish-speaking countries.

Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment.

Stanford NLP's Python toolkit for tokenization, POS, lemma, dependency parsing, and NER across 70+ languages.

a library from Facebook for creating embeddings of word-level, paragraph-level, document-level and for text classification

word segmentation and POS tagging using deep learning.

Fast vectorization, topic modeling, distances and GloVe word embeddings in R.

Higher level NLP built on spaCy

Julia package for text analysis

Adversarial attacks, adversarial training, and data augmentation in NLP

Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language Toolkit (NLTK) and Pattern, and plays nicely with both :+1:

Neural Network based models for Natural Language Processing

tokenization and POS tagging.

Text mining using tidy tools

Implementation of topic modeling based on regularized multilingual PLSA.

Tokenizers optimized for Research and Production.

Natural Language Processing for TensorFlow 2.0 and PyTorch.

A JavaScript implementation of Twitter's text processing library

Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format.

Vietnamese NLP toolkit.

NLP library for Urdu.

on-device Vietnamese text-to-speech with voice cloning.

Persian text cleaning.

Homepage for the NLP++ Language

Vietnamese text processing toolkit.

Vietnamese NLP toolkit.

NLP++ Language Extension for VSCode

Natural language recognition library based on trigrams

Word forms can accurately generate all possible forms of an English word

Julia interface to word2vec

Scala interface to word2vec model; includes operations on vectors like word-distance and word-analogy.

High performance tokenizers for natural language processing and other related tasks

An R package for creating and exploring word2vec and other word embedding models

natural language toolkit for Indonesian.

trained on Wikipedia.

Python stemmer for Bahasa Indonesia based on the Sastrawi stemming algorithm.

tools, datasets, and models across 22 Indic languages.

tokenization, transliteration, MT helpers across 18 Indic languages.

NLP toolkit for Indic languages on PyTorch/Fastai.

deep morphological parser for Hindi and Urdu.

dependency parsing and POS tagging for Kannada, Hindi, and Telugu.

(2025) - first comprehensive memory and speed analysis of transformer, SSM, and hybrid models up to 220K tokens; SSMs are up to 4x faster, hybrids balance recall and efficiency.

(2024) - hybrid Mamba-Transformer-MoE architecture.

(2025) - identifies undertraining of high-frequency RoPE dimensions and applies evolutionary-search rescaling; extends LLaMA3-8B to 128K with 80x fewer training tokens than Meta's recipe.

long-context degradation patterns in NLP tasks.

and Mamba-2 - selective state-space models, linear-time long-context alternative to attention.

(2025) - 456B-parameter hybrid combining lightning (linear) attention with sparse softmax attention; matches GPT-4o-level NLP performance at up to 4M-token inference contexts.

(2025) - trainable sparse attention combining coarse-grained compression with fine-grained selection; large speedups at 64K with no NLP-benchmark degradation.

extending context windows with minimal fine-tuning.

(2024) - tradeoffs for QA over long inputs.

and YaRN - rotary position embeddings and context-length extension.

RNN-transformer hybrid scaling to large parameter counts.

(2025) - neural long-term memory module that learns to memorize historical context at test time; scales beyond 2M tokens, outperforms transformers and modern linear-recurrent models on language modeling and reasoning.

(2024) - LLMs for context-aware translation.

transformer; reset the field.

similarity-based generation metric.

(2025) - survey of how instruction-following, in-context learning, and preference alignment have restructured MT methodology.

PyTorch sequence modeling toolkit.

quality comparison on professional MT.

LLMs as machine translation systems.

400+ language MT.

efficient C++ NMT framework.

(2025) - benchmarks sub-10B open LLMs on 28-language MT; matches GPT-4-turbo and Google Translate.

MT for 200 languages.

speech and text MT, 100+ languages.

(PORTULAN, 2023-2024) - encoder-only Portuguese LMs for both PT-PT and PT-BR.

BERT for Brazilian Portuguese.

(Part AI, 2024) - Llama-3-based Persian instruction model.

alternative IndoBERT with the IndoLEM benchmark.

pretrained Indonesian LM with the IndoNLU benchmark suite.

(2023-2024) - large-scale community datasets and Cendol instruction-tuned LMs for Indonesian and regional languages.

(2023-2024) - open Thai instruction-tuned models.

Persian BERT.

(2023-2024) - Persian instruction-tuned LM.

(Maritaca AI, 2023-2024) - Portuguese-focused open LMs.

open Southeast-Asian LMs covering Indonesian.

(2024) - Singapore AI's open Southeast-Asian LM with strong Indonesian.

(SCB 10X, 2024) - open Thai LLM family.

pretrained Thai language model.

(2024) - instruction-tuned Hindi LLM.

(SDAIA, 2024) - Arabic-first foundation models.

Arabic BERT family.

efficient Arabic pretraining (released alongside AraBERT).

open Chinese LM.

sequence-to-sequence pretrained model for Vietnamese.

BERT for Spanish.

(2024) - Indic-focused foundation models.

BERT models for MSA, dialectal, and Classical Arabic.

Tsinghua's bilingual Chinese-English LMs.

whole-word masking BERT for Chinese.

efficient open MoE model with strong Chinese.

(LG, 2024) - bilingual Korean-English open LM family.

Naver's Korean foundation model.

(2022-2024) - multilingual BERT for 23 Indic languages.

(2023-2024) - high-quality MT for 22 Indic languages.

(2023-2024) - bilingual Arabic-English open LM family.

models trained on the KLUE benchmark.

Korean BERT from SKT.

(2024) - open foundation LM for Basque, also covers Spanish.

improved Chinese BERT with MLM-as-correction pretraining.

(Sarvam AI, 2023) - bilingual Hindi-English LLaMA continuation.

pretrained LM for Vietnamese.

(VinAI, 2023-2024) - open generative LM for Vietnamese.

open Korean LMs.

Alibaba's open Chinese-strong LM family.

(2024) - Spanish-instruction-tuned open model.

Spanish RoBERTa trained on the Spanish National Library corpus.

(BSC, 2024) - multilingual LM with strong Spanish coverage from the Barcelona Supercomputing Center.

(2024) - multilingual LM trained from scratch on 10 Indic languages.

(2024) - Mistral-based Vietnamese chat model.

01.AI's bilingual open LMs.

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).

is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and so on. Also some non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, lemmatization and disambiguation.

(McGill, 2026) - suite of open LLMs (4B-14B) continued-pretrained on 26B tokens across 20 African languages with a comprehensive empirical study of data mixing.

(Cohere For AI, 2024) - massively multilingual instruction-tuned models covering 23-101 languages.

(2025) - open multilingual LLMs (9B and 83B) covering the top 25 languages by speaker population (~90% of global speakers); surpasses comparably-sized open multilingual models on XCOPA, XNLI, MGSM, FLORES-200.

176B-parameter open multilingual LM, 46 natural languages.

encoder for 500+ languages, focus on low-resource.

(Princeton/Mila, 2025) - Llama-3.1-8B adapted for low-resource African languages via the curated WURA corpus; SOTA open-source results on IrokoBench and AfriQA.

(Xiaomi, 2026) - open multilingual MT scaled across 46 languages, matching commercial systems like Google Translate and Gemini 3 Pro.

multilingual T5 covering 101 languages.

(2024-2025) - LMs targeting Southeast Asian languages.

(Meta, 2023-2024) - multilingual and multimodal speech-text translation, 100+ languages.

(Google, 2026) - open translation-specialized models built on Gemma 3, covering 55 language pairs via SFT and RL with quality-reward models.

cross-lingual masked LM trained on CommonCrawl, 100 languages.

curated list of resources for Danish language technology.

NLP resources in Danish.

dependency parser for Dutch (also does POS tagging and lemmatization).

Dutch speech-recognition models based on Kaldi.

Python binding to Frog, an NLP suite for Dutch (POS tagging, lemmatization, dependency parsing, NER).

Dutch surface realiser for natural language generation, based on the SimpleNLG implementation.

industrial-strength NLP with a Dutch pipeline.

curated list of open-access, open-source, and off-the-shelf resources and tools developed with a focus on German.

curated list of free resources for Hungarian NLP.

curated list of resources dedicated to Polish NLP: models, tools, and datasets.

curated list of Portuguese NLP resources and tools.

curated list of Ukrainian NLP datasets, models, etc.

curated list focused on machine translation and speech processing.

(2024) - cost-quality tradeoffs.

canonical English NER benchmark.

contextual string embeddings, strong NER across languages.

(2026) - eight open LLMs across four NER benchmarks; PEFT with structured outputs matches encoder-based NER.

(2023) - small, generalist NER model that handles arbitrary entity types at inference.

guideline-following information extraction with LMs.

LLMs for named entity recognition.

BiLSTM-CRF, the long-time go-to NER architecture.

end-to-end relation extraction as seq2seq.

production-ready.

instruction-tuned LM for open-set NER across languages.

The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages

implementation in ElasticSearch

A collection of papers, corpora and linguistic resources for NLP in Hebrew

a good pos-tagger for Russian

foundational neural parsing architecture.

strong neural constituency parser.

and Stanza - production parsers across many languages.

light-weight transformer-based multilingual NLP toolkit.

cross-linguistically consistent treebanks, 100+ languages.

bidirectional transformer pretraining; foundation for most encoder-based NLP work since 2018.

disentangled attention; strong on classification, NER, NLI.

(2024) - efficient MoE pretraining; competitive open base model.

replaced-token-detection pretraining, sample-efficient.

(Google, 2024-2025) - open small/mid-size models with strong NLP-task performance.

(Meta, 2024-2025) - widely adopted open-weight family; default base for fine-tuning across NLP tasks.

efficient dense and sparse-MoE open models.

(2024) - modernized encoder with rotary embeddings, FlashAttention, 8K context; current go-to encoder for classification, NER, retrieval.

(2025) - 250M-parameter encoder integrating modern architecture improvements (RoPE, 4K context, optimized depth-to-width); state of the art on MTEB, surpasses ModernBERT and RoBERTa-large under identical fine-tuning.

(AI2, 2025) - fully open: weights, training data, code; reproducibility benchmark.

(Alibaba, 2024-2025) - strong multilingual coverage, especially Chinese; often top open model on multilingual benchmarks.

robustly optimized BERT pretraining; common encoder baseline.

and FLAN-T5 - text-to-text framing for NLP tasks; strong instruction-tuned encoder-decoder baselines.

encoder vs decoder vs encoder-decoder for NLP transfer.

what BERT learns about language.

(Anthropic, 2025) - introduces cross-layer transcoders and attribution graphs to construct an interpretable replacement model; enables prompt-level circuit tracing of feature-to-feature causal interactions.

(2026) - identifies circuits at the per-prompt level (rather than per-task); reveals mechanism clustering by prompt family.

(2023) - identifying training examples driving model behavior.

causal tracing of factual recall.

open platform browsing SAE features across models.

(Anthropic, 2025) - applies attribution graphs to Claude 3.5 Haiku across multi-hop reasoning, rhyme planning, and jailbreak case studies.

methodology, limitations, alternatives.

SAE methodology for LM interpretability.

(EMNLP 2025) - reference survey of SAE architectures, training strategies, feature explanation, and evaluation.

structural probing for linguistic knowledge.

(Anthropic, 2024) - sparse autoencoders extracting interpretable features from production-scale LMs.

foundation for the sparse-feature view of transformer representations.

(2025) - shows transcoders (reconstructing layer outputs from inputs) yield more interpretable features than SAEs; introduces skip transcoders.

Notable contributions include Human-Computer Cooperation or Word-by-Word Question Answering and modeling development of phonetic representations.

Notable projects include Avenue Project, a syntax driven machine translation system for endangered languages like Quechua and Aymara and previously, Noah's Ark which created AQMAR to improve NLP tools for Arabic.

Responsible for creating BOLT ( interactive error handling for speech translation systems) and an un-named project to characterize laughter in dialogue.

famous for creating the Penn Treebank and the Penn Discourse Treebank.

Notable contributions include a tool to reconstruct long dead languages, referenced here and by taking corpora from 637 languages currently spoken in Asia and the Pacific and recreating their descendant.

Recently in the news for developing speech recognition software to create a diagnostic test or Parkinson's Disease, here.

One of the top NLP research labs in the world, notable for creating Stanford CoreNLP and their coreference resolution system

retrieval-augmented LM for few-shot QA.

multi-paragraph reading comprehension.

and FiD - retrieve-then-read; the standard pre-LLM open-domain QA pipeline.

open-domain QA over Wikipedia.

general AI assistant benchmark including multi-step QA.

multi-hop reasoning.

real-user questions over Wikipedia.

(2023) - retrieval, generation, and self-critique.

extractive reading comprehension.

distantly-supervised QA.

Free online course covering text processing, large-scale data analysis, processing pipelines, and training neural network models for custom NLP tasks.

a16z AI playbook is a great link to forward to your managers or content for your presentations

collection of blog posts covering a wide array of NLP topics with detailed implementation

NLTK Tutorials, Jupyter notebooks

guide to managing larger linguistic annotation projects

Beginner-friendly tutorials including getting started guides, deep learning for NLP, and visual explanations of techniques like BERT, GloVe, and TF-IDF.

from Google's Senior Creative Engineer explains Machine Learning for engineer's and executives alike

by Hal Daumé III

An online and print book introducing NLP concepts using NLTK. The book's authors also wrote the NLTK library.

Oxford

Collection of Github notebooks

for commentary on the best of NLP research.

and The Illustrated Transformer

Hugging Face 🤗

foundational result; intermediate reasoning steps improve performance.

(2025) - open GRPO-based RL training system with four key improvements (decoupled clipping, dynamic sampling, token-level loss, entropy bonus); reproduces and surpasses DeepSeek-R1-Zero-level reasoning.

(2025) - open reasoning model trained with pure RL; replicated o1-style behavior in the open.

(2025) - long-context RL with policy optimization (no MCTS, no PRM) reaching o1-level performance; introduces long-CoT distillation into short-CoT models.

chain-of-thought for NLP reasoning tasks.

process-supervised reward models for reasoning.

(2024-2025) - test-time-compute reasoning systems.

(2025) - 1000+ controlled experiments on data recipes for open reasoning models; SOTA on AIME 2025 matching closed distillation baselines.

(2025) - small policy model paired with a process preference model trained via MCTS rollouts; enables small LMs to bootstrap reasoning without distilling from larger models.

(2025) - small open reasoning recipe via budget-forcing.

(2024) - systematic study of inference-time compute tradeoffs.

majority vote over sampled CoT chains.

and Reflexion - self-correction at inference time.

(2025) - generative process reward models that produce chain-of-thought verification per step, matching discriminative PRMs with 1% of the supervision labels.

search over reasoning trees.

(2025) - value-model-based RL with length-adaptive GAE and token-level clipping; surpasses value-free GRPO methods on AIME 2024 with stable training.

canonical archive of papers from ACL, EMNLP, NAACL, EACL, COLING, and related venues.

the rolling review process feeding ACL-affiliated venues.

tracks state-of-the-art results across common NLP tasks and datasets.

papers, benchmarks, and leaderboards for NLP tasks.

regular roundups of NLP research and trends.

long-form essays on ML and NLP research.

illustrated summaries of recent papers.

and BGE-M3 (2024) - multilingual, multi-functionality embeddings; top of MTEB across languages.

and ColBERTv2 - late-interaction retrieval; strong on out-of-domain.

(2026) - extends late-interaction retrieval by integrating query and document attention weights into ColBERT scoring; improves recall on MS-MARCO, BEIR, and LoTTE.

dual-encoder retrieval baseline.

and E5-Mistral - widely used dense embedding families.

(2025) - Gemini-derived dense embeddings; SOTA on MMTEB across 250+ languages and on cross-lingual retrieval (XOR-Retrieve, XTREME-UP).

(2024) - unified generation and embedding from one model.

nested embeddings supporting variable dimensionality at inference.

(2025) - community expansion of MTEB to 500+ tasks across 250+ languages.

(2024) - fully open, reproducible embedding model.

(2025) - decoder-based embedding series (0.6B-8B) built on Qwen3; #1 on MTEB Multilingual and MTEB Code, surpassing prior proprietary models.

the original retrieval-augmented framework; foundation for modern QA pipelines.

(2025) - first reranking model trained with test-time compute via DeepSeek-R1 reasoning-trace distillation; SOTA on instruction-following and OOD retrieval.

(2025) - embedding model for reasoning-intensive retrieval with ReMixer data synthesis and Redapter adaptive training; record nDCG@10 of 38.1 on BRIGHT.

generative AI across modalities.

general-purpose large language model resources.

production ML, including LLM serving.

prompting techniques and template libraries.

retrieval-augmented generation systems and tooling.

NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis

Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence parsing

Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and Traditional).

API and Github demo

SpaCy NLP models (custom and pre-trained ones) served through a RESTful API for named entity recognition (NER), POS tagging, and more.

High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis

Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more

Natural Language Interface for apps and devices

(NVIDIA, 2024) - top open multilingual ASR model.

and VieNeu-TTS - open TTS.

industrial-grade ASR toolkit; 170× realtime on GPU, 50+ languages, built-in VAD, punctuation, speaker diarization, and emotion detection. Includes non-autoregressive SenseVoice and LLM-based Fun-ASR-Nano models.

unified speech and text translation.

foundational self-supervised speech pretraining.

multilingual ASR; the modern open default.

widely used denoising seq2seq baseline.

LLMs vs fine-tuned summarizers.

and SCROLLS - long-document summarization benchmarks.

structured prompting for summarization.

gap-sentences pretraining for summarization.

foundational neural abstractive summarization.

extractive graph-based summarization.

(2025) - explicit reasoning improves fluency but hurts factual grounding; longer reasoning budgets can harm faithfulness.

fast few-shot for many-class settings.

strong, fast linear baseline.

using LLMs for text classification labeling, with caveats.

canonical fine-grained sentiment dataset.

few-shot text classification without prompts.

current encoder-fine-tuning baseline.

contextualized vectors learned from MT.

deep contextualized word representations.

implementation; subword n-grams handle OOV well, still useful for low-resource languages.

explainer blog

sentence representations from NLI.

word sense disambiguation.

language-model fine-tuning for text classification.

implementation - explainer blog

and Unigram LM - the two dominant subword schemes.

tokenizer-free byte-level model.

(Meta, 2024) - dynamic byte-level patching that matches BPE-tokenized models at scale; revives the tokenizer-free direction.

subword units for neural MT; foundation of modern tokenizers.

tokenization-free encoder operating on Unicode characters.

(ICLR 2025) - first formal unified framework for tokenizer models using stochastic-map category theory; establishes conditions for statistical consistency.

tokenizer fairness across languages.

unsupervised morphological segmentation.

(ICML 2025) - decouples input and output vocabularies; shows a log-linear relationship between input vocabulary size and training loss, scaling vocabulary independently of model size.

(2026) - post-hoc vocabulary additions that coalesce multi-token character sequences for low-resource languages, reducing inference cost without retraining.

language-agnostic subword tokenization.

(2025) - superword tokenization that improves on BPE for downstream tasks.

(2025) - quantifies how tokenization fertility predicts model accuracy across languages, exposing structural cost penalties for morphologically complex and low-resource languages.

tokenization, tagging, lemmatization, parsing for Universal Dependencies.

clustering-based topic modeling on top of contextual embeddings; common modern default.

hierarchical topic modeling with anchor words.

LDA, LSI, HDP in Python.

foundational topic model.

jointly learns topic and document vectors.

CS 685, UMass Amherst CS

Lecture series from IIT Madras taking from the basics all the way to autoencoders and everything. The github notebooks for this course are also available here

free course on LLMs, embeddings, semantic search, and NLP applications.

Richard Socher and Christopher Manning's Stanford Course

Lectures series from Oxford

by Yandex Data School, covering important ideas from text embedding to machine translation including sequence modeling, language models and so on.

4-course program covering sentiment analysis, word embeddings, RNNs, LSTMs, attention mechanisms, and Transformer models like BERT and T5 for tasks including machine translation and summarization.

This covers a blend of traditional NLP topics (including regex, SVD, naive bayes, tokenization) and recent neural network approaches (including RNNs, seq2seq, GRUs, and the Transformer), as well as addressing urgent ethical issues, such as bias and disinformation. Find the Jupyter Notebooks here

hands-on NLP with Transformers, Datasets, and Tokenizers libraries.

Lectures go from introduction to NLP and text processing to Recurrent Neural Networks and Transformers.

Carnegie Mellon Language Technology Institute there

Free beginner-friendly course covering NLP fundamentals through transformers, with Python/Jupyter notebooks.

seminar series with guest lectures from authors of recent transformer and NLP research.

end-to-end course on building language models, including data, tokenization, training, and evaluation.