GenerativeAIExamples

Форк
0

README.md

NVIDIA Generative AI Examples

documentation

Introduction

State-of-the-art Generative AI examples that are easy to deploy, test, and extend. All examples run on the high performance NVIDIA CUDA-X software stack and NVIDIA GPUs.

NVIDIA NGC

Generative AI Examples can use models and GPUs from the NVIDIA NGC: AI Development Catalog.

Sign up for a free NGC developer account to access:

  • GPU-optimized containers used in these examples
  • Release notes and developer documentation

Retrieval Augmented Generation (RAG)

A RAG pipeline embeds multimodal data -- such as documents, images, and video -- into a database connected to a LLM. RAG lets users chat with their data!

Developer RAG Examples

The developer RAG examples run on a single VM. The examples demonstrate how to combine NVIDIA GPU acceleration with popular LLM programming frameworks using NVIDIA's open source connectors. The examples are easy to deploy with Docker Compose.

Examples support local and remote inference endpoints. If you have a GPU, you can inference locally with TensorRT-LLM. If you don't have a GPU, you can inference and embed remotely with NVIDIA API Catalog endpoints.

ModelEmbeddingFrameworkDescriptionMulti-GPUTRT-LLMNVIDIA EndpointsTritonVector Database
mixtral_8x7bai-embed-qa-4LangChainNVIDIA API Catalog endpoints chat bot [code, docs]NoNoYesYesMilvus or pgvector
llama-2UAE-Large-V1LlamaIndexCanonical QA Chatbot [code, docs]YesYesNoYesMilvus or pgvector
llama-2all-MiniLM-L6-v2LlamaIndexChat bot, GeForce, Windows [repo]NoYesNoNoFAISS
llama-2ai-embed-qa-4LangChainChat bot with query decomposition agent [code, docs]NoNoYesYesMilvus or pgvector
mixtral_8x7bai-embed-qa-4LangChainMinimilastic example: RAG with NVIDIA AI Foundation Models [code, README]NoNoYesYesFAISS
mixtral_8x7b
Deplot
Neva-22b
ai-embed-qa-4CustomChat bot with multimodal data [code, docs]NoNoYesNoMilvus or pvgector
llama-2UAE-Large-V1LlamaIndexChat bot with quantized LLM model [docs]YesYesNoYesMilvus or pgvector
llama3-70bnonePandasAIChat bot with structured data [code, docs]NoNoYesNonone
llama-2ai-embed-qa-4LangChainChat bot with multi-turn conversation [code, docs]NoNoYesNoMilvus or pgvector

Enterprise RAG Examples

The enterprise RAG examples run as microservices distributed across multiple VMs and GPUs. These examples show how to orchestrate RAG pipelines with Kubernetes and deployed with Helm.

Enterprise RAG examples include a Kubernetes operator for LLM lifecycle management. It is compatible with the NVIDIA GPU operator that automates GPU discovery and lifecycle management in a Kubernetes cluster.

Enterprise RAG examples also support local and remote inference with TensorRT-LLM and NVIDIA API Catalog endpoints.

ModelEmbeddingFrameworkDescriptionMulti-GPUMulti-nodeTRT-LLMNVIDIA EndpointsTritonVector Database
llama-2NV-Embed-QALlamaIndexChat bot, Kubernetes deployment [README]NoNoYesNoYesMilvus

Generative AI Model Examples

The generative AI model examples include end-to-end steps for pre-training, customizing, aligning and running inference on state-of-the-art generative AI models leveraging the NVIDIA NeMo Framework

ModelResources(s)FrameworkDescription
gemmaDocs, LoRA, SFTNeMoAligning and customizing Gemma, and exporting to TensorRT-LLM format for inference
codegemmaDocs, LoRANeMoCustomizing Codegemma, and exporting to TensorRT-LLM format for inference
starcoder-2LoRA, InferenceNeMoCustomizing Starcoder-2 with NeMo Framework, optimizing with NVIDIA TensorRT-LLM, and deploying with NVIDIA Triton Inference Server
small language models (SLMs)Docs, Pre-training and SFT, EvalNeMoTraining, alignment, and running evaluation on SLMs using various techniques

Tools

Example tools and tutorials to enhance LLM development and productivity when using NVIDIA RAG pipelines.

NameDescriptionNVIDIA Endpoints
EvaluationRAG evaluation using synthetic data generation and LLM-as-a-judge [code, docs]Yes
ObservabilityMonitoring and debugging RAG pipelines [code, docs]Yes

Open Source Integrations

These are open source connectors for NVIDIA-hosted and self-hosted API endpoints. These open source connectors are maintained and tested by NVIDIA engineers.

NameFrameworkChatText EmbeddingPythonDescription
NVIDIA AI Foundation EndpointsLangchainYesYesYesEasy access to NVIDIA hosted models. Supports chat, embedding, code generation, steerLM, multimodal, and RAG.
NVIDIA Triton + TensorRT-LLMLangchainYesYesYesThis connector allows Langchain to remotely interact with a Triton inference server over GRPC or HTTP for optimized LLM inference.
NVIDIA Triton Inference ServerLlamaIndexYesYesNoTriton inference server provides API access to hosted LLM models over gRPC.
NVIDIA TensorRT-LLMLlamaIndexYesYesNoTensorRT-LLM provides a Python API to build TensorRT engines with state-of-the-art optimizations for LLM inference on NVIDIA GPUs.

Support, Feedback, and Contributing

We're posting these examples on GitHub to support the NVIDIA LLM community and facilitate feedback. We invite contributions via GitHub Issues or pull requests!

Known Issues

  • Some known issues are identified as TODOs in the Python code.
  • The datasets provided as part of this project are under a different license for research and evaluation purposes.
  • This project downloads and installs third-party open source software projects. Review the license terms of these open source projects before use.

Описание

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Языки

Python

  • Smarty
  • Jupyter Notebook
  • HTML
  • JavaScript
  • Shell
  • Go
  • Dockerfile
  • CSS
  • Jinja
  • Makefile
Сообщить о нарушении

Использование cookies

Мы используем файлы cookie в соответствии с Политикой конфиденциальности и Политикой использования cookies.

Нажимая кнопку «Принимаю», Вы даете АО «СберТех» согласие на обработку Ваших персональных данных в целях совершенствования нашего веб-сайта и Сервиса GitVerse, а также повышения удобства их использования.

Запретить использование cookies Вы можете самостоятельно в настройках Вашего браузера.