Generative AI with Large Language Models provides a technical understanding of how Large Language Models are developed, adapted, evaluated, and integrated into generative AI applications. Developed by DeepLearning.AI in collaboration with AWS, the course covers the complete lifecycle of an LLM-based application, from model architecture and pre-training to fine-tuning, alignment, optimization, and deployment.

The course begins with the foundations of generative AI and the Transformer architecture, explaining how attention mechanisms enable modern language models to process context and generate text. It examines the relationship between model size, training data, and computational resources, including scaling laws, compute-optimal models, and the challenges involved in pre-training large models. Prompt engineering and generation parameters are introduced as methods for controlling model behaviour without modifying the underlying model.

A major part of the course focuses on adapting pre-trained models to specific tasks. Instruction fine-tuning is introduced for improving a model’s ability to follow instructions, together with single-task and multi-task fine-tuning. The course then examines Parameter-Efficient Fine-Tuning (PEFT) techniques such as LoRA and soft prompts, which make it possible to adapt large models while training substantially fewer parameters. Model evaluation and benchmarks are used to assess the performance of the resulting models.

The course continues with Reinforcement Learning from Human Feedback (RLHF) as a technique for aligning model behaviour with human preferences. This includes collecting human feedback, training reward models, reinforcement learning, Proximal Policy Optimization (PPO), reward hacking, and techniques for scaling human feedback. Together, these methods demonstrate how a general pre-trained model can progressively be transformed into a model better suited for human interaction and specific application requirements.

The final part extends the model into complete LLM-powered applications. Techniques such as Chain-of-Thought prompting, Program-Aided Language Models (PAL), and ReAct demonstrate different approaches for combining language-model generation with reasoning, planning, external information, and software actions. The course concludes by examining application architectures, model optimization for deployment, and responsible AI.

Overall, the course provides an end-to-end understanding of the technical lifecycle behind modern generative AI systems. It connects the foundations of transformer-based models with practical methods for adapting, aligning, evaluating, optimizing, and integrating LLMs into applications.

Topics and techniques

• Transformers and attention – architecture and mechanisms underlying modern LLMs.
• Pre-training and scaling – training data, computational requirements, scaling laws, and domain adaptation.
• Prompt engineering – controlling model output through prompts and generation parameters.
• Instruction fine-tuning – adapting pre-trained models for specific and multiple tasks.
• PEFT, LoRA, and soft prompts – parameter-efficient approaches to model adaptation.
• Model evaluation – benchmarks and techniques for assessing LLM performance.
• RLHF – aligning models with human preferences using reward models and reinforcement learning.
• Reasoning and action – Chain-of-Thought, PAL, and ReAct.
• LLM application architectures – integrating models with external applications and deploying generative AI systems.
• Responsible AI – considering reliability, alignment, risks, and appropriate use of generative AI.

Practical exercises

The practical exercises use Python and AWS environments to apply the techniques to a dialogue summarization use case. The first lab uses a pre-trained model for dialogue summarization and explores how prompting influences its output.

The second lab progresses from prompting to fine-tuning, using a generative AI model adapted specifically for dialogue summarization and demonstrating how instruction fine-tuning and parameter-efficient techniques can improve task performance.

The final lab applies reinforcement learning and RLHF concepts to FLAN-T5, fine-tuning the model to generate summaries with more positive sentiment. Together, the exercises demonstrate the progression from using a pre-trained LLM, through task-specific adaptation, to aligning model behaviour using reinforcement learning.