Sequence Models provides a technical and practical understanding of neural network architectures designed to process sequential data, such as text, speech, audio, and time-dependent information. As the fifth and final course in the Deep Learning Specialization, it progresses from recurrent neural networks through word embeddings and attention mechanisms to the Transformer architecture, providing an important foundation for understanding modern Natural Language Processing and Large Language Models.

The course begins with Recurrent Neural Networks (RNNs) and how information can be propagated through sequences. It introduces forward propagation and backpropagation through time, together with challenges such as vanishing gradients. More advanced recurrent architectures, including Gated Recurrent Units (GRUs), Long Short-Term Memory networks (LSTMs), bidirectional RNNs, and deep RNNs, demonstrate how neural networks can retain and use information across longer sequences.

The course then applies sequence models to Natural Language Processing (NLP). Word representations and embeddings provide numerical representations of semantic relationships between words, with techniques such as Word2Vec and GloVe demonstrating how these representations can be learned and applied to language-processing tasks.

The course progresses to sequence-to-sequence models and attention mechanisms, enabling models to selectively focus on relevant parts of an input sequence when generating an output. These concepts are applied to neural machine translation and speech recognition and establish the foundation for moving from recurrent processing toward attention-based architectures.

The final module introduces the Transformer architecture, including self-attention and multi-head attention. Transformers replace recurrent processing with attention mechanisms that can model relationships across an entire sequence and enable greater parallelization during training. These principles form an important technical foundation for contemporary Large Language Models and Generative AI.

Overall, the course demonstrates the evolution of sequence modelling from RNNs and LSTMs through attention mechanisms to Transformers. It combines the theoretical foundations of sequential neural networks with practical implementations for language modelling, NLP, machine translation, speech processing, and Transformer-based applications.

Course modules

Recurrent Neural Networks

The first module introduces neural networks specifically designed for sequential information. It covers RNN architectures, backpropagation through time, language modelling, and sequence generation, together with the problem of vanishing gradients. More advanced architectures including GRUs, LSTMs, bidirectional RNNs, and deep RNNs demonstrate different approaches for maintaining and processing information over longer sequences.

Natural Language Processing & Word Embeddings

The second module focuses on representing language numerically so that neural networks can process semantic relationships between words. It introduces word embeddings, embedding matrices, Word2Vec, negative sampling, and GloVe. The module also explores sentiment classification and examines how undesirable biases encoded in learned word representations can be identified and reduced.

Sequence Models & Attention Mechanism

The third module develops sequence-to-sequence architectures for applications such as machine translation and speech recognition. It introduces beam search and the BLEU score for generating and evaluating sequences before moving to the attention mechanism, which allows a neural network to dynamically determine which parts of an input sequence are most relevant when producing each part of an output. Speech recognition and trigger-word detection extend these techniques to audio data.

Transformer Network

The final module introduces the Transformer architecture and the transition from recurrent to attention-based sequence processing. It develops the concepts of self-attention and multi-head attention and combines them into a complete Transformer network. Practical applications demonstrate how Transformers can be used for tasks such as Named Entity Recognition and Question Answering, providing a direct connection to the architectures underlying contemporary LLMs.

Topics and techniques

• Recurrent Neural Networks (RNNs) – modelling and generating sequential information.
• GRUs and LSTMs – maintaining information across longer sequences.
• Word Embeddings – Word2Vec, GloVe, semantic representations, and debiasing.
• Sequence-to-Sequence Models – transforming input sequences into output sequences.
• Attention Mechanisms – dynamically focusing on relevant parts of an input.
• Natural Language Processing – language modelling, sentiment analysis, translation, and NER.
• Transformer Networks – self-attention and multi-head attention.
• Speech and audio processing – speech recognition and trigger-word detection.

Practical exercises

The practical exercises use Python, TensorFlow, and Jupyter notebooks to progressively implement sequence models. Initial assignments build an RNN from first principles, followed by character-level language modelling to generate dinosaur names and an LSTM-based jazz improvisation model.

Exercises with word vectors explore semantic relationships and bias and apply learned representations in an Emojify application. The attention module develops a Neural Machine Translation system and a trigger-word detection system, demonstrating how sequence models can process both text and audio.

The final programming assignment constructs a Transformer architecture with TensorFlow, including self-attention and multi-head attention. Additional labs apply Transformer models to preprocessing, Named Entity Recognition, and Question Answering.

Together, the exercises provide a practical progression from implementing recurrent networks and word representations to attention mechanisms and Transformer architectures, illustrating the evolution of the neural network techniques that underpin modern language AI.