Large Multimodal Model Prompting with Gemini provides a practical introduction to multimodal generative AI and how Large Multimodal Models can understand and reason across different types of information. Using Google’s Gemini model family, the course extends conventional text-based prompt engineering to workflows that combine text, images, and video within the same model interaction.

A central concept is multimodal prompting, where information from several modalities is included within a prompt and interpreted jointly by the model. Rather than treating images, text, and video as separate inputs processed by specialized systems, Gemini can use cross-modal attention to identify relationships between them and combine these representations when generating a response.

The course explores how the structure of a multimodal prompt influences model behaviour. Techniques such as role assignment, task decomposition, prompt formatting, and ordering of text and visual information are used to make instructions more explicit and improve the quality and consistency of generated results. Model parameters including temperature, top-p, and top-k are introduced as mechanisms for controlling the balance between deterministic and more creative generation.

These principles are applied to image and video understanding. Image-based examples demonstrate how Gemini can analyze several images simultaneously, identify relationships between visual elements, extract structured information, and combine visual observations with textual instructions. Video examples extend these principles to temporal information, demonstrating semantic search, question answering, and summarization across longer video sequences.

The final part extends the model beyond the information supplied directly in its context through function calling. Gemini can determine when an external function or API is required, provide the appropriate arguments, receive external information, and incorporate the result into its response. This connects multimodal reasoning with external data and software capabilities and provides a foundation for more advanced AI applications and agentic workflows.

Overall, the course provides an applied understanding of how multimodal models extend LLM-based applications beyond text. It combines prompt engineering, multimodal reasoning, parameter control, image and video understanding, and external tools to demonstrate how richer AI applications can be designed around models capable of interpreting several information modalities simultaneously.

Course topics

Introduction to Gemini Models

The first section introduces the Gemini model family and the capabilities of multimodal foundation models. It examines different Gemini variants and how models can be selected according to application requirements such as capability, latency, and computational cost. The section establishes the distinction between conventional text-based LLM interaction and models capable of jointly processing multiple modalities.

Multimodal Prompting and Parameter Control

This section introduces prompting with combinations of text, images, and video. It examines how multimodal prompts are structured and how model parameters such as temperature, top-p, and top-k influence the generated response. These controls demonstrate how model behaviour can be adjusted between more predictable and more diverse or creative output.

Best Practices for Multimodal Prompting

The course develops practical techniques for constructing more effective multimodal prompts. Role assignment, explicit instructions, task decomposition, output formatting, and prompt-image ordering demonstrate how prompt structure influences the model’s interpretation of both textual and visual information.

Creating Use Cases with Images

This section applies multimodal prompting to image understanding and cross-modal reasoning. Examples include an interior-design assistant that combines images of a room with textual preferences and a receipt-processing application that extracts and structures information from visual input. The examples demonstrate how Gemini can reason across several images and relate visual observations to textual instructions.

Developing Use Cases with Videos

Multimodal reasoning is extended from static images to video understanding. The course explores semantic search within video, including a “needle in the haystack” approach for locating specific information in long sequences. It also demonstrates video question answering and summarization, taking advantage of Gemini’s large context window to analyze information distributed over time.

Integrating Real-Time Data with Function Calling

The final technical section introduces function calling as a mechanism for connecting Gemini with external APIs and live information. The model can interpret a user’s request, identify an appropriate external function, construct its parameters, and incorporate the returned information into its response. This extends the model from multimodal interpretation toward AI applications that combine reasoning with external actions and dynamically retrieved data.

Topics and techniques

• Large Multimodal Models – combining text, images, and video within a common AI model.
• Multimodal Prompting – structuring prompts across multiple information modalities.
• Cross-Modal Reasoning – identifying relationships between textual, visual, and video information.
• Parameter Control – temperature, top-p, and top-k for controlling model behaviour.
• Prompt Engineering – role assignment, task decomposition, formatting, and input ordering.
• Image Understanding – analysis, comparison, and structured information extraction from images.
• Video Understanding – semantic search, question answering, and summarization.

Function Calling – connecting the model with APIs, external functions, and real-time information.

Practical exercises

The practical work uses Python, Jupyter notebooks, Gemini, and Google Cloud to experiment with multimodal prompts and model behaviour. The notebooks allow different combinations of text, images, and video to be supplied to the model while parameters and prompt structures are varied to examine their effect on generated results.

Image-based exercises apply Gemini to practical scenarios such as interior design and receipt itemization, demonstrating how visual information can be combined with textual instructions and how relationships can be analyzed across multiple images. Video exercises use Gemini’s large context capabilities for semantic video search, question answering, and content summarization.

The final exercises integrate Gemini with external information through function calling and APIs, demonstrating how natural-language requests can be translated into structured function calls and how retrieved real-time information can be incorporated into the model’s response.

Together, the exercises demonstrate a progression from basic multimodal prompting and parameter control to complete applications that combine text, images, video, reasoning, and external data sources.