Introduction

In this technical workshop, you will learn how to use the latest AI techniques to analyze image and video material. You will work with vision transformers, multimodal AI models and powerful LLM integrations to extract insights from visual data. In addition, you'll discover how to fine-tune existing models on your own data to achieve better performance in specific domains.

We combine frameworks such as PyTorch, Hugging Face Transformers, OpenAI Whisper, and LLMs such as GPT-4, Gemini and Claude to automatically transform images and videos into structured and actionable insights.

What you will learn during this training

1. Vision Transformers for advanced image recognition.

You will work with the latest generation of deep learning models for image analysis: Vision Transformers (ViTs).

• Application of pretrained ViTs such as ViT-B/16, DINOv2 and SAM (Segment Anything Model).
• Fine-tuning vision transformers on domain-specific data (e.g., medical images, satellite imagery, product recognition)
• Object classification, segmentation and image description with transformers
• Comparing ViTs to traditional CNN approaches.

2. Multimodal AI: combining image, text and audio

Discover how multimodal models combine image and video with language to generate contextually richer insights.

• Using models such as CLIP, Flamingo and Gemini to link image information to text
• Building visual question-answer systems: "What is happening in this image?"
• Text generation based on visual input (image captioning and narrative generation)
• Automatic tagging of videos or images based on content

3. Video analysis and transcription with Whisper and LLM integration.

You will learn how to use AI to convert videos to text, and how to use LLMs for deeper interpretation.

• Automatically transcribe video files with Whisper
• Speaker recognition, timestamping and structuring of video content
• Use of LLMs (such as GPT or Claude) to analyze, summarize and tag transcripts
• Detect themes, sentiment and actions from video reviews or events

4. Fine-tuning LLMs and visual AI models on own data.

You will learn how to adapt existing foundation models for your specific use case or domain.

• Fine-tuning LLMs with instructional data for task-oriented output (e.g., legal, medical or technical contexts)
• Fine-tuning vision transformers with small data sets via transfer learning
• Prompt engineering versus model training: when do you use what?
• Use of tools such as LoRA, PEFT and Hugging Face Accelerate for efficient fine-tuning

Hands-on projects

During this workshop, you will work on a complete AI pipeline: from visual input to textual insights. You will use real datasets or your own material to build concrete applications.

Practical exercises and experiments

Vision Transformers in action:

• You apply a ViT to an image classification or segmentation task
• You fine-tune an existing vision model on a proprietary dataset

Multimodal analysis with CLIP and Gemini:

• You generate textual descriptions of images and link them to labels
• You build a mini-VQA (Visual Question Answering) prototype.

Video to text with Whisper + LLM:

• You transcribe a video to text with Whisper
• You let GPT or Gemini automatically generate a summary, tags or questions

Fine-tuning and personalization:

• You prepare your own dataset for fine-tuning an LLM or vision model
• You test the performance of the trained model on a specific task

Approach and working form

This workshop is intensive, technical and hands-on. You will work with modern open-source tools and frameworks, and have the space to conduct experiments on your own dataset or with sample material. The session is interactive and focused on building, testing and optimizing models.

For whom

This course is designed for AI engineers, ML specialists, data scientists and developers with experience in Python and machine learning. Ideal for those who want to work with cutting-edge techniques for visual and multimodal AI and adapt their own models to specific applications.

Interested in this training?

Please feel free to contact us. We are happy to think with you about a custom interpretation for your team or organization.