Multimodal, Retrieval & Automation
Go beyond text — images, video, audio, speech, and language — then extend models with your own data and automate real work.
By the end of this session you can
- Generate and edit an image, then transcribe and summarise a clip.
- Translate and culturally localise a message, then verify its quality.
- Explain how RAG lets a model answer from your documents, in plain terms.
- Map a real repetitive task into an agent or automation flow.
- State a personal rule for the ethics of face-swap and synthetic media.
Topic 1
Image, video, and audio models
The generative modalities beyond text, and when each is the right tool.
- Image: Midjourney, Stable Diffusion, Flux, Nano Banana / Imagen — text-to-image and image editing.
- Video: Sora, Veo, Runway, Kling, Luma — text-to-video and image-to-video.
- Audio: text-to-speech, voice cloning, and music (Suno, Udio).
Topic 2
Speech-to-text
Turning audio into text you can search, summarise, and caption.
- Whisper and Deepgram for transcription.
- Live transcription and captioning workflows.
Topic 3
Multimodal and multi-language AI
One model that sees, hears, reads, and speaks — across languages.
- Why multimodal is the direction of travel.
- Translation, cross-lingual prompting, and cultural localisation.
- Verifying quality when you don't speak the target language.
Topic 4
The face-swap and synthetic-media ethics line
How face-swap and talking-avatar tools work — and where consent draws the line.
- Creative and business uses versus the deepfake harm line.
- Disclosure norms and a personal rule you can defend.
Topic 5
Retrieval-Augmented Generation (RAG)
Giving a model your own documents so it answers from your data, not just its training.
- Knowledge bases and vector databases in plain terms.
- Embeddings and semantic search — searching by meaning, not keywords.
- Fine-tuning vs. RAG vs. prompting — which to reach for.
Topic 6
Agents, tool use, and workflow automation
How agents call tools and take real actions, and how to automate repetitive work.
- Tool use: browsing, running code, and taking actions on your behalf.
- Automation with Zapier, Make, and n8n.
- Model Context Protocol (MCP) and connectors — the emerging plug-in standard.
Practice
Today is heavy on doing. Work through the lab and the automation design sprint; the knowledge-base homework and the ethics essay are due Friday.
Multimodal + a mini RAG base
- 1Generate an image, then edit it with a follow-up instruction (multi-turn).
- 2Transcribe a short audio/video clip and summarise it.
- 3Translate and culturally localise a message, then verify quality.
- 4Build a mini RAG knowledge base from personal notes/PDFs and chat with it.
Automation design sprint
- 1Each group maps one real repetitive task into an agent/automation flow.
- 2Pitch the flow to the class.
Personal knowledge base
- 1Build a small knowledge base of 5–10 documents.
- 2Record 3 questions it answered that a plain chatbot could not.
The ethical line for synthetic media
- 1Where is the ethical line for face-swap, voice cloning, and synthetic media?
- 2What rule would you personally follow?