# AI Video Scraping: Transforming Data Acquisition with Multimodal Models
In an ever-changing digital landscape, artificial intelligence (AI) persistently forges ahead, delivering groundbreaking solutions to common obstacles. One such innovation is **video scraping**, a method that utilizes AI to extract essential information from screen recordings. This technique, introduced by AI researcher Simon Willison, highlights the capabilities of multimodal AI models like Google’s **Gemini** in processing and evaluating video content, paving the way for new avenues in data extraction and engagement with digital contexts.
## What Is Video Scraping?
Video scraping entails capturing a screen recording and inputting it into an AI model to extract organized data from the footage. This strategy proves particularly beneficial when information is dispersed across various sources, such as emails, websites, or applications, where manual collection would consume considerable time.
Willison showcased this by creating a 35-second video of his emails, which featured scattered payment amounts and dates. Instead of entering the data manually, he utilized **Google’s AI Studio** to analyze the video using the **Gemini 1.5 Flash** model. The AI efficiently retrieved the pertinent data, which Willison subsequently formatted into a **JSON** (JavaScript Object Notation) file and later into a **CSV** (Comma-Separated Values) table for spreadsheet applications.
The entire process proved not only precise but also economical. Willison noted that the complete video analysis consumed merely 11,018 tokens, costing less than a tenth of a cent—a minute expense, especially given the time conserved. Currently, Google AI Studio remains available for free usage in certain contexts, making it an inviting resource for experimentation.
## The Strength of Multimodal AI Models
Video scraping is facilitated by advancements in **multimodal AI models**, which can handle diverse forms of input, including text, images, audio, and video. These models, such as **Google’s Gemini** and **OpenAI’s GPT-4o**, decompose multimedia inputs into **tokens**—small data segments utilized by the AI to predict subsequent token sequences. This functionality empowers AI to understand and engage with intricate data formats, like video, in ways previously deemed impossible.
Multimodal models signify a considerable advancement over traditional **Large Language Models (LLMs)**, which primarily focus on textual content. As AI progresses, the designation **Token Prediction Model (TPM)** may become more fitting for these systems, reflecting their expanded capabilities beyond just language inputs.
## Overcoming Data Obstacles
For data journalists like Willison, video scraping presents a robust tool for transforming **unstructured data** into **structured data**. This is particularly advantageous when conventional data extraction approaches, such as web scraping, face challenges due to complex or inaccessible formats. By recording a video of displayed data on a screen, users can overcome these hurdles and input the visual information directly into an AI model for examination.
In a past experiment, Willison employed video scraping to catalog the titles of books on his bookshelf. He filmed a brief video of the shelves and asked **Gemini 1.5 Pro** to retrieve the book titles, organizing them into a structured format. This example illustrates the adaptability of video scraping, which can be utilized across various data extraction endeavors.
## Video as a Novel Input Method
The ramifications of video scraping extend beyond the realm of data journalism. As AI models grow increasingly proficient in processing video, they could aid users in navigating intricate digital environments. For instance, an AI might observe a user struggling with a poorly designed website and intervene to execute necessary actions, such as completing an order or submitting a form.
Leading tech firms are already investigating these possibilities. **Google**, **OpenAI**, and **Microsoft** have all showcased prototypes allowing AI to “perceive” and interact with activities on a user’s screen. **Microsoft’s Copilot Vision**, for example, embodies a concept that permits AI to monitor a user’s screen and assist with tasks in real-time. Likewise, **OpenAI’s ChatGPT Mac App** has hinted at a feature enabling the AI to engage with on-screen content, although it has yet to be made available to the public.
## Privacy Challenges and Ethical Considerations
While video scraping unveils thrilling opportunities, it also prompts considerable privacy concerns. The capacity for AI to “see” and scrutinize everything displayed on a user’s screen could be misused for nefarious aims, such as surveillance or unauthorized data gathering. Applications like **Rewind AI** and **Microsoft’s Recall** already implement similar methods to record and archive all user actions on their computers, igniting discussions about privacy and data safety.
Willison’s methodology regarding video scraping, however, prioritizes user autonomy. By selectively recording and submitting specific videos for analysis, users maintain control over the information they share with the AI. This approach contrasts with continuous recording applications, which
