Best in Category: Expressing
Interaction Awards 2023
Advances in computer vision and natural language AI continue to unlock new ways to explore the billions of images across our devices, websites, and the public domain. However, it remains difficult to express subjective concepts, such as visual tones (‘vibrant’) or moods (‘eclectic’) to current systems, where visual interpretation is shaped by culture and community and is personal to the individual.
CAVstudio is a mood-board tool for training AI models on visually subjective concepts using imagery. Drag-and-drop images that represent your concept, and the AI reflects its understanding through a curated selection of sample images, arranged and cropped to best highlight the visual qualities it thinks you are trying to express. It’s powered by a new technology called CAVs (Concept Activation Vectors) which reduces required training data from 1000s of images to only a handful, and makes training fast. What normally takes hours can be achieved in seconds – removing barriers to entry and making AI training accessible to a broader audience.
Concepts can then be used to search an image bank, surfacing other images that represent the concept, or live in a camera, to see the world through the eyes of somebody else. In this way, CAVstudio empowers anyone working with visual imagery to train and collaborate with AI systems according to their own personal viewpoint of the world, irrespective of language, geography or technical expertise. A stepping stone to more equitable AI.
As users of modern software products, we are all increasingly subjected to decisions made by AI systems. Machine Learning models are expensive to train in data, time and cost. As such, they’re mostly produced in small quantities by technology companies, often resulting in homogenised models with a biased, western-centric world view. Although great for objective tasks like recognising a giraffe, they fail to understand subjectivity, where visual interpretation is shaped by culture and community, and is personal to the individual.
What is visual subjectivity? When describing a Turner painting as ‘atmospheric’, or the work of Yinka Ilori as ‘graphical joy’, these are subjective concepts that humans appreciate but AI models can not. For AI to be genuinely useful at a personal level, we need new forms of expressive controls that make teaching AI subjective concepts accessible to all.
CAVstudio is a mood-board tool for training AI models on visually subjective concepts using imagery. Drag-and-drop images that represent your concept, and the AI reflects its understanding through a curated selection of sample images, arranged and cropped to best highlight the visual qualities it thinks you are trying to express. It’s powered by a new technology called CAVs (Concept Activation Vectors) which reduces required training data from 1000s of images to only a handful, and makes training fast. What normally takes hours can be achieved in seconds – removing barriers to entry and making AI training accessible to a broader audience.
The tool was co-designed and iteratively prototyped with creative communities. Intuitive, expressive controls expose the inner workings of the AI model – from toggling between neural network layers, to up/down weighting biases in the model. Inspection tools like heatmap and focus modes let the user see the visual qualities the AI is drawn to. By directly leveraging the speed and power of AIs, we inspire new discoveries and alternate perspectives for people working with imagery.
“Using Alex’s concept on my own photos let me see the world through a different lens – escaping the ordinary, initiating more creativity.” – Rachel Maggart, Curator.
“As I used the features of the tool, looking at exactly where it cropped and focussed, I began to enjoy the visual relationships and felt that the essence of materiality was being activated effectively in the results.” – Preservation Assistant, British Museum
CAVstudio is open source, letting anyone take their personalised AI and incorporate it into their own product: from searching large image sets for hidden meaning and connections, to embedding AIs directly in cameras to shoot alongside you, to discovering new creative concepts in your own photo library.
In this way, CAVstudio empowers anyone working with visual imagery to train and collaborate with AI systems, irrespective of language, geography or technical expertise. By combining the technological leap of CAVs with inclusive design principles, we expose a paradigm shift in AI Interaction Design: away from jargonised text prompts and black boxes, towards apparent interfaces with affordances for deeper, more meaningful human-machine collaboration and discovery. A stepping stone to more equitable AI.
Collaboration with Google.
