Spring AI with Multimodality and Images
This article will teach you how to create a Spring Boot application that handles images and text using the Spring AI multimodality feature. Multimodality is the ability to understand and process information from different sources simultaneously. It covers text, images, audio, and other data formats. We will perform simple experiments with multimodality and images. This is the fourth part of my series of articles about Spring Boot and AI. It is worth reading the following posts before proceeding with the current one:
- https://piotrminkowski.com/2025/01/28/getting-started-with-spring-ai-and-chat-model: The first tutorial introduces the Spring AI project and its support for building applications based on chat models like OpenAI or Mistral AI.
- https://piotrminkowski.com/2025/01/30/getting-started-with-spring-ai-function-calling: The second tutorial shows Spring AI support for Java function calling with the OpenAI chat model.
- https://piotrminkowski.com/2025/02/24/using-rag-and-vector-store-with-spring-ai: The third tutorial shows Spring AI support for RAG (Retrieval Augmented Generation) and vector store.
Source Code
Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.
Motivation for Multimodality with Spring AI
The multimodal large language model (LLM) capabilities allow it to process and generate text alongside other modalities, including images, audio, and video. This feature covers a use case when we want LLM to detect something specific inside an image or describe its content. Let’s assume we have a list of input images. We want to find the image in that list that matches our description. For example, this description can ask a model to find the image that contains a specified item. The Spring AI Message API provides all the necessary elements to support multimodal LLMs. Here’s a diagram that illustrates our scenario.
Use Multimodality with Spring AI
We don’t need to include any specific library other than the Spring AI starter for a particular AI model. The default option is spring-ai-openai-spring-boot-starter. Our application uses images stored in the src/main/resources/images directory. Spring AI multimodality support requires the image to be passed inside the Media object. We load all the pictures from the classpath inside the constructor.
Recognize Items in the Image
The GET /images/find/{object} tries to find the image that contains the item determined by the object path variable. AI model must return a position on the image in the input list. To achieve that, we create an UserMessage object that contains a user query and a list of the Media objects. Once the model returns the position, the endpoint reads the image from the list and returns its content in the image/png format.
@RestController
@RequestMapping("/images")
public class ImageController {
private final static Logger LOG = LoggerFactory
.getLogger(ImageController.class);
private final ChatClient chatClient;
private List<Media> images;
private List<Media> dynamicImages = new ArrayList<>();
public ImageController(ChatClient.Builder chatClientBuilder) {
this.chatClient = chatClientBuilder
.defaultAdvisors(new SimpleLoggerAdvisor())
.build();
this.images = List.of(
Media.builder().id("fruits").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/fruits.png")).build(),
Media.builder().id("fruits-2").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/fruits-2.png")).build(),
Media.builder().id("fruits-3").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/fruits-3.png")).build(),
Media.builder().id("fruits-4").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/fruits-4.png")).build(),
Media.builder().id("fruits-5").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/fruits-5.png")).build(),
Media.builder().id("animals").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/animals.png")).build(),
Media.builder().id("animals-2").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/animals-2.png")).build(),
Media.builder().id("animals-3").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/animals-3.png")).build(),
Media.builder().id("animals-4").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/animals-4.png")).build(),
Media.builder().id("animals-5").mimeType(MimeTypeUtils.IMAGE_PNG).data(new ClassPathResource("images/animals-5.png")).build()
);
}
@GetMapping(value = "/find/{object}", produces = MediaType.IMAGE_PNG_VALUE)
@ResponseBody byte[] analyze(@PathVariable String object) {
String msg = """
Which picture contains %s.
Return only a single picture.
Return only the number that indicates its position in the media list.
""".formatted(object);
LOG.info(msg);
UserMessage um = new UserMessage(msg, images);
String content = this.chatClient.prompt(new Prompt(um))
.call()
.content();
assert content != null;
return images.get(Integer.parseInt(content)-1).getDataAsByteArray();
}
}Let’s make a test call. We will look for the picture containing a banana. Here’s the AI model response after calling the http://localhost:8080/images/find/banana. You can try to make other test calls and find an image with e.g. an orange or a tomato.
Describe Image Contents
On the other hand, we can ask the AI model to generate a short description of all images included as the Media content. The GET /images/describe endpoint merges two lists of images.
@GetMapping("/describe")
String[] describe() {
UserMessage um = new UserMessage("Explain what do you see on each image.",
List.copyOf(Stream.concat(images.stream(), dynamicImages.stream()).toList()));
return this.chatClient.prompt(new Prompt(um))
.call()
.entity(String[].class);
}Once we call the http://localhost:8080/images/describe URL we will receive a compact description of all input images. The two highlighted descriptions have been generated for images from the dynamicImages List. These images were generated by the AI image model. We will discuss this in the next section.
Generate Images with AI Model
To generate an image using AI API we must inject the ImageModel bean. It provides a single call method that allows us to communicate with AI Models dedicated to image generation. This method takes the ImagePrompt object as an argument. Typically, we use the ImagePrompt constructor that takes instructions for image generation and options that customize the height, width, and number of images. We will generate a single (N=1) image with 1024 pixels in height and width. The AI model returns the image URL (responseFormat). Once the image is generated, we create an UrlResource object, create the Media object, and put it into the dynamicImages List. The GET /images/generate/{object} endpoint returns a byte array representation of the image object.
@RestController
@RequestMapping("/images")
public class ImageController {
private final ChatClient chatClient;
private final ImageModel imageModel;
private List<Media> images;
private List<Media> dynamicImages = new ArrayList<>();
public ImageController(ChatClient.Builder chatClientBuilder,
ImageModel imageModel) {
this.chatClient = chatClientBuilder
.defaultAdvisors(new SimpleLoggerAdvisor())
.build();
this.imageModel = imageModel;
// other initializations
}
@GetMapping(value = "/generate/{object}", produces = MediaType.IMAGE_PNG_VALUE)
byte[] generate(@PathVariable String object) throws IOException {
ImageResponse ir = imageModel.call(new ImagePrompt("Generate an image with " + object, ImageOptionsBuilder.builder()
.height(1024)
.width(1024)
.N(1)
.responseFormat("url")
.build()));
UrlResource url = new UrlResource(ir.getResult().getOutput().getUrl());
LOG.info("Generated URL: {}", ir.getResult().getOutput().getUrl());
dynamicImages.add(Media.builder()
.id(UUID.randomUUID().toString())
.mimeType(MimeTypeUtils.IMAGE_PNG)
.data(url)
.build());
return url.getContentAsByteArray();
}
}Do you remember the description of that image returned by the GET /images/describe endpoint? Here’s our image with strawberry generated by the AI model after calling the http://localhost:8080/images/generate/strawberry URL.
Here’s a similar test for the banana input parameter.
Use Vector Store with Spring AI Multimodality
Let’s consider how we can leverage vector store in our scenario. We cannot insert image representation directly to a vector store since most popular vendors like OpenAI or Mistral AI do not provide image embedding models. We could integrate directly with a model like clip-vit-base-patch32 to generate image embeddings, but this article won’t cover such a scenario. Instead, a vector store may contain an image description and its location (or name). The GET /images/load endpoint provides a method for loading image descriptions into a vector store. It uses Spring AI multimodality support to generate a compact description of each image in the input list and then puts it into the store.
@GetMapping("/load")
void load() throws JsonProcessingException {
String msg = """
Explain what do you see on the image.
Generate a compact description that explains only what is visible.
""";
for (Media image : images) {
UserMessage um = new UserMessage(msg, image);
String content = this.chatClient.prompt(new Prompt(um))
.call()
.content();
var doc = Document.builder()
.id(image.getId())
.text(mapper.writeValueAsString(new ImageDescription(image.getId(), content)))
.build();
store.add(List.of(doc));
LOG.info("Document added: {}", image.getId());
}
}Finally, we can implement another endpoint that generates a new image and asks the AI model to generate an image description. Then, it performs a similarity search in a vector store to find the most similar image based on its text description.
@GetMapping("/generate-and-match/{object}")
List<Document> generateAndMatch(@PathVariable String object) throws IOException {
ImageResponse ir = imageModel.call(new ImagePrompt("Generate an image with " + object, ImageOptionsBuilder.builder()
.height(1024)
.width(1024)
.N(1)
.responseFormat("url")
.build()));
UrlResource url = new UrlResource(ir.getResult().getOutput().getUrl());
LOG.info("URL: {}", ir.getResult().getOutput().getUrl());
String msg = """
Explain what do you see on the image.
Generate a compact description that explains only what is visible.
""";
UserMessage um = new UserMessage(msg, new Media(MimeTypeUtils.IMAGE_PNG, url));
String content = this.chatClient.prompt(new Prompt(um))
.call()
.content();
SearchRequest searchRequest = SearchRequest.builder()
.query("Find the most similar description to this: " + content)
.topK(2)
.build();
return store.similaritySearch(searchRequest);
}Let’s test the GET /images/generate-and-match/{object} endpoint using the pineapple parameter. It returns the description of the fruits.png image from the classpath.
By the way, here’s the fruits.png image located in the /src/main/resources/images directory.
Final Thoughts
Spring AI provides multimodality and image generation support. All the features presented in this article work fine with OpenAI. It supports both the image model and multimodality. To read more about the support offered by other models, refer to the Spring AI chat and image model docs.
This article shows how we can use Spring AI and AI models to interact with images in various ways.
Leave a Reply