feat: LLM image input support #1106

aludwiko · 2025-10-31T15:51:06Z

refs: https://github.com/lightbend/akka-runtime/issues/4421

samples/helloworld-agent/src/main/java/com/example/application/ImageOutRawResponseAgent.java

samples/helloworld-agent/src/main/java/com/example/application/ImageOutBase64ResponseAgent.java

aludwiko · 2025-11-05T14:02:10Z

akka-javasdk/src/main/java/akka/javasdk/agent/SessionMemory.java

+   */
+  void addInteraction(
+      String sessionId,
+      SessionMessage.CompoundUserMessage userMessage,


Initially, I used CompoundUserMessage for all interactions, text-only and multimodal, but I think most of the time, the user message will be text-based so it's worth optimizing it and supporting both options. Especially that we need to be backward compatible with events anyway.

aludwiko · 2025-11-05T14:03:06Z

akka-javasdk/src/main/java/akka/javasdk/agent/SessionMemoryEntity.java

  }

+  // keeping UserMessage instead of CompoundUserMessage for compaction
  public record CompactionCmd(UserMessage userMessage, AiMessage aiMessage, long sequenceNumber) {}


I think that supporting only text-based UserMessage for compaction makes more sense than CompoundUserMessage.

aludwiko · 2025-11-05T14:04:50Z

akka-javasdk/src/main/java/akka/javasdk/agent/SessionMessage.java

+
+    record TextMessageContent(String text) implements MessageContent {}
+
+    record ImageUriMessageContent(


Yet another representation of the same thing, but this time for persistence, so String instead of URI, maybe I should also have dedicated DetailLevel, currently I'm reusing it.

aludwiko · 2025-11-05T14:06:58Z

samples/doc-snippets/src/main/java/com/example/application/CompactionAgent.java

+            .stream()
+            .map(content ->
+              switch (content) {
+                case SessionMessage.MessageContent.ImageUriMessageContent __ -> "";


Should this be sth else than ignoring?

could be text?

image from {url}

aludwiko · 2025-11-05T14:12:21Z

Ok, SDK is ready for the first round. SessionMemoryEntity deserves more attention. My next steps are to test it and add some documentation for the new feature.

aludwiko · 2025-11-06T10:13:30Z

akka-javasdk/src/main/java/akka/javasdk/agent/MessageContent.java

+   * @param uri The URI pointing to the image
+   * @param detailLevel The level of detail for image processing
+   */
+  record ImageUriMessageContent(URI uri, ImageMessageContent.DetailLevel detailLevel)


The issue with URI is that it can be hacked to send base64 bytes in the form of data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..., which then raises the "large payloads" issues again. Perhaps the public API should only allow URLs for now?

Or maybe URL in the runtime SPI as well 🤔

oh, indeed it should be something that is retrieved from the URL/URI

Changed to URL, but only on the public SDK level. I could change the SPI as well, unless we think that it's better to keep it more flexible. On the other hand, supporting URIs with base64 makes less sense than creating a dedicated ImageBytesMessageContent and being more explicit. Is there a place for URIs for some custom protocol that we could later use to fetch/store blobs?

patriknw

looking good

akka-javasdk-testkit/src/main/java/akka/javasdk/testkit/TestModelProvider.java

patriknw · 2025-11-06T12:55:42Z

akka-javasdk-testkit/src/main/java/akka/javasdk/testkit/TestModelProvider.java

+                                            imageContent.image().url().toURL(),
+                                            toDetailLevel(imageContent.detailLevel()));
+                                      } catch (MalformedURLException e) {
+                                        throw new RuntimeException("Can't transform to URL", e);


include the url in the error message?

patriknw · 2025-11-06T12:58:05Z

...javasdk-tests/src/test/java/akkajavasdk/components/agent/SomeMultiModalUserMessageAgent.java

+        .userMessage(
+            UserMessage.from(
+                MessageContent.TextMessageContent.from("testing"),
+                MessageContent.ImageMessageContent.from("https://example.com")))


should it be ImageMessageContent.fromUrl to make it more clear?
we might have fromBytes and fromBase64 later?

Ok, changed others as well to keep it consistent.

patriknw · 2025-11-06T13:08:40Z

samples/doc-snippets/src/main/java/com/example/application/CompactionAgent.java

+            .stream()
+            .map(content ->
+              switch (content) {
+                case SessionMessage.MessageContent.ImageUriMessageContent __ -> "";


could be text?

image from {url}

samples/helloworld-agent/pom.xml

samples/helloworld-agent/src/main/resources/application.conf

more examples separation of multimodal response types updating changes from the runtime supporting only images in the user message instance of docs SDK IT test docs URI -> URL addressing PR comments reverting version

aludwiko · 2025-11-07T09:43:59Z

akka-javasdk/src/main/scala/akka/javasdk/impl/GrpcEndpointDescriptorFactory.scala

-        unwrapFailedCompletion(),
-        system,
-        materializer)
+      handlerFactory.partialInstancePerRequest(serviceFactory, description.name, unwrapFailedCompletion(), system)


not related, but required after dependencies bump

patriknw

LGTM

github-actions bot added documentation documentation related java-sdk labels Oct 31, 2025

aludwiko commented Nov 3, 2025

View reviewed changes

samples/helloworld-agent/src/main/java/com/example/application/ImageOutRawResponseAgent.java Outdated Show resolved Hide resolved

patriknw reviewed Nov 4, 2025

View reviewed changes

samples/helloworld-agent/src/main/java/com/example/application/ImageOutBase64ResponseAgent.java Outdated Show resolved Hide resolved

aludwiko commented Nov 5, 2025

View reviewed changes

aludwiko changed the title ~~feat: LLM image input/output support~~ feat: LLM image input support Nov 5, 2025

aludwiko commented Nov 6, 2025

View reviewed changes

patriknw reviewed Nov 6, 2025

View reviewed changes

aludwiko added 2 commits November 7, 2025 10:35

feat: LLM image input/output support

35e088d

more examples separation of multimodal response types updating changes from the runtime supporting only images in the user message instance of docs SDK IT test docs URI -> URL addressing PR comments reverting version

fixes after merge

2b321f8

aludwiko force-pushed the 4421-support-llm-image-inputoutput branch from 6eae418 to 2b321f8 Compare November 7, 2025 09:43

aludwiko commented Nov 7, 2025

View reviewed changes

aludwiko and others added 2 commits November 7, 2025 11:32

renames + session memory tests

028d985

Update Dependencies.scala

9c0cff0

patriknw approved these changes Nov 20, 2025

View reviewed changes


		record TextMessageContent(String text) implements MessageContent {}

		record ImageUriMessageContent(

feat: LLM image input support #1106

Are you sure you want to change the base?

feat: LLM image input support #1106

Uh oh!

Conversation

aludwiko commented Oct 31, 2025

Uh oh!

Uh oh!

Uh oh!

aludwiko Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aludwiko Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aludwiko commented Nov 5, 2025

Uh oh!

aludwiko Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patriknw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patriknw left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aludwiko Nov 5, 2025 •

edited

Loading

aludwiko Nov 5, 2025 •

edited

Loading

aludwiko Nov 6, 2025 •

edited

Loading