[JS] VLMPipeline implementation #3112

Retribution98 · 2025-12-12T14:13:29Z

Description

This PR introduces comprehensive support for the JavaScript bindings in the VLMPipeline.

VLMPipeline is implemented similarly to LLMPipeline, but with some improvements:

Error handling in vLmPerformInferenceThread was modified for a much better user experience. It no longer throws fatal errors. Now, the thread uses a callback to reject the promise and throw a catchable exception to the user. If that is not possible, an error message is logged.
Streamer and callback were separated to create clearer code. These functions are used in different cases and have different semantics. A callback is always used to resolve a promise with the final result or reject it with an error. The streamer is only used if it is provided by the user to get chunks.
A wrapper property, is_initializing, was added to prevent the pipeline from being loaded incorrectly twice.
A wrapper property, is_generating, was added to prevent the incorrect attempt to call generate in parallel. This leads to an InferRequest exception, which may confuse the user.
VLMPerfMetrics should extend PerfMetrics. This was refactored to avoid code duplication. The Node addon API doesn't support inheritance from the box, so a workaround was used to resolve this issue.
The optional parameters for generate and stream were grouped into the options argument. This hook allows the use of named arguments and simplifies calls.

CVS-172789

Checklist:

Tests have been updated or added to cover the new code.
This patch fully addresses the ticket.
I have made corresponding changes to the documentation. https://retribution98.github.io/openvino.genai/

Signed-off-by: Kirill Suvorov <[email protected]>

Copilot

Pull request overview

This PR introduces comprehensive JavaScript bindings support for the VLMPipeline (Visual Language Model Pipeline) with several improvements to error handling, code structure, and API design.

Key Changes:

Implemented VLMPipeline similar to LLMPipeline with enhanced error handling that uses callbacks to reject promises instead of fatal errors
Separated streamer and callback logic for clearer semantics and different use cases
Added wrapper properties is_initializing and is_generating to prevent incorrect concurrent operations
Refactored performance metrics to use inheritance through a base template class to avoid code duplication

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/js/tests/vlmPipeline.test.js	Comprehensive test suite for VLMPipeline functionality including text generation, streaming, image/video handling, and chat operations
src/js/tests/utils.js	Helper functions to create test image and video tensors for VLM testing
src/js/tests/models.js	Added VLLM model reference for testing
src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp	Core VLMPipeline implementation with improved error handling and async generation
src/js/src/vlm_pipeline/start_chat_worker.cpp	AsyncWorker implementation for starting chat sessions
src/js/src/vlm_pipeline/perf_metrics.cpp	VLM-specific performance metrics wrapper extending base metrics
src/js/src/vlm_pipeline/init_worker.cpp	AsyncWorker for pipeline initialization with concurrent init prevention
src/js/src/vlm_pipeline/finish_chat_worker.cpp	AsyncWorker for finishing chat sessions
src/js/src/perf_metrics.cpp	Refactored to use base template class for code reuse
src/js/src/llm_pipeline/llm_pipeline_wrapper.cpp	Updated to use shared helper function for decoded results
src/js/src/helper.cpp	Added tensor conversion helpers and result conversion utilities
src/js/src/addon.cpp	Registered VLMPipeline and VLMPerfMetrics classes
src/js/lib/utils.ts	Added VLMPipelineProperties type definition
src/js/lib/pipelines/vlmPipeline.ts	TypeScript VLMPipeline class with generate, stream, and chat methods
src/js/lib/pipelines/llmPipeline.ts	Moved DecodedResults to separate module for reuse
src/js/lib/perfMetrics.ts	Type definitions for performance metrics including VLM-specific metrics
src/js/lib/index.ts	Exported VLMPipeline factory and related types
src/js/lib/decodedResults.ts	Shared DecodedResults classes for LLM and VLM pipelines
src/js/lib/addon.ts	TypeScript interface definitions for VLMPipeline addon
src/js/include/vlm_pipeline/vlm_pipeline_wrapper.hpp	Header for VLMPipelineWrapper class
src/js/include/vlm_pipeline/start_chat_worker.hpp	Header for VLMStartChatWorker AsyncWorker
src/js/include/vlm_pipeline/perf_metrics.hpp	Header for VLMPerfMetricsWrapper
src/js/include/vlm_pipeline/init_worker.hpp	Header for VLMInitWorker AsyncWorker
src/js/include/vlm_pipeline/finish_chat_worker.hpp	Header for VLMFinishChatWorker AsyncWorker
src/js/include/perf_metrics.hpp	Updated to use base template class
src/js/include/helper.hpp	Added function declarations for tensor and result conversion
src/js/include/base/perf_metrics.hpp	Base template class for performance metrics wrappers using CRTP pattern
src/js/include/addon.hpp	Added VLM pipeline and metrics references to AddonData

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/tests/vlmPipeline.test.js

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

src/js/src/helper.cpp

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

src/js/tests/vlmPipeline.test.js

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/src/helper.cpp

Copilot · 2025-12-15T08:09:28Z

src/js/include/vlm_pipeline/init_worker.hpp

+                  std::shared_ptr<ov::genai::VLMPipeline>& pipe,
+                  std::shared_ptr<bool> is_initializing,
+                  const std::string model_path,
+                  std::string device,


The device parameter should be passed by const reference to avoid unnecessary string copying in the constructor.

Suggested change

std::string device,

const std::string& device,

src/js/include/helper.hpp

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/src/helper.cpp

src/js/tests/vlmPipeline.test.js

site/docs/use-cases/image-processing/_sections/_run_model/_code_example_js.mdx

src/js/lib/decodedResults.ts

src/js/lib/perfMetrics.ts

src/js/lib/pipelines/vlmPipeline.ts

src/js/tests/models.js

src/js/tests/vlmPipeline.test.js

src/js/tests/utils.js

src/js/tests/vlmPipeline.test.js

site/docs/use-cases/image-processing/_sections/_usage_options/index.mdx

Co-authored-by: Yaroslav Tarkan <[email protected]>

Copilot

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/lib/decodedResults.ts

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

site/docs/use-cases/image-processing/_sections/_run_model/_code_example_js.mdx

Copilot

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/tests/vlmPipeline.test.js

Copilot · 2025-12-18T12:11:10Z

src/js/lib/decodedResults.ts

+      return this.texts[0];
+    }
+    const lines = this.scores.map((score, i) => `${score.toFixed(6)}: ${this.texts[i]}`);
+    return lines.join("\n");


The toString() implementation differs from the original in llmPipeline.ts. The original handled the last element separately to avoid a trailing newline, but this version always adds newlines between all elements. While functionally similar, this creates an inconsistency in output format compared to the original implementation.

Suggested change

return lines.join("\n");

let result = "";

for (let i = 0; i < lines.length; i++) {

result += lines[i];

if (i !== lines.length - 1) {

result += "\n";

}

}

return result;

almilosz · 2025-12-18T12:04:01Z

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

+    return env.Undefined();
+}
+
+Napi::Value VLMPipelineWrapper::start_chat(const Napi::CallbackInfo& info) {


Suggested change

Napi::Value VLMPipelineWrapper::start_chat(const Napi::CallbackInfo& info) {

void VLMPipelineWrapper::start_chat(const Napi::CallbackInfo& info) {

almilosz · 2025-12-18T12:55:53Z

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

+    return env.Undefined();
+}
+
+Napi::Value VLMPipelineWrapper::set_chat_template(const Napi::CallbackInfo& info) {


Please remove return value from void functions

Suggested change

Napi::Value VLMPipelineWrapper::set_chat_template(const Napi::CallbackInfo& info) {

void VLMPipelineWrapper::set_chat_template(const Napi::CallbackInfo& info) {

almilosz · 2025-12-18T14:05:48Z

src/js/src/helper.cpp

+    obj.Set("texts", cpp_to_js<std::vector<std::string>, Napi::Value>(env, results.texts));
+    obj.Set("scores", cpp_to_js<std::vector<float>, Napi::Value>(env, results.scores));
+    obj.Set("perfMetrics", PerfMetricsWrapper::wrap(env, results.perf_metrics));
+    obj.Set("subword", Napi::String::New(env, results));


Please check if subword is used in context of DecodedResults
I think it can be a leftover to keep backward compatibility.

We will remove it when LLMPipeline is refactored

almilosz

LGTM, a few comments

almilosz · 2025-12-18T14:57:00Z

src/js/src/vlm_pipeline/init_worker.cpp

+};
+
+void VLMInitWorker::OnOK() {
+    *this->is_initializing = false;


is this variable needed also for LLMPipeline? If yes please add it in separate PR

almilosz · 2025-12-18T15:02:51Z

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

+                        InstanceMethod("setGenerationConfig", &VLMPipelineWrapper::set_generation_config)});
+}
+
+Napi::Value VLMPipelineWrapper::init(const Napi::CallbackInfo& info) {


Suggested change

Napi::Value VLMPipelineWrapper::init(const Napi::CallbackInfo& info) {

void VLMPipelineWrapper::init(const Napi::CallbackInfo& info) {

VALIDATE_ARGS_COUNT now returns Env().Undefined(). It doesn't allow it to be done now. :(
Let's fix it later.

almilosz · 2025-12-18T15:14:29Z

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

+                                          [context, this](Napi::Env) {  // Finalizer used to clean threads up
+                                              context->native_thread.join();
+                                              delete context;
+                                          });


This Finalizer is repeated for each pipeline. We can extract it like here and reuse it: https://github.com/openvinotoolkit/openvino/blob/805d41a1c7ae209619651e57863d961ee32c65c2/src/bindings/js/node/src/infer_request.cpp#L197

almilosz · 2025-12-18T15:46:11Z

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp

+        } else {
+            // If no exceptions from streamer, call the final callback with the result
+            napi_status status =
+                context->callback.BlockingCall([result, &report_error](Napi::Env env, Napi::Function jsCallback) {


here result is passed by value. Such copy may affect performance. Maybe use shared_ptr or capture?

src/js/lib/utils.ts

[JS] VLMPipeline implementation

30bf26f

Signed-off-by: Kirill Suvorov <[email protected]>

Copilot AI review requested due to automatic review settings December 12, 2025 14:13

github-actions bot added the category: JS API GenAI JS API label Dec 12, 2025

Copilot AI reviewed Dec 12, 2025

View reviewed changes

Lint fixes

a819027

Retribution98 requested a review from Copilot December 12, 2025 14:41

Copilot AI reviewed Dec 12, 2025

View reviewed changes

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp Show resolved Hide resolved

src/js/tests/vlmPipeline.test.js Show resolved Hide resolved

src/js/tests/vlmPipeline.test.js Show resolved Hide resolved

Retribution98 requested review from almilosz and yatarkan December 12, 2025 14:44

github-actions bot added the category: GHA CI based on Github actions label Dec 15, 2025

Update Node.js tests timeout for mac workflow

5ca69c7

Retribution98 force-pushed the js_vlmpipeline branch from 400a8c4 to 5ca69c7 Compare December 15, 2025 08:06

Copilot AI review requested due to automatic review settings December 15, 2025 08:06

Copilot AI reviewed Dec 15, 2025

View reviewed changes

Skip VLM tests for mac workflow

cfc0102

github-actions bot removed the category: GHA CI based on Github actions label Dec 15, 2025

Update helpers

b84a1c8

Copilot AI review requested due to automatic review settings December 15, 2025 12:46

Fix lint

340ed96

Copilot AI reviewed Dec 15, 2025

View reviewed changes

src/js/src/helper.cpp Show resolved Hide resolved

src/js/tests/vlmPipeline.test.js Show resolved Hide resolved

src/js/tests/vlmPipeline.test.js Show resolved Hide resolved

Update docs

643ee72

github-actions bot added the category: GH Pages Docs Github Pages documentation label Dec 15, 2025

yatarkan reviewed Dec 16, 2025

View reviewed changes

site/docs/use-cases/image-processing/_sections/_usage_options/index.mdx Outdated Show resolved Hide resolved

Apply suggestions from code review

566d2de

Co-authored-by: Yaroslav Tarkan <[email protected]>

Copilot AI review requested due to automatic review settings December 17, 2025 10:16

Copilot AI reviewed Dec 17, 2025

View reviewed changes

src/js/lib/decodedResults.ts Outdated Show resolved Hide resolved

src/js/src/vlm_pipeline/vlm_pipeline_wrapper.cpp Show resolved Hide resolved

Update after code review

4f15802

yatarkan reviewed Dec 18, 2025

View reviewed changes

site/docs/use-cases/image-processing/_sections/_run_model/_code_example_js.mdx Outdated Show resolved Hide resolved

site/docs/use-cases/image-processing/_sections/_run_model/_code_example_js.mdx Show resolved Hide resolved

Remove unrequired console.log

b104f1b

Copilot AI review requested due to automatic review settings December 18, 2025 12:08

Copilot AI reviewed Dec 18, 2025

View reviewed changes

Fix comments in tests

0b5a83c

yatarkan approved these changes Dec 18, 2025

View reviewed changes

yatarkan enabled auto-merge December 18, 2025 12:46

Retribution98 disabled auto-merge December 18, 2025 12:56

almilosz reviewed Dec 18, 2025

View reviewed changes

Retribution98 added this pull request to the merge queue Dec 19, 2025

Merged via the queue into openvinotoolkit:master with commit 109ee68 Dec 19, 2025
97 checks passed

-    return lines.join("\n");
+    let result = "";
+    for (let i = 0; i < lines.length; i++) {
+      result += lines[i];
+      if (i !== lines.length - 1) {
+        result += "\n";
+      }
+    }
+    return result;

	Napi::Value VLMPipelineWrapper::start_chat(const Napi::CallbackInfo& info) {
	void VLMPipelineWrapper::start_chat(const Napi::CallbackInfo& info) {

	Napi::Value VLMPipelineWrapper::set_chat_template(const Napi::CallbackInfo& info) {
	void VLMPipelineWrapper::set_chat_template(const Napi::CallbackInfo& info) {

	Napi::Value VLMPipelineWrapper::init(const Napi::CallbackInfo& info) {
	void VLMPipelineWrapper::init(const Napi::CallbackInfo& info) {

[JS] VLMPipeline implementation #3112

[JS] VLMPipeline implementation #3112

Uh oh!

Conversation

Retribution98 commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Retribution98 commented Dec 12, 2025 •

edited

Loading

almilosz Dec 18, 2025 •

edited

Loading