feat: add streamedListObjects for unlimited object retrieval

[daniel-jonathan]

Summary

Adds streamedListObjects method for retrieving unlimited objects via the streaming API. Node.js-only implementation with NDJSON parsing, error handling, and automatic resource cleanup.

Requires OpenFGA server v1.2.0+

Fixes #236

Changes

• Add streaming.ts with NDJSON parser for Node.js streams
• Add StreamedListObjectsResponse interface to apiModel.ts
• Add OpenFgaClient.streamedListObjects() async generator method
• Add createStreamingRequestFunction to common.ts for streaming requests
• Export parseNDJSONStream from index.ts
• Add streaming tests and examples
• Update CHANGELOG with feature and version requirement

Implementation

• Streams beyond 1000-object limit
• Proper error propagation through async iterators
• Automatic stream cleanup on early exit
• Supports Readable, AsyncIterable, string, Buffer, and Uint8Array inputs
• Telemetry integration via API layer

Usage

const fgaClient = new OpenFgaClient({ 
  apiUrl: "http://localhost:8080", 
  storeId: "01H0H015178Y2V4CX10C2KGHF4" 
});

for await (const response of fgaClient.streamedListObjects({
  user: "user:anne",
  relation: "owner",
  type: "document"
})) {
  console.log(response.object);
}

Testing

All 153 tests passing (10/10 suites)

• 17 streaming tests covering parsing, errors, cleanup, edge cases
• 90.8% coverage on streaming.ts
• Live verified: 3-object and 2000-object streaming

Related

• Fixes [Feature] Support for StreamedListObjects Endpoint #236
• Parent issue: [All SDKs] Support streaming endpoints (StreamedListObjects) sdk-generator#76
• Related PR: feat: add streamedListObjects for unlimited object retrieval sdk-generator#654

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This PR adds a new streaming variant of the ListObjects API to the JavaScript SDK. It includes a new streamedListObjects method that streams NDJSON-formatted results from the server, allowing retrieval of more than 1000 objects. The implementation spans the API layer, client wrapper, streaming parser, and includes comprehensive examples and tests.

Changes

Cohort / File(s)	Summary
Core API Layer src/api.ts	Adds streamedListObjects method across all API interfaces (AxiosParamCreator, Fp, Factory, and OpenFgaApi class) to support streaming requests via createStreamingRequestFunction.
API Model Types src/apiModel.ts	Introduces new StreamedListObjectsResponse interface with object property for streamed response elements.
Client Wrapper src/client.ts	Adds streamedListObjects method returning AsyncGenerator<StreamedListObjectsResponse> that parses NDJSON stream and yields results.
Streaming Support src/common.ts, src/streaming.ts, src/index.ts	Implements createStreamingRequestFunction for axios streaming with telemetry; adds parseNDJSONStream utility to parse line-delimited JSON from Node.js Readable streams; exports parseNDJSONStream publicly.
Test Mocks & Coverage tests/helpers/nocks.ts, tests/streaming.test.ts	Adds streamedListObjects mock helper for NDJSON endpoint responses; comprehensive test suite for NDJSON parser covering chunked data, various input types, and error handling.
Examples example/streamed-list-objects/README.md, example/streamed-list-objects/model.json, example/streamed-list-objects/streamedListObjects.mjs	Complete example demonstrating streaming 2000 tuples, comparing streamed vs. standard endpoints, with access control model definition.
Local Example example/streamed-list-objects-local/README.md, example/streamed-list-objects-local/streamedListObjectsLocal.mjs	Simplified local example showing streamedListObjects with user-relation queries and store lifecycle.
Documentation CHANGELOG.md	Documents new streamedListObjects feature for Node.js environments with NDJSON parsing support.

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant Client as OpenFgaClient
    participant API as OpenFgaApi
    participant HTTP as axios
    participant Server as OpenFGA Server
    participant Parser as parseNDJSONStream

    User->>Client: streamedListObjects(request)
    Client->>API: streamedListObjects()
    API->>HTTP: GET /stores/{id}/streamed-list-objects<br/>(responseType: stream, telemetry)
    HTTP->>Server: Request with streaming
    Server-->>HTTP: NDJSON stream (line-delimited JSON)
    HTTP-->>API: Response stream
    API-->>Client: Response.data (stream)
    Client->>Parser: parseNDJSONStream(stream)
    
    loop For each line in stream
        Parser->>Parser: UTF-8 decode, buffer, split on \\n
        Parser->>Parser: Parse JSON object
        Parser-->>Client: yield StreamedListObjectsResponse
    end
    
    Client-->>User: AsyncGenerator<StreamedListObjectsResponse>
    User->>User: Iterate and process results

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~35 minutes

• src/streaming.ts: The NDJSON parser handles multiple input modalities (Readable, Buffer, string, async iterable) with UTF-8 decoding, line buffering, and error handling. Verify edge cases around partial UTF-8 sequences, decoder state management, and listener cleanup.
• src/common.ts: New createStreamingRequestFunction mirrors existing request flow but with streaming response. Verify telemetry capture (fromRequest/fromResponse) and histogram emission work correctly with streaming.
• tests/streaming.test.ts: Extensive test suite validates chunked data, error scenarios, and various input types. Spot-check for completeness of edge cases (e.g., incomplete UTF-8 at buffer boundary).
• Example files: Ensure model.json schema is valid and examples execute correctly with 2000 tuples to verify streaming advantage over standard pagination.

Possibly related issues

• [All SDKs] Support streaming endpoints (StreamedListObjects) sdk-generator#76: Directly addresses the "Support streaming endpoints (StreamedListObjects)" work item with implementation of streaming API, NDJSON parser, and async generator interface.

Suggested reviewers

• jimmyjames

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The pull request title "feat: add streamedListObjects for unlimited object retrieval" clearly and directly describes the primary change in the changeset. The title references the new streamedListObjects feature and explains its key benefit (unlimited object retrieval beyond the standard 1000-object limit). The title is specific, concise, and uses appropriate conventional prefix (feat:) without vague language or excessive noise. A developer reviewing the commit history would immediately understand that this PR adds streaming capability to retrieve objects without the standard limitation.
Linked Issues Check	✅ Passed	The pull request successfully implements both primary requirements from linked issue #236. First, SDK methods to call the StreamedListObjects API have been added across all layers: the api.ts exposes the streamedListObjects method through OpenFgaApi, OpenFgaApiFp, OpenFgaApiFactory, and OpenFgaApiAxiosParamCreator; the client.ts provides OpenFgaClient.streamedListObjects() as an AsyncGenerator interface for consumers. Second, the implementation aligns with the Python SDK's async-generator pattern as noted in the PR objectives. The code includes supporting infrastructure (parseNDJSONStream for NDJSON parsing, createStreamingRequestFunction for streaming requests, StreamedListObjectsResponse model), comprehensive examples demonstrating usage, and test coverage validating the streaming behavior.
Out of Scope Changes Check	✅ Passed	All changes in the pull request are directly scoped to the objectives of adding StreamedListObjects support. The modifications span necessary layers: API contract changes (api.ts), client-facing implementation (client.ts), data models (apiModel.ts), streaming utilities (streaming.ts), request infrastructure (common.ts), public exports (index.ts), test infrastructure (tests/helpers/nocks.ts, tests/streaming.test.ts), usage examples (example/ directory), and documentation (CHANGELOG.md). No unrelated bug fixes, general refactoring, or tangential features are present. The examples and supporting files are integral to demonstrating the feature rather than out-of-scope additions.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 90.54054% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.56%. Comparing base (847b9ff) to head (d8ffd07).

Files with missing lines	Patch %	Lines
streaming.ts	90.21%	7 Missing and 2 partials ⚠️
common.ts	86.36%	1 Missing and 2 partials ⚠️
api.ts	90.90%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #280      +/-   ##
==========================================
+ Coverage   89.45%   89.56%   +0.11%     
==========================================
  Files          24       25       +1     
  Lines        1299     1447     +148     
  Branches      234      266      +32     
==========================================
+ Hits         1162     1296     +134     
- Misses         81       91      +10     
- Partials       56       60       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (10)

apiModel.ts (1)

864-876: Clarify docstring: SDK-yielded type vs wire shape.

The client parses NDJSON lines like { result: { object } } and yields { object }. Consider adjusting the comment to “Element yielded by streamedListObjects” to avoid implying this is the on-the-wire shape. Otherwise the interface looks good.

example/streamed-list-objects-local/README.md (1)

5-7: Align Node version and add browser note.

• Change “Node.js 18+” to match SDK minimum (e.g., “Node.js ≥16.15.0”, unless 18+ is explicitly required here).
• Add: “Not supported in browsers; uses Node Readable streams.”

Please confirm whether any example dependency requires Node 18+ specifically (e.g., APIs not in Node 16.15).

common.ts (1)

353-418: Harden streaming request: headers default, NDJSON Accept, cancellation, telemetry parity.

• Ensure headers exists before auth header injection.
• Set Accept: application/x-ndjson if not provided.
• Allow AbortSignal passthrough (so callers can cancel).
• Optionally record request-body attributes (parity with non-stream paths, if/when added there).

Apply this diff:

 export const createStreamingRequestFunction = function (axiosArgs: RequestArgs, axiosInstance: AxiosInstance, configuration: Configuration, credentials: Credentials, methodAttributes: Record<string, string | number> = {}) {
   configuration.isValid();
@@
-  return async (axios: AxiosInstance = axiosInstance): Promise<any> => {
-    await setBearerAuthToObject(axiosArgs.options.headers, credentials!);
+  return async (axios: AxiosInstance = axiosInstance): Promise<any> => {
+    // ensure headers object
+    axiosArgs.options.headers = axiosArgs.options.headers || {};
+    await setBearerAuthToObject(axiosArgs.options.headers, credentials!);
@@
-    const axiosRequestArgs = { ...axiosArgs.options, responseType: "stream", url: url };
+    // default NDJSON accept unless caller overrides
+    if (!axiosArgs.options.headers["Accept"]) {
+      axiosArgs.options.headers["Accept"] = "application/x-ndjson";
+    }
+    const axiosRequestArgs = { ...axiosArgs.options, responseType: "stream", url };
@@
-    attributes = TelemetryAttributes.fromResponse({
+    attributes = TelemetryAttributes.fromResponse({
       response,
       attributes,
     });

Optional: If you add signal?: AbortSignal to client options, pass it via axiosArgs.options.signal.

Please confirm the server’s preferred Accept header for streamed list objects (e.g., application/x-ndjson). If different, we should set it accordingly.

tests/helpers/nocks.ts (1)

249-264: Simulate chunked NDJSON to better exercise the parser.

Current mock emits a single large chunk. Emit smaller chunks to surface boundary/chunking bugs in tests.

-    return nock(basePath)
-      .post(`/stores/${storeId}/streamed-list-objects`)
-      .reply(200, () => Readable.from([ndjsonResponse]), {
-        "Content-Type": "application/x-ndjson"
-      });
+    return nock(basePath)
+      .post(`/stores/${storeId}/streamed-list-objects`)
+      .reply(200, () => {
+        // send ~32-byte chunks to simulate real streaming
+        const chunks = Array.from(ndjsonResponse.matchAll(/.{1,32}/gs), m => m[0]);
+        return Readable.from(chunks);
+      }, {
+        "Content-Type": "application/x-ndjson"
+      });

example/streamed-list-objects-local/streamedListObjectsLocal.mjs (1)

55-57: Ensure store cleanup in a finally block.

If an earlier step throws, the store may be orphaned. Move deleteStore into finally and guard on storeId.

Example:

let storeId;
try {
  // ... create store, set storeId, do work ...
} finally {
  if (storeId) {
    await new OpenFgaClient(new ClientConfiguration({ apiUrl, storeId })).deleteStore().catch(() => {});
  }
}

api.ts (2)

387-425: Be explicit about NDJSON in request headers.

Setting Accept helps interoperability and test clarity.

-      const localVarHeaderParameter = {} as any;
+      const localVarHeaderParameter = {} as any;
+      localVarHeaderParameter["Accept"] = "application/x-ndjson";

---

955-970: Align telemetry with other methods.

Include store id and body-derived attributes for consistency.

-      return createStreamingRequestFunction(localVarAxiosArgs, globalAxios, configuration, credentials, {
-        [TelemetryAttribute.FgaClientRequestMethod]: "StreamedListObjects"
-      });
+      return createStreamingRequestFunction(localVarAxiosArgs, globalAxios, configuration, credentials, {
+        [TelemetryAttribute.FgaClientRequestMethod]: "StreamedListObjects",
+        [TelemetryAttribute.FgaClientRequestStoreId]: storeId ?? "",
+        ...TelemetryAttributes.fromRequestBody(body),
+      });

example/streamed-list-objects/README.md (1)

16-22: Doc: mention building the SDK or using the published package.

Examples import from ../../dist; advise building first or installing from npm.

Suggested snippet:

# From repo root
pnpm install
pnpm build
cd example/streamed-list-objects
node streamedListObjects.mjs

example/streamed-list-objects/streamedListObjects.mjs (2)

19-36: writeTuples drops remainder; handle non-multiples of 100.

If quantity isn’t divisible by 100, the tail is lost.

-    const chunks = Math.floor(quantity / 100);
+    const chunks = Math.floor(quantity / 100);
+    const remainder = quantity % 100;
@@
     for (let chunk = 0; chunk < chunks; ++chunk) {
       const tuples = [];
       for (let t = 0; t < 100; ++t) {
         tuples.push({
           user: "user:anne",
           relation: "owner",
           object: `document:${chunk * 100 + t}`
         });
       }
       await fgaClient.writeTuples(tuples);
     }
+    if (remainder) {
+      const tuples = [];
+      for (let t = 0; t < remainder; ++t) {
+        tuples.push({
+          user: "user:anne",
+          relation: "owner",
+          object: `document:${chunks * 100 + t}`
+        });
+      }
+      await fgaClient.writeTuples(tuples);
+    }

---

94-99: Ensure cleanup in finally.

Move deleteStore into finally and guard on storeId to avoid orphaned stores on errors.

Example:

let storeId;
try {
  storeId = await createStore(fgaClient);
  // ...
} finally {
  if (storeId) await fgaClient.deleteStore().catch(() => {});
}

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9b81c81 and 055d1dd.

📒 Files selected for processing (14)

• CHANGELOG.md (1 hunks)
• api.ts (5 hunks)
• apiModel.ts (1 hunks)
• client.ts (3 hunks)
• common.ts (1 hunks)
• example/streamed-list-objects-local/README.md (1 hunks)
• example/streamed-list-objects-local/streamedListObjectsLocal.mjs (1 hunks)
• example/streamed-list-objects/README.md (1 hunks)
• example/streamed-list-objects/model.json (1 hunks)
• example/streamed-list-objects/streamedListObjects.mjs (1 hunks)
• index.ts (1 hunks)
• streaming.ts (1 hunks)
• tests/helpers/nocks.ts (2 hunks)
• tests/streaming.test.ts (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (8)

example/streamed-list-objects/streamedListObjects.mjs (2)

client.ts (4)

• createStore (324-326)
• writeTuples (568-572)
• streamedListObjects (825-845)
• listObjects (797-807)

example/streamed-list-objects-local/streamedListObjectsLocal.mjs (2)

• model (13-29)
• storeId (9-9)

tests/helpers/nocks.ts (1)

tests/helpers/default-config.ts (1)

• defaultConfiguration (56-56)

streaming.ts (1)

index.ts (1)

• parseNDJSONStream (27-27)

client.ts (2)

apiModel.ts (1)

• StreamedListObjectsResponse (869-876)

streaming.ts (1)

• parseNDJSONStream (96-168)

api.ts (3)

apiModel.ts (1)

• ListObjectsRequest (804-847)

common.ts (6)

• RequestArgs (30-33)
• DUMMY_BASE_URL (23-23)
• setSearchParams (51-66)
• serializeDataIfNeeded (88-96)
• toPathString (102-104)
• createStreamingRequestFunction (353-418)

validation.ts (1)

• assertParamExists (8-12)

example/streamed-list-objects-local/streamedListObjectsLocal.mjs (1)

client.ts (2)

• OpenFgaClient (233-945)
• ClientConfiguration (59-92)

tests/streaming.test.ts (1)

streaming.ts (1)

• parseNDJSONStream (96-168)

common.ts (4)

base.ts (1)

• RequestArgs (15-18)

configuration.ts (1)

• Configuration (58-202)

telemetry/attributes.ts (1)

• TelemetryAttributes (35-130)

telemetry/histograms.ts (1)

• TelemetryHistograms (20-32)

🔇 Additional comments (10)

client.ts (1)

24-25: Imports look correct.

Also applies to: 52-53

common.ts (1)

345-347: No functional change here.

tests/helpers/nocks.ts (1)

3-4: LGTM: Node stream import fits the streaming mock.

api.ts (1)

1442-1454: OO wrapper wiring looks correct.

index.ts (1)

27-27: LGTM: public export of parseNDJSONStream.

example/streamed-list-objects/model.json (1)

1-251: LGTM: model is coherent and matches example relations.

tests/streaming.test.ts (4)

14-19: LGTM! Well-organized test suite structure.

The imports and test suite organization are clean and appropriate for testing the NDJSON streaming parser.

---

20-96: Excellent coverage of core parsing scenarios.

These tests cover the fundamental NDJSON parsing cases effectively. The chunked data test (lines 48-65) is particularly valuable as it validates the buffering logic when JSON objects are split across stream chunks—a critical real-world scenario.

---

98-162: Comprehensive input type coverage.

These tests validate the parser's flexibility with different input types (Buffer, string, async iterable). The test for JSON without a trailing newline (lines 109-117) is particularly important as it validates the final buffer flush logic. The as any type casts are acceptable here to test the parser's runtime behavior with various input types.

---

164-257: Thorough async generator protocol and cleanup testing.

These tests rigorously validate the async iterator implementation, including error handling, early cancellation, buffering behavior, and proper cleanup of event listeners. The listener cleanup assertions (lines 193-195, 241-243) are particularly valuable for preventing memory leaks. While these tests exercise internal implementation details, the guarantees they provide about correctness and resource management justify their inclusion.

[rhamzeh]

Can you also review @SoulPancake's PR here: openfga/go-sdk#252

Ideally both would have the same semantics, example, tests, config options, README, etc..

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​openfga/​syntax-transformer@​0.2.0

View full report

[aaguiarz]

The example itself does not make a lot of sense, not sure if we are using the same in every SDK. We should not write tuples to the same relation we do list_objects (we can solve the same query with Read, which is much faster).

The model should be something like

define owner : [user]
define viewer : [user]
define can_read : owner or viewer

We can write tuples to owner/viewer and call list_objects on can_read

cc @rhamzeh

[daniel-jonathan]

@aaguiarz

New Example Approach Summary

The Change

Old: Write and query the same relation (direct tuples)

Write: user:anne can_read document:1-2000
Query: can_read

New: Write to base relations, query computed relation

Model:
  define owner: [user]
  define viewer: [user]
  define can_read: owner or viewer  ← COMPUTED

Write: 
  user:anne owner document:1-1000
  user:anne viewer document:1001-2000

Query: can_read (returns all 2000 via computation)

Why

• Shows OpenFGA's value: Computed permissions from base relations
• Demonstrates streaming need: Computed relations return large result sets
• Educational: Real-world authorization pattern