Skip to content

Conversation

@ianhi
Copy link
Contributor

@ianhi ianhi commented Nov 4, 2025

Continuation of @jhamman's #3369 with addressing my review comments
Claude summary of the changes on top of Joe's PR:

  1. Format Propagation ✅
  • Zarr format segments (zarr2:, zarr3:) now propagate from URLs to array/group creation
  • Modified resolve_url_with_path() to return (Store, str, ZarrFormat | None)
  • Updated all API functions (create, open, open_array, open_group) to use URL format when user doesn't specify
  • 15 new tests for format propagation with comprehensive on-disk verification
  1. Nested Adapter Chains ✅
  • Implemented recursive URL resolution in ZipAdapter to support nested adapters
  • Added _create_nested_zip_store() method that handles nested ZEP 8 URLs
  • Enables ZEP 8 spec examples like file:outer.zip|zip:inner.zip|zip:data.zarr
  • 8 comprehensive tests covering arrays, groups, edge cases, and complex hierarchies
  1. Exception Handling Cleanup ✅
  • Removed all broad except Exception: handlers from _zep8.py
  • Simplified control flow by removing redundant try-except blocks
  • Only specific KeyError exceptions caught where appropriate
  • Parser exceptions now propagate naturally for easier debugging
  1. URL Logic Refactoring
  • Fixed s3+https scheme handling
  • Improved URL validation logic
  • Better separation of concerns between parsing and resolution
  1. Storage Options Validation
  • Enhanced storage_options handling and validation
  • Better error messages for invalid configurations

Test Coverage: 167 tests passing (1 skipped), up from ~150 tests


Missing ZEP 8 Features Analysis

Based on the ZEP 8 specification, here are the adapter schemes defined vs. implemented:

✅ Currently Implemented (9 adapters)

  • file: - FileSystemAdapter
  • memory: - MemoryAdapter
  • https:, http: - RemoteAdapter
  • s3:, s3+http:, s3+https: - S3Adapter (via RemoteAdapter)
  • gs: - GCSAdapter (via RemoteAdapter)
  • zip: - ZipAdapter
  • log: - LoggingAdapter (custom, not in spec)
  • zarr2:, zarr3:, zarr: - Format segments (handled by resolution layer)

❌ Missing from Spec (11 adapters)

Storage/Database Adapters:

  1. ocdbt: - OCDBT format (versioned KV store)
  2. icechunk: - Icechunk format (versioned Zarr store)

Compression Adapters:
3. gzip: - Transparent gzip decompression
4. zstd: - Transparent zstd decompression

Data Format Adapters:
5. n5: - N5 format support
6. tiff:, jpeg:, png:, bmp:, avif:, webp: - Image format adapters
7. neuroglancer-precomputed: - Neuroglancer format
8. json: - JSON pointer access

Utility Adapters:
9. byte-range:start-end - Byte range extraction
10. ..: - Parent directory traversal (for relative URLs)

Other Missing Features:

  • Relative URL pipeline syntax - Spec lines 489-543 (explicitly noted as not supported in zarr-python
    implementation notes)
  • Format auto-detection - Spec lines 420-443 (MAY support, optional feature)

jhamman and others added 26 commits August 11, 2025 13:13
- Add comprehensive ZEP 8 URL parsing and resolution system
- Implement StoreAdapter ABC for extensible storage adapters
- Add built-in adapters for file, memory, S3, GCS, HTTPS schemes
- Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3:
- Add URLSegment parsing with validation
- Integrate with zarr.open_group and zarr.open_array APIs
- Include demo script and comprehensive test suite
- Pass all existing tests + 35 new ZEP 8-specific tests
@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Nov 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants