Metadata Collection Prototype #1062
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Design doc for a new generic collection type that uses lightweight metadata fetches (id + modified_gmt) to define list structure, then selectively fetches only missing or stale entities. Key design decisions: - Metadata defines list structure (enables loading placeholders) - KV store for metadata persistence (memory or disk-backed) - Generic over entity type (posts, media, etc.) - Batch fetch via `include` param for efficiency
Foundation types for metadata-based sync: - `SyncableEntity` trait: entities with `id` and `modified_gmt` fields - `EntityMetadata<Id>`: lightweight struct holding just id + modified_gmt Also adds `Clone` and `Copy` derives to `WpGmtDateTime` since the inner `DateTime<Utc>` is `Copy`.
Represents items in a metadata-backed list where entities may be: - `Loaded`: full entity available from cache - `Loading`: fetch in progress, shows placeholder with metadata - `Failed`: fetch failed, includes error message for retry UI Includes `HasId` helper trait for extracting IDs from loaded entities.
Abstracts metadata persistence so we can swap between in-memory and disk-backed storage. The in-memory implementation is useful for prototyping; can be replaced with SQLite/file-based later. Includes unit tests for all KvStore operations.
Lightweight fetch that returns only id + modified_gmt for posts, enabling the metadata-first sync strategy. Unlike `fetch_posts_page`, this does NOT upsert to the database - the metadata is used transiently to determine which posts need full fetching. Also adds `Hash` derive to `wp_content_i64_id` and `wp_content_u64_id` macros so ID types can be used as HashMap keys.
Batch fetch full post data for specific IDs using the `include` parameter. Used for selective sync - fetching only posts that are missing or stale in the cache. Returns early with empty Vec if no IDs provided.
Core collection type that: - Uses KV store to persist list metadata (id + modified_gmt) - Builds list with Loaded/Loading states based on cache status - Provides methods to find missing/stale entities for selective fetch - Supports pagination with append for subsequent pages Includes comprehensive unit tests for all operations.
- Format derive macros across multiple lines - Use or_default() instead of or_insert_with(Vec::new) - Add type aliases for complex closure types - Collapse nested if-let into single condition - Simplify test entity to unit struct
Documents what was built, where each component lives, test coverage, and differences from the original sketch. Also lists next steps.
Consolidated design after discussion covering: - Service-owned stores (`EntityStateStore`, `ListMetadataStore`) - Read-only traits for collection access - `MetadataCollection<F>` generic only over fetcher - Cross-collection state consistency - State transitions (Missing/Fetching/Cached/Stale/Failed) Also updated v1 doc with intermediate v2 notes.
Implements Phase 1 of the v3 MetadataCollection design: - `EntityMetadata` - Non-generic struct with `i64` id + `Option<WpGmtDateTime>` (optional modified_gmt for entities like Comments that lack this field) - `EntityState` - Enum tracking fetch lifecycle (Missing, Fetching, Cached, Stale, Failed) - `CollectionItem` - Combines metadata with state for list items - `SyncResult` - Result of sync operations with counts and pagination info - `MetadataFetchResult` - Updated to non-generic version Removes superseded prototype code: - Old generic `EntityMetadata<Id>` - `KvStore<Id>` trait and `InMemoryKvStore<Id>` - `ListItem<T, Id>` enum - `MetadataCollection<T, Id>` (old version) - `SyncableEntity` trait Updates `PostService::fetch_posts_metadata` to use new non-generic types.
Implements Phase 2 of the v3 MetadataCollection design: - `EntityStateStore` - Memory-backed store for entity fetch states - Maps `i64` ID to `EntityState` (Missing, Fetching, Cached, etc.) - Thread-safe via `RwLock<HashMap>` - `filter_fetchable()` excludes currently-fetching IDs to prevent duplicates - `EntityStateReader` trait - Read-only access for collections - `ListMetadataStore` - Memory-backed store for list structure - Maps filter key (String) to `Vec<EntityMetadata>` - Supports `set` (replace) and `append` (pagination) - `ListMetadataReader` trait - Read-only access for collections Both stores are memory-only; state resets on app restart.
Implements Phase 3 of the v3 MetadataCollection design: - `MetadataFetcher` trait - Async trait for fetching metadata and entities - `fetch_metadata(page, per_page, is_first_page)` - Fetch list structure - `ensure_fetched(ids)` - Fetch full entities by ID - `MetadataCollection<F>` - Generic collection over fetcher type - `refresh()` - Fetch page 1, replace metadata, sync missing - `load_next_page()` - Fetch next page, append, sync missing - `items()` - Get `CollectionItem` list with states - `is_relevant_update()` - Check DB updates for relevance - Batches large fetches into 100-item chunks (API limit) Also adds `tokio` as dev-dependency for async tests.
Add metadata sync infrastructure to PostService for efficient list syncing: - Add `state_store_with_edit_context` field for tracking per-entity fetch state (Missing, Fetching, Cached, Stale, Failed). Each context needs its own state store since the same entity can have different states across contexts. - Add `metadata_store` field for list structure per filter key. Shared across all contexts - callers include context in the key string (e.g., "site_1:edit:posts:publish"). - Add `fetch_and_store_metadata()` method that fetches lightweight metadata (id + modified_gmt) and stores it in the metadata store. - Update `fetch_posts_by_ids()` to track entity state: - Filters out already-fetching IDs to prevent duplicate requests - Sets Fetching state before API call - Sets Cached on success, Failed on error or missing posts - Add `PostMetadataFetcherWithEditContext` implementing `MetadataFetcher` trait, delegating to PostService methods. - Add reader accessor methods for collections to get read-only access: `state_reader_with_edit_context()`, `metadata_reader()`, `get_entity_state_with_edit_context()`.
Create the concrete type that wraps MetadataCollection for UniFFI:
- Add `PostMetadataCollectionWithEditContext` struct combining:
- `MetadataCollection<PostMetadataFetcherWithEditContext>` for sync logic
- Service reference for loading full entity data
- Filter for this collection
- Add `PostMetadataCollectionItem` record type with:
- `id`: Post ID
- `state`: EntityState (Missing, Fetching, Cached, Stale, Failed)
- `data`: Optional FullEntityAnyPostWithEditContext
- Add `create_post_metadata_collection_with_edit_context` to PostService
- Make types UniFFI-compatible:
- Add `uniffi::Enum` to EntityState
- Add `uniffi::Record` to SyncResult (change usize to u64)
- Use interior mutability (RwLock<PaginationState>) in MetadataCollection
for compatibility with UniFFI's Arc-wrapped objects
- Add `read_posts_by_ids_from_db` helper to PostService for bulk loading
- Document state representation approaches in design doc:
- Two-dimensional (DataState + FetchStatus)
- Flattened explicit states enum
Add Kotlin wrapper for PostMetadataCollectionWithEditContext to enable reactive UI updates when database changes occur. Changes: - Add `ObservableMetadataCollection` class with observer pattern - Update `DatabaseChangeNotifier` to support metadata collections - Add `getObservablePostMetadataCollectionWithEditContext` extension on `PostService` - Update implementation plan with Phase 7 completion
Add example screen demonstrating the metadata-first sync strategy with visual state indicators for each item (Missing, Fetching, Cached, Stale, Failed). Changes: - Add `PostMetadataCollectionViewModel` with manual refresh/loadNextPage controls - Add `PostMetadataCollectionScreen` with filter controls and state indicators - Wire up navigation and DI for the new screen - Update implementation plan marking Phase 8 complete
WordPress REST API defaults to filtering by 'publish' status, which caused drafts, pending, and other non-published posts to return "Not found" when fetching by ID. Changes: - Add explicit status filter including all standard post statuses (publish, draft, pending, private, future)
When fetching metadata, compare the API's `modified_gmt` against cached posts in the database. Posts with different timestamps are marked as `Stale`, triggering a re-fetch on the next sync. Changes: - Add `select_modified_gmt_by_ids` to PostRepository for efficient batch lookup - Add `detect_and_mark_stale_posts` to PostService for staleness comparison - Call staleness detection in `fetch_and_store_metadata` after storing metadata
Design for moving list metadata from in-memory KV store to database: - Three DB tables: list_metadata, list_metadata_items, list_metadata_state - MetadataService owns list storage, state transitions, version management - PostService orchestrates sync, owns entity state, does staleness detection - Split observers for data vs state changes in Kotlin wrapper - Version-based concurrency control for handling concurrent refreshes
Detailed plan with 5 phases, 17 commits, ordered from low-level to high-level: - Phase 1: Database foundation (DbTable, migration, types, repository) - Phase 2: MetadataService in wp_mobile - Phase 3: Integration with PostService, collection refactor - Phase 4: Observer split (data vs state) - Phase 5: Testing and cleanup Includes dependency order, risk areas, and verification checkpoints.
Implement the database layer for list metadata storage, replacing the in-memory KV store. This enables proper observer patterns and persistence between app launches. Database schema (3 tables): - list_metadata: headers with pagination, version for concurrency control - list_metadata_items: ordered entity IDs (rowid = display order) - list_metadata_state: sync state (idle, fetching_first_page, fetching_next_page, error) Changes: - Add DbTable variants: ListMetadata, ListMetadataItems, ListMetadataState - Add migration 0007-create-list-metadata-tables.sql - Add list_metadata module with DbListMetadata, DbListMetadataItem, DbListMetadataState structs and ListState enum - Add db_types/db_list_metadata.rs with column enums and from_row impls - Add repository/list_metadata.rs with read and write operations
Add helper methods for atomic state transitions during sync operations: - `begin_refresh()`: Atomically increment version, set state to FetchingFirstPage, and return info needed for the fetch - `begin_fetch_next_page()`: Check pagination state, set state to FetchingNextPage, and return page number and version for stale check - `complete_sync()`: Set state to Idle on success - `complete_sync_with_error()`: Set state to Error with message These helpers ensure correct state transitions and enable version-based concurrency control to detect when a refresh invalidates an in-flight load-more operation. Also adds `RefreshInfo` and `FetchNextPageInfo` structs to encapsulate the data returned from begin operations.
Implement MetadataService in wp_mobile to provide persistence for list metadata (ordered entity IDs, pagination, sync state). This enables data to survive app restarts unlike the in-memory ListMetadataStore. The service wraps ListMetadataRepository and implements ListMetadataReader trait, allowing MetadataCollection to use either memory or database storage through the same interface. Features: - Read operations: get_entity_ids, get_metadata, get_state, get_pagination - Write operations: set_items, append_items, update_pagination, delete_list - State management: set_state, complete_sync, complete_sync_with_error - Concurrency helpers: begin_refresh, begin_fetch_next_page Also fixes pre-existing clippy warnings in posts.rs (collapsible_if).
Add MetadataService as a field in PostService to provide database-backed list metadata storage. This enables list structure and pagination to persist across app restarts. Changes: - Add `metadata_service` field to PostService - Add `persistent_metadata_reader()` and `metadata_service()` accessors - Add `sync_post_list()` method that orchestrates full sync flow using MetadataService for persistence - Extend SyncResult with `current_page` and `total_pages` fields for pagination tracking The existing in-memory `metadata_store` is preserved for backwards compatibility with existing code paths. Future work will migrate callers to use the persistent service.
Mark completed phases and update with actual commit hashes: - Phase 1 (Database Foundation): Complete - Phase 2 (MetadataService): Complete - Phase 3 (Integration): Partial (3.2 done, 3.1 deferred) - Phase 5 (Testing): Partial (tests inline with implementation) Add status summary table and update dependency diagram with completion markers.
Add `reset_stale_fetching_states_internal` to `WpApiCache` that resets `FetchingFirstPage` and `FetchingNextPage` states to `Idle` after migrations. This prevents perpetual loading indicators when the app is killed during a fetch operation. The reset runs in `perform_migrations()` because `WpApiCache` is typically created once at app startup, unlike `MetadataService` which is instantiated per-service. Changes: - Add `reset_stale_fetching_states_internal()` with comprehensive docs - Call from `perform_migrations()` after migrations complete - Add session handover document for MetadataService implementation
Migrate `PostMetadataCollectionWithEditContext` from in-memory metadata store to persistent database storage via `MetadataService`. Changes: - Add `fetch_and_store_metadata_persistent()` to PostService - Create `PersistentPostMetadataFetcherWithEditContext` that writes to database - Update `create_post_metadata_collection_with_edit_context` to use: - `persistent_metadata_reader()` instead of `metadata_reader()` - New persistent fetcher - `DbTable::ListMetadataItems` in relevant_tables for update notifications - Export new fetcher from sync module This completes Phase 3.3 of the MetadataService implementation plan. The in-memory `ListMetadataStore` is preserved for backwards compatibility until Phase 3.4 removes it.
Phase 4 of MetadataService implementation: Enable granular observer control so UI can show loading indicators separately from data refreshes. Changes: - Split `is_relevant_update` into `is_relevant_data_update` and `is_relevant_state_update` in `MetadataCollection` - Add relevance checking methods to `ListMetadataReader` trait: - `get_list_metadata_id()` - get DB rowid for a key - `is_item_row_for_key()` - check if item row belongs to key - `is_state_row_for_list()` - check if state row belongs to list - `get_sync_state()` - get current ListState for a key - Add `sync_state()` method to query current sync state (Idle, FetchingFirstPage, FetchingNextPage, Error) - Add repository methods for relevance checking in `wp_mobile_cache` - Implement all trait methods in `MetadataService` The original `is_relevant_update()` still works (combines both checks) for backwards compatibility. Phase 4.2 (Kotlin wrapper split observers) not included - requires platform-specific updates.
…ents Finish the observer split implementation and update the example app with proper state tracking and UI improvements. Changes: Kotlin wrapper (`ObservableMetadataCollection.kt`): - Split observers into `dataObservers` and `stateObservers` - Add `addDataObserver()`, `addStateObserver()` methods - Add `syncState()` suspend function for querying ListState - Make `loadItems()` a suspend function Rust (`post_metadata_collection.rs`, `posts.rs`, `metadata_collection.rs`): - Make `load_items()` and `sync_state()` async for UniFFI background dispatch - Add state management in `fetch_and_store_metadata_persistent`: - Call `begin_refresh()`/`begin_fetch_next_page()` at start - Call `complete_sync()`/`complete_sync_with_error()` on completion - Remove dead `RelevanceCache` struct and `relevance_cache` field Example app: - Add `syncState: ListState` to ViewModel state - Use split observers with coroutine dispatch - Add `SyncStateIndicator` showing database-backed sync state - Change Idle status color to dark green for better visibility - Add back buttons to both collection screens for navigation testing - Fix `loadNextPage()` to call `refresh()` when no pages loaded yet Documentation: - Update implementation plan with Phase 4.2, 5.3 completion - Update session handover with bug fixes and design decisions
Remove the legacy in-memory ListMetadataStore in favor of database-backed MetadataService. All collections now use persistent storage. Changes: - Delete `list_metadata_store.rs` (replaced by `list_metadata_reader.rs`) - Keep `ListMetadataReader` trait in new `list_metadata_reader.rs` - Remove `PostMetadataFetcherWithEditContext` (non-persistent fetcher) - Remove from `PostService`: - `metadata_store` field - `metadata_reader()` method - `fetch_and_store_metadata()` method - Remove tests using `MockFetcher` with in-memory store - Update module exports in `mod.rs`
- Remove verbose observer debug prints from Kotlin (ViewModel, ObservableMetadataCollection) - Consolidate PostService fetch_metadata_persistent prints into single summary line - Use consistent [PostService] prefix
Load pagination state from database when creating a collection, ensuring state persists across filter changes and app restarts. Changes: Rust: - Add `get_current_page()` and `get_total_pages()` to `ListMetadataReader` trait - Implement pagination methods in `MetadataService` - `MetadataCollection::new()` now loads persisted pagination from database Kotlin ViewModel: - Fix race condition: `refresh()` and `loadNextPage()` now set `syncState` when completing, preventing stale state from overwriting observer updates - `setFilter()` reads pagination from collection (which reads from DB) - Load `syncState` asynchronously on filter change
Mark all phases complete and document final session work: - Phase 3.4: In-memory store removal - Phase 5.5: Debug print cleanup - Phase 5.6: State persistence on filter change - Bug fixes: Race condition, state persistence - UI improvements: Back buttons, colors
- Update migration count from 6 to 7 in Kotlin and Swift tests - Fix detekt TooManyFunctions: add @Suppress annotation with explanation - Fix detekt ForbiddenComment: rephrase TODO as design note
Merge 6 separate design/implementation/handover docs into one consolidated document with: - Architecture diagram (ASCII) - Implementation status table - Key files reference - Database schema - State transitions - Design decisions - Bug fixes and learnings - Commit history
The EntityStateStore is memory-only and resets on app restart, causing items to show placeholders instead of cached data. load_items() now queries the cache for all items regardless of their EntityState. Data availability is independent of fetch state. Changes: - Load data for all item IDs, not just those with is_cached() state - Update doc comments to clarify state vs data availability - Add design document explaining the root cause and fix
Replace separate `state` and `data` fields with a single `PostItemState` enum that encodes both sync status and data availability in its variants. Variants with data: Cached, Stale, FetchingWithData, FailedWithData Variants without data: Missing, Fetching, Failed This makes data presence type-safe and eliminates inconsistent states. The match in load_items() combines EntityState with cache lookup to produce the appropriate variant.
Extract PostItemState enum into a reusable macro that can generate type-safe item state enums for any entity type. The macro generates variants that encode both sync status and data availability: Missing, Fetching, FetchingWithData, Cached, Stale, Failed, FailedWithData. Usage: wp_mobile_item_state!(PostItemState, FullEntityAnyPostWithEditContext);
Use PostListParams directly instead of AnyPostFilter for greater flexibility in the metadata collection API. This allows callers to use any supported WordPress REST API parameter without needing to maintain a separate filter abstraction. Changes: - Add cache key generation function using PostListParamsField enum - Update PersistentPostMetadataFetcherWithEditContext to use PostListParams - Update PostMetadataCollectionWithEditContext to use PostListParams - Update PostService methods to accept PostListParams - Add Clone derive to PostListParams in wp_api - Update Kotlin wrappers and example app
Update PostMetadataCollectionScreen and PostMetadataCollectionViewModel to use the new PostItemState enum instead of EntityState. The PostItemState enum encodes both sync status and data availability in a single type. Changes: - Extract data from PostItemState variants that carry data - Handle FetchingWithData and FailedWithData variants in StateIndicator - Update stateDisplayName helper for new variants
Replace manual `isSyncing` boolean with computed property derived from `syncState`. This eliminates the race condition where the completion handler's state update could be missed, leaving the spinner stuck. Changes: - Remove `isSyncing` field from `PostMetadataCollectionState` - Add computed `isSyncing` property based on `syncState` - Remove manual `isSyncing` toggling in `refresh()` and `loadNextPage()` - Let state observer handle all sync state updates
Enables the metadata collection infrastructure to work with Pages and
custom post types by accepting a `PostEndpointType` parameter instead
of hardcoding `PostEndpointType::Posts`.
Changes:
- Add `endpoint_type` parameter to `fetch_posts_metadata()`, `fetch_and_store_metadata_persistent()`, `fetch_posts_by_ids()`, and `sync_post_list()`
- Add `endpoint_type` field to `PersistentPostMetadataFetcherWithEditContext`
- Add `endpoint_type_cache_key()` function with prefixed keys (`post_type_posts`, `post_type_pages`, `post_type_custom_{name}`)
- Update cache key format to include endpoint type: `site_{id}:edit:{endpoint}:{params}`
- Update Kotlin extension and example ViewModel to pass `PostEndpointType.Posts`
Support hierarchical post types (pages, custom post types) by storing parent ID and menu order in metadata. These fields enable proper hierarchy display and ordering in the UI. Changes: - Add `parent` and `menu_order` columns to `list_metadata_items` table - Add fields to `DbListMetadataItem`, `ListMetadataItemInput`, and `EntityMetadata` - Update `fetch_posts_metadata()` to request and extract `Parent` and `MenuOrder` fields - Thread new fields through `MetadataService` read/write operations
Improves naming consistency with `last_first_page_fetched_at`. The field tracks when any page was last fetched, so `last_fetched_at` is clearer than the ambiguous `last_updated_at`. Changes: - Rename column in SQL migration - Update `DbListMetadata` struct field - Update `ListMetadataColumn` enum variant - Update repository SQL and tests
Extends the trait with additional pagination info methods: - `get_total_items()` - total count from API headers - `get_per_page()` - items per page setting These enable MetadataCollection and UI to access full pagination info from the database without maintaining duplicate in-memory state.
Eliminates in-memory pagination state that duplicated database-backed state. Now reads current_page and total_pages directly from ListMetadataReader (database) instead of maintaining a separate copy. Changes: - Remove `PaginationState` struct and `RwLock<PaginationState>` field - Keep only `per_page: u32` as configuration (not state) - `current_page()`, `total_pages()`, `has_more_pages()` now read from DB - Add `total_items()` method to expose total count from API - Simplify `refresh()` and `load_next_page()` by removing state updates This ensures single source of truth for pagination state and prevents potential drift between in-memory and database values.
These methods were designed to filter UpdateHook callbacks by checking if a specific row ID belongs to a collection's key. However, querying the database during hook callbacks causes deadlocks, so the collection just returns true for any update to the relevant tables instead. Removed from ListMetadataReader trait: - get_list_metadata_id() - is_item_row_for_key() - is_state_row_for_list() And their implementations in MetadataService.
Default implementations could mask bugs by silently returning arbitrary values if an implementor forgot to implement a method. Since MetadataService is the only implementor and already provides all methods, requiring explicit implementation ensures compile-time safety.
Consolidate 6 separate trait methods into 2: - `get_list_info()` returns pagination + state via single JOIN query - `get_items()` returns entity metadata (renamed from `get()`) This eliminates redundant database queries when accessing multiple pagination fields (e.g., `has_more_pages()` previously made 2 queries). Changes: - Add `ListInfo` struct combining state, pagination fields - Add `DbListHeaderWithState` for database layer - Add `get_header_with_state()` JOIN query to repository - Update `MetadataCollection` to use `list_info()` for all pagination access - Add new tests for `get_list_info()` trait method
…ervers Complete the ListInfo refactoring through the entire stack: Rust: - Add uniffi::Record to ListInfo for Kotlin export - Add list_info() method to PostMetadataCollectionWithEditContext - Rename is_relevant_state_update() to is_relevant_list_info_update() - Check both ListMetadata and ListMetadataState tables for list info changes Kotlin wrapper: - Add ListInfo.hasMorePages and ListInfo.isSyncing extensions - Expose listInfo() method, remove individual accessors - Rename stateObservers to listInfoObservers Kotlin ViewModel: - Store ListInfo directly in state instead of separate fields - Derive isSyncing, hasMorePages, currentPage from listInfo - Use addListInfoObserver for pagination + state changes Observer split is now: - Data observers: entity tables + list_metadata_items - ListInfo observers: list_metadata + list_metadata_state
Add wp_mobile_metadata_item! macro that generates both the state enum and collection item struct with metadata fields (parent, menu_order). This enables the Android hierarchical post list to build the page tree immediately from list metadata, without waiting for full post data.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please ignore this PR for now, I am just opening a draft PR so I can access the built artifacts from WPAndroid.