Skip to content

Conversation

@emmiegit
Copy link
Member

@emmiegit emmiegit commented May 1, 2025

This resolves a number of issues related to hosted text blocks:

  • WJ-1224 — Expose hosted text blocks via URL
  • WJ-99 — Support [[html]]
  • WJ-1301 — Support [[code]]
  • WJ-1302 — Support named code blocks

This glues together several pieces from previous PRs to finally support text blocks within Wikijump. There are a few components of this:

  • Retrieving text block data from ftml
  • Storing text blocks in the separate S3 bucket
  • Storing text block metadata in the database
  • Adding route support for text blocks
  • Adding special error support for error cases throughout wws

The flow is this:

  1. A call to RenderService is made. This is now divided into RenderService::render() (regular calls) and RenderService::render_page() (specifically named pages).
  2. At RenderService::render_page(), it goes through all the text blocks in the tree and adds them.
  3. At TextBlockService::add_blocks(), this goes and replaces all existing text blocks with the new ones in S3 and the database.

Then when you request a block:

  1. wws receives a request for a text block route. It passes the text block type and index.
  2. For code blocks, it allows names/aliases to be processed. These are converted to its associated index.
  3. Once we have an index, we can generate the s3_filename, which is what the text block is called in its S3 bucket.
  4. The object is downloaded to memory and an HTTP response is formed using its data and headers.

The special error changes also remove a number of stubs for failure cases in wws and improves its handling in the code for a smoother developer experience.

A separate preprocess/parse timeout is required to resolve some borrow issues, since we need the syntax tree intact to do text block processing after rendering.

This PR bumps the versions of deepwell and wws to v2025.5.4.

There are also some miscellaneous improvements:

  • wws will no longer crash locally when deepwell hasn't started yet
  • Adds the notion of "response results" for more convenient returning at the web request level
  • Adds ETags to file and text block outputs.
  • Updated the default port in wws argument parsing.
  • Added fallback error codes in comments for easier grepping.
  • Added helper functions for fetching different values (with HTTP response errors on failure).

@emmiegit emmiegit self-assigned this May 1, 2025
@emmiegit emmiegit force-pushed the WJ-1224-text-blocks branch 3 times, most recently from dcc0153 to 2b9564e Compare May 5, 2025 04:44
@emmiegit emmiegit changed the title [WJ-1224] Implement hosted text blocks [WJ-1224] [WJ-99] [WJ-1301] [WJ-1302] Implement hosted text blocks May 5, 2025
@emmiegit emmiegit requested a review from Zokhoi May 5, 2025 05:49
@emmiegit emmiegit force-pushed the WJ-1224-text-blocks branch from 9c43fae to cdec375 Compare May 5, 2025 05:57
emmiegit added 23 commits May 5, 2025 01:59
This makes a generic helper function which can handle both cases.
This removes the unused warning and also it's cool to be able to resolve
the actual index number from a name.
This is needed for special page rendering.
It's also just a more natural way to extract the page ID.
Since this is the hash of its content and the hash always changes
when the file does (pointing to a different blob), this is what
we want in terms of cache management.
This is a QoL for local, where 'cargo watch'
may be still rebuilding deepwell when wws restarts.
@emmiegit emmiegit force-pushed the WJ-1224-text-blocks branch from cdec375 to 8c2569c Compare May 5, 2025 05:59
@emmiegit emmiegit marked this pull request as ready for review May 5, 2025 05:59
@emmiegit emmiegit merged commit f8e538e into develop May 5, 2025
15 checks passed
@emmiegit emmiegit deleted the WJ-1224-text-blocks branch May 5, 2025 10:55
@emmiegit
Copy link
Member Author

emmiegit commented May 5, 2025

thanks Zokhoi!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants