docs: improve on documentation

Author: ctron

docs/adrs/00009-re-process-documents.md renamed to docs/adrs/00011-re-process-documents.md

@@ -1,4 +1,4 @@
-# 00009. Re-process documents
+# 00011. Re-process documents
 
 Date: 2025-08-08
 
@@ -28,7 +28,7 @@ any stored values in the database. It therefore is necessary to re-process docum
 This ADR makes the following assumptions:
 
 * All documents are stored in the storage
-* It is expected that the step of upgrading has to be performed by someone
+* It is expected that the step of upgrading has to be performed by someone, it is not magically happening
 * Running such migrations is expected to take a long time
 * The management of infrastructure (PostgreSQL) is not in the scope of Trustify
 
@@ -52,7 +52,7 @@ We could provide an endpoint to the UI, reporting the fact that the system is in
 * 👍 Can fully migrate database (create mandatory field as optional -> re-process -> make mandatory)
 * 👍 Might allow for an out-of-band migration of data, before running the upgrade (even on a staging env)
 * 👍 Would allow to continue serving data while the process is running
-* 👎 Might be tricky to create a combined re-processing of multiple ones
+* 👎 Might be tricky to create a combined re-processing of multiple ones at the same time
 * 👎 Might block an upgrade if re-processing fails
 
 We do want to support different approaches of this migration. Depending on the needs of the user, the size of the
@@ -77,8 +77,14 @@ transaction mode of read-only.
 
 Migrations which do re-process data have to be written in a way, that they can be run and re-run without failing
 during the migration of the schema (e.g. add "if not exist"). In this case, the data migration job can be run
-"out of band" (beforehand) and the data be processed. Then, the actual upgrade and schema migration can run.
+"out of band" (beforehand) and the data be processed. Then, the actual upgrade and schema migration can run, keeping
+the SeaORM process.
 
+* The migration will block the upgrade process until it is finished
+* Ansible and the operator will need to handle this as well
+* The system will become read-only during a migration
+* The UI should let the user know the system is in read-only mode. This is a feature which has to be rolled out before
+  the data migration can be used.
 
 ## Open items
 
@@ -124,10 +130,3 @@ Have the operator orchestrate the process of switching the database into read-on
 
 This adds a lot of user-friendliness. However, it also is rather complex and so we should, as a first step, have this
 as a manual step.
-
-## Consequences
-
-* The migration will block the upgrade process until it is finished
-* Ansible and the operator will need to handle this as well
-* The system will become read-only during a migration
-* The UI needs to provide a page for monitoring the migration state. The backend needs to provide appropriate APIs.

migration/src/data/document/mod.rs

@@ -14,6 +14,7 @@ use trustify_entity::source_document;
 use trustify_module_storage::service::{StorageBackend, StorageKey};
 use uuid::Uuid;
 
+/// A document eligible for re-processing.
 #[allow(async_fn_in_trait)]
 pub trait Document: Sized + Send + Sync {
     type Model: Partitionable + Send;

migration/src/data/migration.rs

@@ -10,6 +10,7 @@ use std::{ffi::OsString, ops::Deref, sync::LazyLock};
 use tokio::task_local;
 use trustify_module_storage::{config::StorageConfig, service::dispatch::DispatchBackend};
 
+/// A migration which also processes data.
 pub struct MigrationWithData {
     pub storage: DispatchBackend,
     pub options: Options,
@@ -87,6 +88,7 @@ where
     }
 }
 
+/// A [`SchemaManager`], extended with data migration features.
 pub struct SchemaDataManager<'c> {
     pub manager: &'c SchemaManager<'c>,
     storage: &'c DispatchBackend,

migration/src/data/mod.rs

@@ -80,6 +80,10 @@ impl From<()> for Options {
 }
 
 impl Options {
+    /// Check if we should skip a data migration. Returns `true` if it should be skipped.
+    ///
+    /// Skipping means that the "data" part of the migration should not be processes. The schema
+    /// part still will be processes.
     pub fn should_skip(&self, name: &str) -> bool {
         if self.skip_all {
             // we skip all migration
@@ -104,6 +108,7 @@ impl From<&Options> for Partition {
     }
 }
 
+/// A trait for processing documents
 pub trait DocumentProcessor {
     fn process<D>(
         &self,
@@ -116,6 +121,42 @@ pub trait DocumentProcessor {
 }
 
 impl<'c> DocumentProcessor for SchemaManager<'c> {
+    /// Process documents for a schema *data* migration.
+    ///
+    /// ## Pre-requisites
+    ///
+    /// The database should be maintenance mode. Meaning that the actual application should be
+    /// running from a read-only clone for the time of processing.
+    ///
+    /// ## Partitioning
+    ///
+    /// This will partition documents and only process documents selected for *this* partition.
+    /// The partition configuration normally comes from outside, as configuration through env-vars.
+    ///
+    /// This means that there may be other instances of this processor running in a different
+    /// process instance. However, not touching documents of our partition.
+    ///
+    /// ## Transaction strategy
+    ///
+    /// The processor will identify all documents, filtering out all which are not part of this
+    /// partition. This is done in a dedicated transaction. As the database is supposed to be in
+    /// read-only mode for the running instance, this is ok as no additional documents will be
+    /// created during the time of processing.
+    ///
+    /// Next, it is processing all found documents, in a concurrent way. Meaning, this single
+    /// process instance, will process multiple documents in parallel.
+    ///
+    /// Each document is loaded and processed within a dedicated transaction. Commiting the
+    /// transaction at the end each step and before moving on the next document.
+    ///
+    /// As handlers are intended to be idempotent, there's no harm in re-running them, in case
+    /// things go wrong.
+    ///
+    /// ## Caveats
+    ///
+    /// However, this may lead to a situation where only a part of the documents is processed.
+    /// But, this is ok, as the migration is supposed to run on a clone of the database and so the
+    /// actual system is still running from the read-only clone of the original data.
     async fn process<D>(
         &self,
         storage: &DispatchBackend,
@@ -188,6 +229,9 @@ impl<'c> DocumentProcessor for SchemaManager<'c> {
 }
 
 /// A handler for data migration of documents.
+///
+/// Handlers have to be written in a way that they can be re-run multiple times on the same
+/// document without failing and creating the exact same output state.
 #[macro_export]
 macro_rules! handler {
     (async | $doc:ident: $doc_ty:ty, $model:ident, $tx:ident | $body:block) => {{

migration/src/data/partition.rs

@@ -5,6 +5,7 @@ use std::{
 };
 use trustify_entity::{advisory, sbom};
 
+/// Information required for partitioning data
 #[derive(Debug, Copy, Clone)]
 pub struct Partition {
     pub current: u64,
@@ -43,6 +44,9 @@ impl Default for Partition {
 }
 
 impl Partition {
+    /// Create a new partition of one.
+    ///
+    /// This will be one processor processing everything.
     pub const fn new_one() -> Self {
         Self {
             current: 0,