Browse Source

Squashed commit of the following:

commit 1d2cd7fc82e1de741cf51c50b2d35d337c476aed
Author: Cesar Rodas <cesar@rodasm.com.py>
Date:   Tue Jul 14 12:20:13 2026 -0300

    Chunk id-batch posting primitives to a safe statement size

    The id-batch primitives (get_postings, get_posting_states, reserve,
    release, deactivate) matched every id in one statement via an OR of
    equality pairs. That OR chain grows with the batch, and SQLite rejects it
    once the expression tree passes its depth limit (default 1000), well
    before the bind-parameter limits are reached; PostgreSQL has its own
    ceilings too. A caller passing a large id set would hit a hard database
    error.

    Split every id batch into fixed-size chunks, keeping the two write
    primitives' chunks inside their existing single transaction so the claim
    or release stays atomic, and summing the affected-row counts. Reads
    accumulate across chunks. The batch size the primitives accept now has no
    practical ceiling.

    Also encode hex without a per-byte allocation, and add a test that drives
    a batch larger than one chunk through reserve, read, and deactivate.

commit 41ce95cb4de2b02e23fd1494b8199f46b46e0c7a
Author: Cesar Rodas <cesar@rodasm.com.py>
Date:   Tue Jul 14 12:13:04 2026 -0300

    Make posting reads deterministic and migrations atomic

    Reviewing the append-only value-table / hot-index split surfaced three
    follow-ups worth fixing.

    Migrations ran statement-by-statement outside any transaction, recording
    a migration as applied only after all of its statements succeeded. The
    new migration that drops and rebuilds the postings table could therefore
    crash half-applied, leaving the schema in a state the migration could not
    be re-run against. Wrap each migration's statements and its bookkeeping
    insert in one transaction so a crash rolls back cleanly and the migration
    is retried whole. Both SQLite and PostgreSQL support transactional DDL.

    Posting reads returned rows in an unspecified order, so LIMIT/OFFSET
    pagination could skip or repeat rows across pages, worst for the live set
    whose source is a UNION of two tables. Order get_postings_by_account by
    the posting id in both backends so the primitive that balance, selection,
    close, and pagination all build on returns a stable sequence.

    Bring the reference docs back in line with the derived-state model:
    postings are immutable and carry no lifecycle column; state is Active,
    Reserved, Spent, or Missing derived from index membership.
Cesar Rodas 2 days ago
parent
commit
edbcc9d619

+ 252 - 179
crates/kuatia-storage-sql/src/lib.rs

@@ -125,11 +125,23 @@ impl SqlStore {
                 continue;
             }
 
+            // Apply every statement and record the migration in one transaction,
+            // so a crash mid-migration rolls back cleanly and the migration is
+            // retried as a whole. Migration 004 drops and rebuilds `postings`;
+            // without the transaction a partial apply would leave the schema in a
+            // state the migration cannot be re-run against. Both SQLite and
+            // PostgreSQL support transactional DDL.
+            let mut tx = self
+                .pool
+                .begin()
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+
             for statement in sql.split(';') {
                 let trimmed = statement.trim();
                 if !trimmed.is_empty() {
                     sqlx::query(trimmed)
-                        .execute(&self.pool)
+                        .execute(&mut *tx)
                         .await
                         .map_err(|e| StoreError::Internal(e.to_string()))?;
                 }
@@ -137,7 +149,11 @@ impl SqlStore {
 
             sqlx::query("INSERT INTO _migrations (name) VALUES ($1)")
                 .bind(*name)
-                .execute(&self.pool)
+                .execute(&mut *tx)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+
+            tx.commit()
                 .await
                 .map_err(|e| StoreError::Internal(e.to_string()))?;
         }
@@ -173,9 +189,11 @@ fn deserialize_json<T: serde::de::DeserializeOwned>(s: &str) -> Result<T, StoreE
 /// opaque saga bytes are stored as hex `TEXT` so a row is legible in any SQL
 /// client and matches the hex form used in logs and `Debug` output.
 fn to_hex(bytes: &[u8]) -> String {
+    const HEX: &[u8; 16] = b"0123456789abcdef";
     let mut s = String::with_capacity(bytes.len() * 2);
-    for b in bytes {
-        s.push_str(&format!("{b:02x}"));
+    for &b in bytes {
+        s.push(HEX[(b >> 4) as usize] as char);
+        s.push(HEX[(b & 0x0f) as usize] as char);
     }
     s
 }
@@ -288,12 +306,23 @@ fn filter_source(filter: PostingFilter) -> &'static str {
     }
 }
 
+/// Maximum posting ids matched by a single statement. `id_predicate` expands to
+/// an `OR` of `n` equality pairs, so the binding constraint is SQLite's
+/// expression-tree depth limit (`SQLITE_MAX_EXPR_DEPTH`, default 1000), which a
+/// chain of `n` `OR`s reaches at roughly `n` deep. It caps well before the
+/// bind-parameter limits (SQLite 32766, PostgreSQL 65535) that `2 * n (+1)`
+/// parameters would hit. `500` stays comfortably under the expression-depth
+/// limit; callers that pass more ids are chunked, so the id-batch primitives
+/// have no practical ceiling on batch size.
+const MAX_IDS_PER_QUERY: usize = 500;
+
 /// Build a portable predicate matching a set of posting ids:
 /// `(transfer_id = $s AND idx = $s+1) OR (transfer_id = $s+2 AND idx = $s+3) ...`
 /// starting at placeholder `$start`. Row-value `IN ((a, b), ...)` is not
 /// portable across SQLite and PostgreSQL; an `OR` of equality pairs is. The
 /// caller binds each id as `(hex(transfer), idx as i16)` in order, matching the
-/// placeholder sequence. `ids` must be non-empty.
+/// placeholder sequence. `ids` must be non-empty and no longer than
+/// [`MAX_IDS_PER_QUERY`]; larger sets are split into chunks by the caller.
 fn id_predicate(count: usize, start: u32) -> String {
     (0..count)
         .map(|i| {
@@ -524,33 +553,36 @@ impl PostingStore for SqlStore {
             return Ok(Vec::new());
         }
 
-        // One set-based query for the whole batch instead of one probe per id,
-        // reusing the portable `id_predicate` and binding each id in order as
-        // `(hex(transfer), idx as i16)`.
-        let sql = format!(
-            "SELECT * FROM postings WHERE {}",
-            id_predicate(ids.len(), 1)
-        );
-        let mut q = sqlx::query(&sql);
-        for id in ids {
-            q = q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
-        }
-        let rows = q
-            .fetch_all(&self.pool)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
-
-        // Index the fetched postings by the same `(hex, idx)` key that was bound.
-        let mut found: HashMap<(String, i16), Posting> = HashMap::with_capacity(rows.len());
-        for row in &rows {
-            let posting = row_to_posting(row)?;
-            let key = (
-                envelope_id_to_hex(&posting.id.transfer),
-                posting.id.index as i16,
+        // Set-based query per chunk instead of one probe per id, reusing the
+        // portable `id_predicate` and binding each id in order as
+        // `(hex(transfer), idx as i16)`. Chunked so a large batch never exceeds
+        // the backend's bind-parameter limit (see `MAX_IDS_PER_QUERY`).
+        let mut found: HashMap<(String, i16), Posting> = HashMap::with_capacity(ids.len());
+        for chunk in ids.chunks(MAX_IDS_PER_QUERY) {
+            let sql = format!(
+                "SELECT * FROM postings WHERE {}",
+                id_predicate(chunk.len(), 1)
             );
-            found.insert(key, posting);
+            let mut q = sqlx::query(&sql);
+            for id in chunk {
+                q = q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            let rows = q
+                .fetch_all(&self.pool)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+
+            // Index the fetched postings by the same `(hex, idx)` key that was bound.
+            for row in &rows {
+                let posting = row_to_posting(row)?;
+                let key = (
+                    envelope_id_to_hex(&posting.id.transfer),
+                    posting.id.index as i16,
+                );
+                found.insert(key, posting);
+            }
         }
 
         // Return in input order, erroring on the first id absent from the batch
@@ -586,6 +618,10 @@ impl PostingStore for SqlStore {
         if asset.is_some() {
             sql.push_str(&format!(" AND asset = ${placeholder}"));
         }
+        // Deterministic order by the posting primary key, matching
+        // `query_postings`, so callers (and pagination built on top) see a
+        // stable sequence.
+        sql.push_str(" ORDER BY transfer_id, idx");
 
         let mut q = sqlx::query(&sql).bind(id);
         if let Some(s) = sub {
@@ -608,48 +644,10 @@ impl PostingStore for SqlStore {
         }
 
         // One set-based query per state table instead of up to three probes per
-        // id. Each reuses the portable `id_predicate` (an OR of equality pairs;
-        // row-value `IN` is not portable across SQLite and PostgreSQL) and binds
-        // every id in order as `(hex(transfer), idx as i16)`.
-        let predicate = id_predicate(ids.len(), 1);
-
-        let active_sql = format!("SELECT transfer_id, idx FROM active_postings WHERE {predicate}");
-        let mut active_q = sqlx::query(&active_sql);
-        for id in ids {
-            active_q = active_q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
-        }
-        let active_rows = active_q
-            .fetch_all(&self.pool)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
-
-        let reserved_sql = format!(
-            "SELECT transfer_id, idx, reservation FROM reserved_postings WHERE {predicate}"
-        );
-        let mut reserved_q = sqlx::query(&reserved_sql);
-        for id in ids {
-            reserved_q = reserved_q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
-        }
-        let reserved_rows = reserved_q
-            .fetch_all(&self.pool)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
-
-        let spent_sql = format!("SELECT transfer_id, idx FROM postings WHERE {predicate}");
-        let mut spent_q = sqlx::query(&spent_sql);
-        for id in ids {
-            spent_q = spent_q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
-        }
-        let spent_rows = spent_q
-            .fetch_all(&self.pool)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
+        // id, reusing the portable `id_predicate` (an OR of equality pairs;
+        // row-value `IN` is not portable across SQLite and PostgreSQL) and
+        // binding every id in order as `(hex(transfer), idx as i16)`. Chunked so
+        // a large batch never exceeds the bind-parameter limit.
 
         // Key membership by the same `(hex, idx)` values that were bound, so the
         // per-id lookup below matches without decoding transfer ids back.
@@ -663,20 +661,63 @@ impl PostingStore for SqlStore {
             Ok((transfer_id, idx))
         };
 
-        let mut active: HashSet<(String, i16)> = HashSet::with_capacity(active_rows.len());
-        for row in &active_rows {
-            active.insert(row_key(row)?);
-        }
-        let mut reserved: HashMap<(String, i16), i64> = HashMap::with_capacity(reserved_rows.len());
-        for row in &reserved_rows {
-            let rid: i64 = row
-                .try_get("reservation")
+        let mut active: HashSet<(String, i16)> = HashSet::new();
+        let mut reserved: HashMap<(String, i16), i64> = HashMap::new();
+        let mut spent: HashSet<(String, i16)> = HashSet::new();
+
+        for chunk in ids.chunks(MAX_IDS_PER_QUERY) {
+            let predicate = id_predicate(chunk.len(), 1);
+
+            let active_sql =
+                format!("SELECT transfer_id, idx FROM active_postings WHERE {predicate}");
+            let mut active_q = sqlx::query(&active_sql);
+            for id in chunk {
+                active_q = active_q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            let active_rows = active_q
+                .fetch_all(&self.pool)
+                .await
                 .map_err(|e| StoreError::Internal(e.to_string()))?;
-            reserved.insert(row_key(row)?, rid);
-        }
-        let mut spent: HashSet<(String, i16)> = HashSet::with_capacity(spent_rows.len());
-        for row in &spent_rows {
-            spent.insert(row_key(row)?);
+            for row in &active_rows {
+                active.insert(row_key(row)?);
+            }
+
+            let reserved_sql = format!(
+                "SELECT transfer_id, idx, reservation FROM reserved_postings WHERE {predicate}"
+            );
+            let mut reserved_q = sqlx::query(&reserved_sql);
+            for id in chunk {
+                reserved_q = reserved_q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            let reserved_rows = reserved_q
+                .fetch_all(&self.pool)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+            for row in &reserved_rows {
+                let rid: i64 = row
+                    .try_get("reservation")
+                    .map_err(|e| StoreError::Internal(e.to_string()))?;
+                reserved.insert(row_key(row)?, rid);
+            }
+
+            let spent_sql = format!("SELECT transfer_id, idx FROM postings WHERE {predicate}");
+            let mut spent_q = sqlx::query(&spent_sql);
+            for id in chunk {
+                spent_q = spent_q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            let spent_rows = spent_q
+                .fetch_all(&self.pool)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+            for row in &spent_rows {
+                spent.insert(row_key(row)?);
+            }
         }
 
         // Reconstruct each id's state in input order, preserving the active >
@@ -712,7 +753,13 @@ impl PostingStore for SqlStore {
             let c = format!("SELECT COUNT(*) as cnt FROM {source} {w}");
             let limit = query.limit.unwrap_or(u32::MAX);
             let offset = query.offset.unwrap_or(0);
-            w.push_str(&format!(" LIMIT {limit} OFFSET {offset}"));
+            // Order by the posting primary key so pagination is deterministic:
+            // without it LIMIT/OFFSET could skip or repeat rows across pages,
+            // especially for `Live`, whose source is a `UNION ALL` with no
+            // inherent order.
+            w.push_str(&format!(
+                " ORDER BY transfer_id, idx LIMIT {limit} OFFSET {offset}"
+            ));
             (format!("SELECT * FROM {source} {w}"), c)
         };
 
@@ -773,43 +820,49 @@ impl PostingStore for SqlStore {
             .await
             .map_err(|e| StoreError::Internal(e.to_string()))?;
 
-        // Reservation is $1; each id pair follows starting at $2.
-        let insert_sql = format!(
-            "INSERT INTO reserved_postings (transfer_id, idx, owner, subaccount, asset, value, reservation) \
-             SELECT transfer_id, idx, owner, subaccount, asset, value, $1 FROM active_postings WHERE {} \
-             ON CONFLICT (transfer_id, idx) DO NOTHING",
-            id_predicate(ids.len(), 2)
-        );
-        let mut insert_q = sqlx::query(&insert_sql).bind(reservation.0);
-        for id in ids {
-            insert_q = insert_q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
-        }
-        insert_q
-            .execute(&mut *tx)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
+        // Chunked so a large id set stays under the bind-parameter limit; all
+        // chunks share one transaction so the whole claim is atomic.
+        let mut claimed: u64 = 0;
+        for chunk in ids.chunks(MAX_IDS_PER_QUERY) {
+            // Reservation is $1; each id pair follows starting at $2.
+            let insert_sql = format!(
+                "INSERT INTO reserved_postings (transfer_id, idx, owner, subaccount, asset, value, reservation) \
+                 SELECT transfer_id, idx, owner, subaccount, asset, value, $1 FROM active_postings WHERE {} \
+                 ON CONFLICT (transfer_id, idx) DO NOTHING",
+                id_predicate(chunk.len(), 2)
+            );
+            let mut insert_q = sqlx::query(&insert_sql).bind(reservation.0);
+            for id in chunk {
+                insert_q = insert_q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            insert_q
+                .execute(&mut *tx)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
 
-        let delete_sql = format!(
-            "DELETE FROM active_postings WHERE {}",
-            id_predicate(ids.len(), 1)
-        );
-        let mut delete_q = sqlx::query(&delete_sql);
-        for id in ids {
-            delete_q = delete_q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
+            let delete_sql = format!(
+                "DELETE FROM active_postings WHERE {}",
+                id_predicate(chunk.len(), 1)
+            );
+            let mut delete_q = sqlx::query(&delete_sql);
+            for id in chunk {
+                delete_q = delete_q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            let del = delete_q
+                .execute(&mut *tx)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+            claimed += del.rows_affected();
         }
-        let del = delete_q
-            .execute(&mut *tx)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
 
         tx.commit()
             .await
             .map_err(|e| StoreError::Internal(e.to_string()))?;
-        Ok(del.rows_affected())
+        Ok(claimed)
     }
 
     async fn release_postings(
@@ -830,43 +883,49 @@ impl PostingStore for SqlStore {
             .await
             .map_err(|e| StoreError::Internal(e.to_string()))?;
 
-        // Reservation is $1; each id pair follows starting at $2.
-        let insert_sql = format!(
-            "INSERT INTO active_postings (transfer_id, idx, owner, subaccount, asset, value) \
-             SELECT transfer_id, idx, owner, subaccount, asset, value FROM reserved_postings \
-             WHERE ({}) AND reservation = $1 ON CONFLICT (transfer_id, idx) DO NOTHING",
-            id_predicate(ids.len(), 2)
-        );
-        let mut insert_q = sqlx::query(&insert_sql).bind(reservation.0);
-        for id in ids {
-            insert_q = insert_q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
-        }
-        insert_q
-            .execute(&mut *tx)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
+        // Chunked so a large id set stays under the bind-parameter limit; all
+        // chunks share one transaction.
+        let mut released: u64 = 0;
+        for chunk in ids.chunks(MAX_IDS_PER_QUERY) {
+            // Reservation is $1; each id pair follows starting at $2.
+            let insert_sql = format!(
+                "INSERT INTO active_postings (transfer_id, idx, owner, subaccount, asset, value) \
+                 SELECT transfer_id, idx, owner, subaccount, asset, value FROM reserved_postings \
+                 WHERE ({}) AND reservation = $1 ON CONFLICT (transfer_id, idx) DO NOTHING",
+                id_predicate(chunk.len(), 2)
+            );
+            let mut insert_q = sqlx::query(&insert_sql).bind(reservation.0);
+            for id in chunk {
+                insert_q = insert_q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            insert_q
+                .execute(&mut *tx)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
 
-        let delete_sql = format!(
-            "DELETE FROM reserved_postings WHERE ({}) AND reservation = $1",
-            id_predicate(ids.len(), 2)
-        );
-        let mut delete_q = sqlx::query(&delete_sql).bind(reservation.0);
-        for id in ids {
-            delete_q = delete_q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
+            let delete_sql = format!(
+                "DELETE FROM reserved_postings WHERE ({}) AND reservation = $1",
+                id_predicate(chunk.len(), 2)
+            );
+            let mut delete_q = sqlx::query(&delete_sql).bind(reservation.0);
+            for id in chunk {
+                delete_q = delete_q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            let del = delete_q
+                .execute(&mut *tx)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+            released += del.rows_affected();
         }
-        let del = delete_q
-            .execute(&mut *tx)
-            .await
-            .map_err(|e| StoreError::Internal(e.to_string()))?;
 
         tx.commit()
             .await
             .map_err(|e| StoreError::Internal(e.to_string()))?;
-        Ok(del.rows_affected())
+        Ok(released)
     }
 
     async fn deactivate_postings(
@@ -874,44 +933,58 @@ impl PostingStore for SqlStore {
         ids: &[PostingId],
         reservation: Option<ReservationId>,
     ) -> Result<u64, StoreError> {
-        // Dumb instruction over the whole id set: a single DELETE removes the
-        // ids from an index so they become spent (present only in the immutable
-        // table). `rows_affected` is the count; the caller interprets a shortfall.
+        // Dumb instruction over the whole id set: a DELETE removes the ids from
+        // an index so they become spent (present only in the immutable table).
+        // `rows_affected` is the count; the caller interprets a shortfall.
+        // Chunked under one transaction so a large id set stays within the
+        // bind-parameter limit while the removal stays atomic.
         if ids.is_empty() {
             return Ok(0);
         }
-        let (sql, rid) = match reservation {
-            // Raw path: remove from the active index.
-            None => (
-                format!(
-                    "DELETE FROM active_postings WHERE {}",
-                    id_predicate(ids.len(), 1)
+        let mut tx = self
+            .pool
+            .begin()
+            .await
+            .map_err(|e| StoreError::Internal(e.to_string()))?;
+        let mut removed: u64 = 0;
+        for chunk in ids.chunks(MAX_IDS_PER_QUERY) {
+            let (sql, rid) = match reservation {
+                // Raw path: remove from the active index.
+                None => (
+                    format!(
+                        "DELETE FROM active_postings WHERE {}",
+                        id_predicate(chunk.len(), 1)
+                    ),
+                    None,
                 ),
-                None,
-            ),
-            // Saga path: remove only the rows reserved by `rid`.
-            Some(rid) => (
-                format!(
-                    "DELETE FROM reserved_postings WHERE ({}) AND reservation = $1",
-                    id_predicate(ids.len(), 2)
+                // Saga path: remove only the rows reserved by `rid`.
+                Some(rid) => (
+                    format!(
+                        "DELETE FROM reserved_postings WHERE ({}) AND reservation = $1",
+                        id_predicate(chunk.len(), 2)
+                    ),
+                    Some(rid),
                 ),
-                Some(rid),
-            ),
-        };
-        let mut q = sqlx::query(&sql);
-        if let Some(rid) = rid {
-            q = q.bind(rid.0);
-        }
-        for id in ids {
-            q = q
-                .bind(envelope_id_to_hex(&id.transfer))
-                .bind(id.index as i16);
+            };
+            let mut q = sqlx::query(&sql);
+            if let Some(rid) = rid {
+                q = q.bind(rid.0);
+            }
+            for id in chunk {
+                q = q
+                    .bind(envelope_id_to_hex(&id.transfer))
+                    .bind(id.index as i16);
+            }
+            let res = q
+                .execute(&mut *tx)
+                .await
+                .map_err(|e| StoreError::Internal(e.to_string()))?;
+            removed += res.rows_affected();
         }
-        let res = q
-            .execute(&self.pool)
+        tx.commit()
             .await
             .map_err(|e| StoreError::Internal(e.to_string()))?;
-        Ok(res.rows_affected())
+        Ok(removed)
     }
 
     async fn insert_postings(&self, postings: &[Posting]) -> Result<u64, StoreError> {

+ 54 - 0
crates/kuatia-storage-sql/tests/sqlite.rs

@@ -350,3 +350,57 @@ async fn migrate_is_idempotent() {
     store.migrate().await.unwrap();
     store.migrate().await.unwrap();
 }
+
+/// The id-batch primitives chunk a large id set so it never exceeds the
+/// backend's bind-parameter limit (SQLite caps a statement at 32766 variables).
+/// A batch larger than one chunk must reserve, read, and deactivate every id
+/// exactly as a small one does.
+#[tokio::test]
+async fn id_batch_primitives_chunk_large_batches() {
+    // Comfortably larger than the internal chunk size so the batch spans
+    // multiple statements.
+    const N: u16 = 9000;
+    let store = new_store().await;
+
+    let tid = EnvelopeId([0xcd; 32]);
+    let postings: Vec<Posting> = (0..N)
+        .map(|i| {
+            Posting::new(
+                PostingId {
+                    transfer: tid,
+                    index: i,
+                },
+                AccountId::new(1),
+                AssetId::new(1),
+                Cent::from(1),
+            )
+        })
+        .collect();
+    let ids: Vec<PostingId> = postings.iter().map(|p| p.id).collect();
+
+    assert_eq!(
+        store.insert_postings(&postings).await.unwrap(),
+        u64::from(N)
+    );
+
+    // Reserve the whole batch in one call: every id is claimed across chunks.
+    let rid = ReservationId::new(1);
+    assert_eq!(
+        store.reserve_postings(&ids, rid).await.unwrap(),
+        u64::from(N)
+    );
+
+    // Reads span chunks and return every id.
+    let states = store.get_posting_states(&ids).await.unwrap();
+    assert_eq!(states.len(), N as usize);
+    assert!(states.iter().all(|s| *s == PostingState::Reserved(rid)));
+    assert_eq!(store.get_postings(&ids).await.unwrap().len(), N as usize);
+
+    // Deactivating the whole batch spends every id.
+    assert_eq!(
+        store.deactivate_postings(&ids, Some(rid)).await.unwrap(),
+        u64::from(N)
+    );
+    let after = store.get_posting_states(&ids).await.unwrap();
+    assert!(after.iter().all(|s| *s == PostingState::Spent));
+}

+ 4 - 1
crates/kuatia-storage/src/mem_store.rs

@@ -176,7 +176,7 @@ impl PostingStore for InMemoryStore {
         // Read from the relevant set directly — no merge back to the immutable
         // record. `Live` is the two disjoint live sets chained together.
         let reserved = || postings.reserved.values().map(|(p, _)| p);
-        let result: Vec<Posting> = match filter {
+        let mut result: Vec<Posting> = match filter {
             PostingFilter::Active => postings.active.values().filter(matches).cloned().collect(),
             PostingFilter::Reserved => reserved().filter(matches).cloned().collect(),
             PostingFilter::Live => postings
@@ -193,6 +193,9 @@ impl PostingStore for InMemoryStore {
                 .cloned()
                 .collect(),
         };
+        // Deterministic order by the posting id, matching the SQL backend and
+        // `query_postings`; the maps above iterate in an unspecified order.
+        result.sort_by_key(|p| p.id);
         Ok(result)
     }
 

+ 5 - 1
crates/kuatia-storage/src/store.rs

@@ -111,7 +111,9 @@ pub trait PostingStore: Send + Sync {
     async fn get_postings(&self, ids: &[PostingId]) -> Result<Vec<Posting>, StoreError>;
     /// Return postings owned by a base account, filtered by subaccount, asset,
     /// and derived lifecycle state (see [`PostingFilter`]). `sub == None` spans
-    /// every subaccount of `id`; `sub == Some(s)` restricts to that one.
+    /// every subaccount of `id`; `sub == Some(s)` restricts to that one. Results
+    /// are ordered by posting id, so callers and pagination built on top see a
+    /// stable sequence.
     async fn get_postings_by_account(
         &self,
         id: i64,
@@ -167,6 +169,8 @@ pub trait PostingStore: Send + Sync {
 
     /// Query postings with filtering and pagination.
     async fn query_postings(&self, query: &PostingQuery) -> Result<Page<Posting>, StoreError> {
+        // `get_postings_by_account` returns a deterministic id-ordered sequence,
+        // so LIMIT/OFFSET here paginate stably without an extra sort.
         let all = self
             .get_postings_by_account(query.account, query.sub, query.asset.as_ref(), query.filter)
             .await?;

+ 3 - 4
doc/accounts.md

@@ -4,10 +4,9 @@
 
 An account is a versioned entity that owns postings. Balance is never
 stored: it is always computed from postings for a given (account, asset)
-pair. The ledger balance sums non-`Inactive` postings (`Active +
-PendingInactive`); the available balance sums only `Active` postings
-(excluding those reserved for an in-flight transfer). `balance()` returns
-the ledger balance.
+pair. `balance()` sums the live postings, meaning those in the active or
+reserved index (`Active ∪ Reserved`); spent postings, which remain only in
+the immutable table, are excluded.
 
 ## Structure
 

+ 30 - 25
doc/crates.md

@@ -25,10 +25,11 @@ dependencies (`sha2`, `serde`, `bitflags`).
 | `AccountSnapshotId { account, snapshot_id }` | Account state hash for version pinning |
 | `Cent` | Smallest monetary unit (private field, backing integer hidden). Backing is `i64` by default, `i128` under the `i128` feature. Checked arithmetic via `checked_add`, `checked_sub`, `checked_neg`, `checked_sum` returning `Result<Cent, OverflowError>` |
 | `OverflowError` | Returned when a `Cent` operation would overflow or underflow |
-| `PostingStatus` | Posting lifecycle: `Active`, `PendingInactive`, `Inactive` |
+| `PostingState` | Derived posting lifecycle (from index-table membership, not stored): `Active`, `Reserved(ReservationId)`, `Spent`, `Missing` |
+| `PostingFilter` | Read filter over derived posting state: `Active`, `Reserved`, `Live` (Active ∪ Reserved), `All` |
 | `Amount` | Parser/formatter for decimal strings. Not stored; use at API boundaries only |
-| `Posting` | Signed amount of one asset owned by one account. Has `status: PostingStatus` and `reservation: Option<ReservationId>` (owner token while `PendingInactive`) |
-| `ReservationId` | Owner token stamped on a reserved posting so only the reserving saga may finalize/release it |
+| `Posting` | Immutable signed amount of one asset owned by one account. Carries no lifecycle field; its state is derived from index-table membership (see `PostingState`) |
+| `ReservationId` | Owner token recorded in the reserved index so only the reserving saga may finalize/release a posting |
 | `NewPosting` | Posting to be created (no id yet, assigned during validation) |
 | `Transfer` | Atomic unit: consumes postings + creates postings + metadata |
 | `EnvelopeBuilder` | Fluent builder for `Transfer` construction |
@@ -48,8 +49,7 @@ in order:
 graph TD
     A[1. Non-empty] --> B[2. No duplicate consumes]
     B --> C[3. Posting existence]
-    C --> D[4. Posting active or reserved]
-    D --> E[5. Account existence & lifecycle]
+    C --> E[5. Account existence & lifecycle]
     E --> F[6. Snapshot pinning]
     F --> BP[7. Book policy]
     BP --> G[8. Per-asset conservation]
@@ -61,9 +61,9 @@ graph TD
 
 1. **Non-empty**: transfer must consume or create at least one posting
 2. **No duplicate consumes**: each posting consumed at most once
-3. **Posting existence**: every consumed posting exists in state
-4. **Posting active or reserved**: consumed postings must be `Active` or
-   `PendingInactive` (prevents double-spend)
+3. **Posting existence**: every consumed posting exists in the immutable table
+   (a `Posting` carries no lifecycle state; double-spend safety is enforced by
+   the reserve claim and the finalize "all spent" guard, not here)
 5. **Account existence & lifecycle**: all referenced accounts exist, not
    frozen, not closed
 6. **Snapshot pinning**: account snapshots (if provided) must match current
@@ -108,7 +108,7 @@ writes, then calls the dumb primitives, interpreting/verifying each count:
 graph LR
     A[resolve] -->|Envelope| W[save PendingSaga: Reserving]
     W --> B[reserve_postings]
-    B -->|Active→PendingInactive| F[finalize]
+    B -->|active index → reserved index| F[finalize]
     F --> V[validate_and_plan re-check]
     V --> M[mark Finalizing]
     M --> D[deactivate → insert → store_transfer → append_event]
@@ -157,13 +157,13 @@ Transfers are built via `TransferBuilder` and committed with
 
 | Method | Description |
 |--------|-------------|
-| `balance(account, asset)` | Sum of non-Inactive postings (computed by Ledger) |
+| `balance(account, asset)` | Sum of live (Active or Reserved) postings (computed by Ledger) |
 | `list_accounts()` | All current account snapshots |
 | `get_account(id)` | Latest account snapshot |
 | `query_transfers(query)` | Paginated, filtered transfer history (by date range, book) |
 | `history(account)` | All transfers involving an account |
-| `postings(account)` | All postings (any status) |
-| `query_postings(query)` | Paginated, filtered postings (by asset, status) |
+| `postings(account)` | All postings (any state) |
+| `query_postings(query)` | Paginated, filtered postings (by asset, `PostingFilter`) |
 | `account_history(id)` | All version snapshots |
 | `get_events_since(seq, limit)` | Query ledger event log after a sequence number |
 
@@ -185,8 +185,8 @@ graph TB
 
 - **`AccountStore`**: `get_account`, `get_accounts`, `create_account`,
   `append_account_version`, `get_account_history`, `list_accounts`
-- **`PostingStore`**: `get_postings`,
-  `get_postings_by_account(account, asset?, status?)`, `query_postings(query)`,
+- **`PostingStore`**: `get_postings`, `get_posting_states`,
+  `get_postings_by_account(account, sub?, asset?, filter)`, `query_postings(query)`,
   and the dumb write primitives `reserve_postings(ids, reservation) -> u64`,
   `release_postings(ids, reservation) -> u64`,
   `deactivate_postings(ids, reservation?) -> u64`,
@@ -210,24 +210,29 @@ recovery rather than a single transaction.
 conditional update and return how many rows changed (the saga decides what a
 short count means):
 
+State is derived from which index a posting is in: active index → `Active`,
+reserved index → `Reserved(rid)`, neither (only the immutable table) → `Spent`.
+Every transition is an insert/delete on an index table; the posting row never
+changes.
+
 ```mermaid
 stateDiagram-v2
     [*] --> Active: insert_postings
-    Active --> PendingInactive: reserve_postings
-    PendingInactive --> Active: release_postings
-    PendingInactive --> Inactive: deactivate_postings(reservation)
-    Active --> Inactive: deactivate_postings(None)
+    Active --> Reserved: reserve_postings
+    Reserved --> Active: release_postings
+    Reserved --> Spent: deactivate_postings(reservation)
+    Active --> Spent: deactivate_postings(None)
 ```
 
-Each cell is the count a primitive returns (1 = flipped, 0 = no-op / not
+Each cell is the count a primitive returns (1 = moved, 0 = no-op / not
 applicable). The saga interprets a 0:
 
-| Operation | Active | PendingInactive (this rid) | Inactive |
-|-----------|--------|----------------------------|----------|
-| `reserve_postings(rid)` | → PendingInactive (1) | 0 | 0 |
+| Operation | Active | Reserved (this rid) | Spent |
+|-----------|--------|---------------------|-------|
+| `reserve_postings(rid)` | → Reserved (1) | 0 | 0 |
 | `release_postings(rid)` | 0 | → Active (1) | 0 |
-| `deactivate_postings(Some rid)` | 0 | → Inactive (1) | 0 |
-| `deactivate_postings(None)` | → Inactive (1) | 0 | 0 |
+| `deactivate_postings(Some rid)` | 0 | → Spent (1) | 0 |
+| `deactivate_postings(None)` | → Spent (1) | 0 | 0 |
 
 There is no all-or-nothing batch rejection: a posting whose condition does not
 hold is skipped (counted as 0, not an error), so a call can apply to some ids
@@ -271,7 +276,7 @@ the saga derives meaning from them.
 
 | Step | Execute | Compensate | Retry |
 |------|---------|------------|-------|
-| `ReservePostingsStep` | `reserve_postings` `Active → PendingInactive`, interpret count | Release back to `Active` | 3 |
+| `ReservePostingsStep` | `reserve_postings`: active index → reserved index, interpret count | Release back to `Active` | 3 |
 | `FinalizeTransferStep` | `Ledger::finalize_envelope`: re-validate (last-step floor/freeze guard) → mark `Finalizing` → `deactivate` → `insert` → `store_transfer` → `append_event`, verifying every end-state | `reverse(transfer_id)` | 3 |
 
 Validation lives inside the finalize step so it runs immediately before the

+ 3 - 2
doc/transfers.md

@@ -216,8 +216,9 @@ validation steps are:
 
 1. Non-empty (must consume or create at least one posting)
 2. No duplicate consumed PostingIds
-3. All consumed postings exist
-4. All consumed postings are Active or PendingInactive
+3. All consumed postings exist in the immutable table (a `Posting` carries no
+   lifecycle state; double-spend safety is enforced by the reserve claim and the
+   finalize "all spent" guard, not by validation)
 5. All referenced accounts exist, not frozen, not closed
 6. Account snapshot pinning (if provided)
 7. Book policy (if a book is loaded): referenced assets/accounts/flags allowed