v0.4.0: Protocol v4 — header-driven sync, tiered engagement, multi-upstream

Protocol v4 sync overhaul:
- Slim PullSyncRequest: per-author timestamps (since_ms) replace full post ID lists
  Request size O(follows) instead of O(posts). Backward-compatible via serde default.
- Tiered pull frequency: 60s ticks, only syncs stale authors (4hr default)
  Full pull only on first tick (bootstrap). Most ticks skip — no stale authors.
- Tiered engagement checks: frequency scales with content age
  5min (<72h), 1hr (3-14d), 4hr (14-30d), 24hr (>30d)
  Single SQL query filters posts due for check.
- Header-driven post discovery: ManifestPush triggers PostFetch for missing
  followed-author posts (capped 10 per manifest). CDN tree = notification system.
- Multi-upstream (3 max): composite PK, priority ordering, engagement diffs
  sent to all upstreams, promote/remove on failure.

DB schema:
- follows.last_sync_ms — Self Last Encounter per author
- posts.last_engagement_ms — last reaction/comment timestamp
- posts.last_check_ms — last engagement check timestamp
- post_upstream: single-row → 3-row with priority column

Lock contention fixes:
- get_blob_for_post: 3 locks → 1
- prefetch_blobs_from_peer: lock-free blob checks
- fetch_engagement_from_peer: explicit lock release before I/O
- serve_post: 4 locks → 2 (eliminated redundant queries)
- run_replication_check: 2 locks → 1
- Badge cycle: N+2 IPC calls → 1 (get_badge_counts)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Scott Reimers 2026-03-21 16:13:45 -04:00
parent 1df00eebf8
commit bbaacf9b6c
10 changed files with 489 additions and 100 deletions

View file

@ -44,7 +44,7 @@
<p>This is the canonical technical reference for ItsGoin. It describes the vision, the architecture, and the current state of every subsystem &mdash; with full implementation detail. This document is versioned; each update records what changed.</p>
<div class="card" style="margin-top: 1rem;">
<strong style="font-size: 0.85rem; text-transform: uppercase; letter-spacing: 0.05em;">Changelog</strong>
<p style="margin-top: 0.5rem;"><strong>v0.4.0</strong> (planned): Protocol v4 &mdash; header-driven sync. ManifestPush as primary post notification. Slim PullSyncRequest (per-author timestamps, not full post ID list). Tiered engagement checks (5min/1hr/4hr/24hr by content age). Multi-upstream (3 max) with fallback chain. Auto-prefetch followed authors &lt;90d. Self Last Encounter per-author tracking. Encrypted-but-not-for-us CDN caching. Serial engagement polling. ~90% bandwidth reduction for established nodes.</p>
<p style="margin-top: 0.5rem;"><strong>v0.4.0</strong> (2026-03-21): Protocol v4 &mdash; header-driven sync. ManifestPush as primary post notification. Slim PullSyncRequest (per-author timestamps, not full post ID list). Tiered engagement checks (5min/1hr/4hr/24hr by content age). Multi-upstream (3 max) with fallback chain. Auto-prefetch followed authors &lt;90d. Self Last Encounter per-author tracking. Encrypted-but-not-for-us CDN caching. Serial engagement polling. ~90% bandwidth reduction for established nodes.</p>
<p><strong>v0.3.6</strong> (2026-03-20): Active CDN replication &mdash; all devices proactively replicate recent posts to peers (desktops &gt; anchors &gt; phones priority). ReplicationRequest/Response (0xE1/0xE2). Device roles (Intermittent/Available/Persistent) advertised in InitialExchange. Bandwidth budgets: replication (pull to cache) + delivery (serve requests), hourly auto-reset, phones 100MB/1GB, desktops 200MB/2GB, anchors 200MB/1GB. Cache management: 1GB default, configurable, eviction cycle activated with share-link priority boost. Engagement distribution fix &mdash; BlobHeader JSON rebuilt after diff ops. Tombstone system &mdash; deleted reactions/comments tombstoned, propagate via pull sync. Persistent notifications via seen_engagement/seen_messages tables. DOS hardening: fan-out cap (10), prefetch cap (20), downstream registration cap (50), delivery budget enforcement. Pull preference reordered: non-anchors first. Network indicator &mdash; header dot (black/red/yellow/green) + capability labels. Tab badges &mdash; contextual counts (new posts, engagement, online, unread). Message read tracking on open/close/send. Stats bar removed.</p>
<p><strong>v0.3.5</strong> (2026-03-20): Private blob encryption &mdash; attachments on encrypted posts (Friends/Circle/Direct) now encrypted with same CEK as post text; public blobs unchanged; CID on ciphertext. Blob prefetch on sync &mdash; attachments eagerly fetched after post pull for offline availability. Crypto refactoring &mdash; extracted reusable primitives (encrypt/decrypt_bytes_with_cek, unwrap_cek_for_recipient, unwrap_group_cek). Intent-based post filtering &mdash; feed/myposts/messages filter on intentKind instead of encryption state. Blob decryption API (get_blob_for_post). Download filename sanitization. Encrypted receipt &amp; comment slots &mdash; private posts carry noise-prefilled encrypted slots in BlobHeader for delivery/read/react receipts and private comments; CDN-propagated as opaque bytes; slot key derived from post CEK; 3 new BlobHeaderDiffOps (WriteReceiptSlot, WriteCommentSlot, AddCommentSlots). Message UI &mdash; DM delivery indicators (checkmark/double/blue/emoji), auto-seen on view, react button on messages.</p>
<p><strong>v0.3.4</strong> (2026-03-18): Comment edit &amp; delete with trust-based propagation. Native notifications via Tauri plugin (messages, posts, reactions, comments). Forward-compatible BlobHeaderDiffOp::Unknown variant. Following Online/Offline lightbox. Comment threading scoping fix. Dropdown text legibility fix. Mobile hamburger nav for website.</p>
@ -1147,7 +1147,7 @@ FAILURE: C &rarr; B &rarr; A: AnchorProbeResult { reachable: false }</code></pre
<p>DM conversations display delivery indicators: single checkmark (sent), double checkmark (delivered/on device), blue double checkmark (seen), emoji (reacted). Opening a conversation auto-marks incoming messages as seen. Messages have a react button for emoji responses.</p>
</section>
<h3>Protocol v4: Header-Driven Sync <span class="badge badge-planned">Planned</span></h3>
<h3>Protocol v4: Header-Driven Sync <span class="badge badge-complete">Complete</span></h3>
<p>Major sync protocol revision that replaces the current pull-everything-from-everyone model with header-driven discovery, per-author tracking, and tiered engagement polling. Reduces bandwidth by ~90% for established nodes.</p>
<h4>Core principle: headers as notification</h4>