Design doc: Protocol v4 spec — header-driven sync, tiered engagement, multi-upstream

New section documenting the planned v0.4.0 protocol revision:
- ManifestPush as primary post notification (CDN tree = notification system)
- Slim PullSyncRequest: per-author timestamps replace full post ID lists
- Self Last Encounter: per-author sync tracking (implements v0.2.0 design intent)
- Tiered engagement checks: 5min (<72h) / 1hr (3-14d) / 4hr (14-30d) / 24hr (>30d)
- Multi-upstream: 3 max per post with fallback chain and BlobDeleteNotice unregistration
- Auto-prefetch followed authors <90d from header discovery
- Encrypted-but-not-for-us CDN caching with natural decay
- Serial engagement polling (one peer, authoritative response)
- Header-driven post discovery flow (7 steps)
- Migration path: backward-compatible via ALPN, v3 fallback for mixed networks
- DB schema additions: last_engagement_ms, last_check_ms, last_sync_ms

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

website/design.html

@@ -44,7 +44,8 @@
             <p>This is the canonical technical reference for ItsGoin. It describes the vision, the architecture, and the current state of every subsystem &mdash; with full implementation detail. This document is versioned; each update records what changed.</p>
             <div class="card" style="margin-top: 1rem;">
                 <strong style="font-size: 0.85rem; text-transform: uppercase; letter-spacing: 0.05em;">Changelog</strong>
-                <p style="margin-top: 0.5rem;"><strong>v0.3.6</strong> (2026-03-20): Active CDN replication &mdash; all devices proactively replicate recent posts to peers (desktops &gt; anchors &gt; phones priority). ReplicationRequest/Response (0xE1/0xE2). Device roles (Intermittent/Available/Persistent) advertised in InitialExchange. Bandwidth budgets: replication (pull to cache) + delivery (serve requests), hourly auto-reset, phones 100MB/1GB, desktops 200MB/2GB, anchors 200MB/1GB. Cache management: 1GB default, configurable, eviction cycle activated with share-link priority boost. Engagement distribution fix &mdash; BlobHeader JSON rebuilt after diff ops. Tombstone system &mdash; deleted reactions/comments tombstoned, propagate via pull sync. Persistent notifications via seen_engagement/seen_messages tables. DOS hardening: fan-out cap (10), prefetch cap (20), downstream registration cap (50), delivery budget enforcement. Pull preference reordered: non-anchors first. Network indicator &mdash; header dot (black/red/yellow/green) + capability labels. Tab badges &mdash; contextual counts (new posts, engagement, online, unread). Message read tracking on open/close/send. Stats bar removed.</p>
+                <p style="margin-top: 0.5rem;"><strong>v0.4.0</strong> (planned): Protocol v4 &mdash; header-driven sync. ManifestPush as primary post notification. Slim PullSyncRequest (per-author timestamps, not full post ID list). Tiered engagement checks (5min/1hr/4hr/24hr by content age). Multi-upstream (3 max) with fallback chain. Auto-prefetch followed authors &lt;90d. Self Last Encounter per-author tracking. Encrypted-but-not-for-us CDN caching. Serial engagement polling. ~90% bandwidth reduction for established nodes.</p>
+                <p><strong>v0.3.6</strong> (2026-03-20): Active CDN replication &mdash; all devices proactively replicate recent posts to peers (desktops &gt; anchors &gt; phones priority). ReplicationRequest/Response (0xE1/0xE2). Device roles (Intermittent/Available/Persistent) advertised in InitialExchange. Bandwidth budgets: replication (pull to cache) + delivery (serve requests), hourly auto-reset, phones 100MB/1GB, desktops 200MB/2GB, anchors 200MB/1GB. Cache management: 1GB default, configurable, eviction cycle activated with share-link priority boost. Engagement distribution fix &mdash; BlobHeader JSON rebuilt after diff ops. Tombstone system &mdash; deleted reactions/comments tombstoned, propagate via pull sync. Persistent notifications via seen_engagement/seen_messages tables. DOS hardening: fan-out cap (10), prefetch cap (20), downstream registration cap (50), delivery budget enforcement. Pull preference reordered: non-anchors first. Network indicator &mdash; header dot (black/red/yellow/green) + capability labels. Tab badges &mdash; contextual counts (new posts, engagement, online, unread). Message read tracking on open/close/send. Stats bar removed.</p>
                 <p><strong>v0.3.5</strong> (2026-03-20): Private blob encryption &mdash; attachments on encrypted posts (Friends/Circle/Direct) now encrypted with same CEK as post text; public blobs unchanged; CID on ciphertext. Blob prefetch on sync &mdash; attachments eagerly fetched after post pull for offline availability. Crypto refactoring &mdash; extracted reusable primitives (encrypt/decrypt_bytes_with_cek, unwrap_cek_for_recipient, unwrap_group_cek). Intent-based post filtering &mdash; feed/myposts/messages filter on intentKind instead of encryption state. Blob decryption API (get_blob_for_post). Download filename sanitization. Encrypted receipt &amp; comment slots &mdash; private posts carry noise-prefilled encrypted slots in BlobHeader for delivery/read/react receipts and private comments; CDN-propagated as opaque bytes; slot key derived from post CEK; 3 new BlobHeaderDiffOps (WriteReceiptSlot, WriteCommentSlot, AddCommentSlots). Message UI &mdash; DM delivery indicators (checkmark/double/blue/emoji), auto-seen on view, react button on messages.</p>
                 <p><strong>v0.3.4</strong> (2026-03-18): Comment edit &amp; delete with trust-based propagation. Native notifications via Tauri plugin (messages, posts, reactions, comments). Forward-compatible BlobHeaderDiffOp::Unknown variant. Following Online/Offline lightbox. Comment threading scoping fix. Dropdown text legibility fix. Mobile hamburger nav for website.</p>
                 <p><strong>v0.3.3</strong> (2026-03-16): Connection rate limiting &mdash; incoming auth failures rate-limited per source IP (3 attempts, exponential backoff to ~256s). Schema versioning &mdash; PRAGMA user_version tracks DB version with migration framework. N2/N3 freshness &mdash; TTL 7d&rarr;5h, full N1/N2 re-broadcast every 4h, startup sweep clears stale entries. Bootstrap isolation recovery &mdash; 24h check verifies bootstrap is in N1/N2/N3, reconnects + sticky N1 advertisement if absent. IPv6 HTTP address fix &mdash; nodes advertise actual public IPv6 (not 0.0.0.0) for share link redirects. Upstream tracking &mdash; post_upstream table records post source for engagement diff routing toward author. Video preload fix &mdash; share links and in-app videos use preload=auto. Following Online/Offline split. DM filter from My Posts. Any-type file attachments with download prompt + trust warning. Image lightbox. Audio player.</p>
@@ -1146,6 +1147,98 @@ FAILURE: C &rarr; B &rarr; A: AnchorProbeResult { reachable: false }</code></pre
             <p>DM conversations display delivery indicators: single checkmark (sent), double checkmark (delivered/on device), blue double checkmark (seen), emoji (reacted). Opening a conversation auto-marks incoming messages as seen. Messages have a react button for emoji responses.</p>
         </section>
 
+            <h3>Protocol v4: Header-Driven Sync <span class="badge badge-planned">Planned</span></h3>
+            <p>Major sync protocol revision that replaces the current pull-everything-from-everyone model with header-driven discovery, per-author tracking, and tiered engagement polling. Reduces bandwidth by ~90% for established nodes.</p>
+
+            <h4>Core principle: headers as notification</h4>
+            <p>The AuthorManifest already carries a post neighborhood (20 previous + 20 following PostIds). When this neighborhood is pushed via <code>ManifestPush</code> (0x94) or travels with blob responses, receiving nodes can diff their local post list against the neighborhood to discover new posts without a full pull sync. The CDN tree becomes the notification system.</p>
+
+            <h4>Self Last Encounter (per-author sync tracking)</h4>
+            <p>Implements the v0.2.0 design intent: track <code>last_sync_ms</code> per followed author. Pull sync triggers only when <code>now - last_sync_ms > check_interval</code>. Updated on:</p>
+            <ul style="padding-left: 1.25rem; margin: 0.5rem 0; color: var(--text-muted);">
+                <li>Successful pull sync response containing that author's posts</li>
+                <li>ManifestPush/BlobHeaderDiff received for that author</li>
+                <li>Blob transfer that includes that author's file header</li>
+            </ul>
+
+            <h4>Slim PullSyncRequest</h4>
+            <p>Current: <code>{ follows: Vec&lt;NodeId&gt;, have_post_ids: Vec&lt;PostId&gt; }</code> &mdash; sends ALL post IDs every request.</p>
+            <p>v4: <code>{ follows: Vec&lt;NodeId&gt;, since_ms: HashMap&lt;NodeId, u64&gt; }</code> &mdash; per-author timestamps. Response contains only posts newer than the requester's timestamp for each author. Drops request size from O(posts) to O(follows).</p>
+
+            <h4>Tiered pull sync frequency</h4>
+            <p>Instead of pulling from ALL mesh peers every 5 minutes, pull is driven by Self Last Encounter staleness:</p>
+            <ul style="padding-left: 1.25rem; margin: 0.5rem 0; color: var(--text-muted);">
+                <li>Pull only triggers for authors where <code>last_sync_ms</code> is stale AND no header updates were received in the interval</li>
+                <li>Default check interval: <strong>4 hours</strong> (reduced from current 5-minute blanket pull)</li>
+                <li>Pull targets known CDN sources first (peers in <code>post_upstream</code> / <code>post_downstream</code> for that author), falling back to mesh peers only if CDN sources are unavailable</li>
+                <li>Serial, not parallel &mdash; one peer at a time, stop when delta is received</li>
+            </ul>
+
+            <h4>Tiered engagement check rates</h4>
+            <p>Engagement header polling (<code>BlobHeaderRequest</code> 0xD1) frequency scales with content age and activity:</p>
+            <table>
+                <tr><th>Content age / last engagement</th><th>Check interval</th></tr>
+                <tr><td>&lt; 72 hours</td><td>5 minutes</td></tr>
+                <tr><td>3&ndash;14 days</td><td>1 hour</td></tr>
+                <tr><td>14&ndash;30 days</td><td>4 hours</td></tr>
+                <tr><td>&gt; 30 days</td><td>24 hours</td></tr>
+            </table>
+            <p>DB tracks <code>last_engagement_ms</code> and <code>last_check_ms</code> per post. A single SQL query filters posts due for check:</p>
+            <pre><code>SELECT post_id FROM posts
+WHERE last_check_ms &lt; now_ms - CASE
+  WHEN last_engagement_ms &gt; now_72h THEN 300000
+  WHEN last_engagement_ms &gt; now_14d THEN 3600000
+  WHEN last_engagement_ms &gt; now_30d THEN 14400000
+  ELSE 86400000
+END</code></pre>
+            <p>Checks are serial (one peer at a time). A single &ldquo;no new engagement&rdquo; response is treated as authoritative &mdash; if that peer missed an update, it would have been replicated to them by now.</p>
+            <p>On connect/wake: if <code>since_last_check &gt; check_rate</code> for any post, an automatic check runs immediately.</p>
+
+            <h4>Multi-upstream (3 max)</h4>
+            <p>Extend <code>post_upstream</code> from 1 entry to up to 3 per post:</p>
+            <ul style="padding-left: 1.25rem; margin: 0.5rem 0; color: var(--text-muted);">
+                <li><strong>Primary upstream</strong>: First source of the post (current behavior)</li>
+                <li><strong>Backup upstreams</strong>: Additional sources discovered via CDN tree or replication</li>
+                <li><strong>Upstream unavailable</strong>: If primary is unreachable, promote backup. If no backups, use the CDN node map to find the author&rsquo;s upstream chain and request connection &mdash; accepted or directed to an alternate via RefuseRedirect</li>
+                <li><strong>Unregistration</strong>: <code>BlobDeleteNotice</code> (0x95) already handles &ldquo;I no longer hold this.&rdquo; Extend to upstream: &ldquo;I&rsquo;m no longer your upstream for this post.&rdquo;</li>
+                <li>Overhead is minimal: 3 rows per post vs 100 rows per post for downstream</li>
+            </ul>
+
+            <h4>Auto-prefetch followed authors</h4>
+            <p>When a post by a followed author appears in any header (PullSyncResponse, ManifestPush, BlobHeaderDiff neighborhood), prefetch the post and its blobs if:</p>
+            <ul style="padding-left: 1.25rem; margin: 0.5rem 0; color: var(--text-muted);">
+                <li>Post age &lt; 90 days</li>
+                <li>Post not already in local storage</li>
+                <li>Within replication budget</li>
+            </ul>
+            <p><strong>Encrypted posts without key</strong>: Store the encrypted post in DB marked as &ldquo;not-for-us.&rdquo; The node contributes to CDN availability for those who CAN decrypt it. Normal cache decay handles cleanup &mdash; these posts are given low eviction priority (stranger relationship score of 0.1) and will be evicted when space is needed.</p>
+
+            <h4>Header-driven post discovery flow</h4>
+            <ol style="padding-left: 1.25rem; margin: 0.5rem 0; color: var(--text-muted);">
+                <li>Author creates post &rarr; updates their AuthorManifest neighborhood (20 before + 20 after)</li>
+                <li>ManifestPush (0x94) propagates to downstream CDN tree</li>
+                <li>Each receiving node diffs the neighborhood against local post list</li>
+                <li>New post IDs discovered &rarr; fetch via PostFetch (0xD4/0xD5) from the peer that sent the manifest</li>
+                <li>Blobs prefetched for followed authors (within budget)</li>
+                <li>Node registers as downstream for the new post</li>
+                <li>Self Last Encounter updated for the author &rarr; suppresses pull sync for that author</li>
+            </ol>
+            <p>This makes the CDN tree the primary content distribution channel, with pull sync serving only as a safety net for missed headers.</p>
+
+            <h4>Migration path</h4>
+            <p>v4 is backward-compatible via ALPN negotiation. Nodes running v3 continue to work with the existing pull model. v4 nodes detect peer capability during InitialExchange and use the optimized paths when both sides support them. The v3 pull cycle remains as a fallback for mixed-version networks.</p>
+
+            <h4>New DB columns required</h4>
+            <table>
+                <tr><th>Table</th><th>Column</th><th>Purpose</th></tr>
+                <tr><td><code>posts</code></td><td><code>last_engagement_ms</code></td><td>Timestamp of most recent reaction/comment</td></tr>
+                <tr><td><code>posts</code></td><td><code>last_check_ms</code></td><td>Timestamp of last engagement check</td></tr>
+                <tr><td><code>follows</code></td><td><code>last_sync_ms</code></td><td>Self Last Encounter per followed author</td></tr>
+                <tr><td><code>post_upstream</code></td><td>(expand to 3 rows)</td><td>Multi-upstream with priority</td></tr>
+            </table>
+
+        </section>
+
         <!-- 20. Encryption -->
         <section id="encryption">
             <h2>20. Encryption</h2>