Personal Second Brain: Architecture & Agent Playbook

01 · Principles

The rules that keep it trustworthy

Six commitments do most of the work. The load-bearing one: nothing becomes a fact until it has been scored and, when uncertain, validated by you.

Agent works, you own the truthIt proposes; you validate. No uncertain fact is committed as certain.

Raw is quarantinedEvery source lands in a raw layer first, with provenance. Nothing there is treated as true.

Local-first & privateThe vault lives on hardware you control. Sensitive data never leaves it unencrypted or hits a vendor cloud.

Facts ≠ goals ≠ synthesisWhat is true, what you want, and what the agent infers are stored separately.

Mesh-portableThe agent runs wherever you are, with the same credentials and capabilities. The network, not the device, is the anchor.

Never fabricatecite source · score confidence · ask, do not guess

02 · Raw → validated

The misinformation firewall

A second brain is only as trustworthy as the gate in front of it. Every source flows into a quarantined raw layer, gets scored for confidence, and must pass a validation gate before it becomes a fact. The gate is scoring plus you.

Sources land in raw, get scored, and pass a gate. Clean and high-confidence promotes automatically; uncertain or sensitive waits for you; fabricated or unverifiable is rejected and never enters the brain.

Why the raw layer. Keeping ingested data quarantined with full provenance means you can always trace a claim back to exactly where it came from, re-score it later, or revoke it. A fact with no source is a liability, so the system refuses to create one.

Why it matters. An LLM that writes confident-sounding guesses into your knowledge base poisons every future answer built on it. Scoring confidence and requiring human validation for anything uncertain is what keeps hallucination, misattribution, and stale rumor out of the second brain.

03 · The pieces

The parts, in plain English

The vault sits on an always-on server. The agent runs on whichever device you are using, reaches the vault over the mesh, and runs a durable session to do its work.

Thin clients

Your laptop and phone. Windows into the brain, not the warehouse. They store little and can be lost or swapped without losing anything.

Why: the data is on the server; the client just connects in over the mesh.

Always-on server

A Mac mini (M4, 24GB RAM, 512GB SSD) that never sleeps. It holds the vault and the shared Keychain (Cloudflare, GitHub, MCP tokens), and runs the toolchain (Node.js, wrangler, gh CLI) the agent uses to deploy and push.

Why: the brain and your credentials have to be there even when your laptop is closed.

The agent runs anywhere

Because the WireGuard session is always up, the agent can run on the laptop, the phone, or the server itself. /remote-control hands it between devices. Location stops mattering.

Why: the persistent network and the always-on vault are the constants, not any one device.

Tailscale

A private mesh VPN on WireGuard. Every device joins your tailnet and talks directly over encrypted tunnels, reachable by name, with nothing exposed to the public internet.

Why: the persistent, encrypted backbone that makes the agent portable.

mosh

Mobile Shell. It logs in over SSH, then runs the session over UDP, surviving wifi-to-cellular handoffs, sleep, and dead zones, reconnecting on its own with instant local echo.

Why: a roaming connection that does not drop when your network does.

tmux

A terminal multiplexer on the server. It keeps the session alive and reattachable, so work persists across disconnects, restarts, and days away.

Why: mosh keeps the link alive; tmux keeps the work alive. Reattach exactly where you left off.

04 · Connectivity stack

Reaching the vault from a roaming device

When the agent runs on a phone or laptop, this is the session it uses to reach the brain. One command, roaming-proof, encrypted, and exactly where you left off.

mosh handles roaming over UDP, SSH proves who you are, tmux keeps the work alive, and Tailscale wraps it all in an encrypted tunnel with no public exposure.

05 · Infra in the brain

The capability layer

The brain does not just remember, it acts. Your infrastructure (Cloudflare, GitHub, MCP servers, active projects) is a domain in the vault, and the tokens that make it work live in the server's Keychain. Any device's agent inherits the same capability, so you never rebuild your setup, and the agent keeps working when your laptop is closed.

Tokens live once, in the server's Keychain. The agent on any node uses them over the mesh to act as you, and the infra domain tells it how.

Same setup everywhere

Move from laptop to phone to server and the agent already has Cloudflare, GitHub, and your MCP servers wired. No re-auth, no re-config, no copying keys around.

Why: the credentials and runbooks live with the brain, not on the device you happen to be holding.

Works when you are away

A server-resident agent uses the server Keychain to deploy, commit, or run jobs on a schedule, even with your laptop closed and in your bag.

Why: autonomy needs credentials that are present without you.

Nothing on the device

Tokens stay in the server Keychain. Lose the phone or the laptop and you lose no secret. The vault records only where each token lives.

Why: minimal blast radius, and the privacy lint enforces location-not-value.

Example. From your phone: "ship this webapp." The agent pushes with gh and deploys with wrangler on Node.js, using the Cloudflare and GitHub tokens in the server's Keychain, and it is live in seconds. This very page went live exactly that way, from a Keychain-held token with no setup on the device.

06 · Facts, goals & synthesis

Three kinds of knowledge, kept apart

Mixing aspiration with reality is how a second brain rots into noise. Separating them lets the agent measure progress and surface signal, on top of the validated facts the firewall lets through.

Facts plus goals yield synthesis. Stored as distinct node types, the agent can ask "are you on track?" and never mistake a stale wish for a current fact.

07 · Ingestion

What the agent pulls in

Reliable, API-backed sources beat scraping. Each pipeline lands in raw, gets scored, and asks you about the rest. Sensitive pipelines store derived insight, not raw content.

workspace-mcp

The self-hosted Google Workspace MCP. Authenticated, structured access to Gmail, Drive, Contacts, and Calendar through the official APIs.

Why: clean, typed data with no scraping and no third-party broker in the middle.

m365-cli

The Microsoft 365 CLI. Scripted, authenticated pulls from Outlook mail, SharePoint, OneDrive, Teams, and Entra.

Why: the same reliable, API-backed ingestion for the Microsoft side of your life.

People & relationships

A clean contact book gives identity; message history gives relationship strength. Edges derive from ~/Library/Messages/chat.db: volume, reciprocity, recency, cadence.

Privacy: store the edges, never the message bodies. Working copies sit on the encrypted volume.

Family & homeschooling

Households, guardians, and children as linked nodes. Per-child learning profiles connect to curriculum and records, with term goals stored as goal nodes.

Privacy: one canonical node per person, links instead of copies.

Finance

Official exports over aggregators that retain your data. Parsed locally into derived insight: allocation, drift, cash flow, net-worth trend, linked to goals.

Privacy: account numbers and balances stay gated; credentials stay in Keychain.

Linting (the QA pass)

Scheduled checks: schema, link integrity and orphans, duplicates, staleness re-confirm, provenance, and fact-versus-goal mislabeling.

Privacy lint: notes hold the location of a secret, never its value.

08 · Privacy posture

Defense in depth

Four boundaries wrap the data, from the network down to individual secrets. Nothing is published to the open internet, and the most sensitive material sits behind the most protection.

Network, disk, sensitive volume, and secrets. Four layers, least-privilege by default, with nothing exposed publicly.

09 · For Matt Raider

Deploy your own

The reference stack, then the build order. Build the raw layer and the validation gate first, so every later step accumulates validated facts rather than errors.

The server. A Mac mini (M4, 24GB RAM, 512GB SSD), always on. Quiet, low-power, and far more than enough for the vault, the agent, and the ingestion jobs running together.

The toolchain. On the mini: Node.js (runtime), wrangler (Cloudflare deploys), gh CLI (GitHub), Tailscale (mesh), mosh and tmux (durable sessions). All tokens in Keychain.

1 · Vault and schemas

Stand up the Markdown vault, the taxonomy, the raw layer, and node schemas for facts, goals, and syntheses. Choose the always-on server.

2 · Raw → validate gate

Wire ingestion into raw, confidence scoring, and the human validation gate before anything else.

3 · Reliable sources

workspace-mcp (Google) and m365-cli (Microsoft), then contacts, iMessage edges, finance, and homeschool.

4 · Facts, goals, synthesis

Promote validated facts, capture goals, and let the agent synthesize against them.

5 · Linting

Schedule the QA pass, including the privacy lint.

6 · Security

FileVault, encrypted volume, Keychain.

7 · Mesh and access

Tailscale with ACLs, SSH keys in Keychain, mosh plus tmux, and /remote-control so the agent runs on any node.

8 · Toolchain & shared credentials

Install Node.js, wrangler, and gh CLI on the mini; put Cloudflare, GitHub, and MCP tokens in the server Keychain; document each as an infra node (location, not value), so any device's agent inherits the same capability.

9 · Automate

Schedule the recurring passes: ingest to raw, score, validate, lint, synthesize.