Cross-Site S3 Sync with SeaweedFS on FreeBSD: Push From Anywhere, Pull at Home (Part 1: The Idea)
I wanted an easy way to back up my phone’s photos — not just at home on the same Wi-Fi as my laptop, but from anywhere. A private S3 bucket fits nicely: it’s just standard S3, so almost anything can talk to it.
On the phone I use PhotoSync. One tap backs up the whole library straight to an S3 endpoint. Wherever I am, it uploads to a public box. Then, in the background, a full copy lands back home on my own network — no internet needed once it’s there.
Here’s the shape of it.
Two boxes
phone uploads here (public) pulled down here (home, NAT)
s3a.example.org ───────────────────────────────────────────────►
content: A → B only
┌──────────────┐ ┌──────────────┐
│ site A │ ◄────────────────────────────────── │ site B │
│ swa (public)│ deletions: B → A │ swb (home) │
└──────────────┘ └──────────────┘
every cross-site connection is opened by B (outbound only)
Site A is a small box on the internet running SeaweedFS with its S3 gateway exposed. That’s what the phone talks to. Site B sits at home behind NAT and pulls down everything A receives, so I end up with a full local copy my other machines can read over the LAN.
That’s the whole idea: push to the public box from anywhere, pull it down at home.
Why pull, not push
B is behind NAT. It can dial out, but nothing on the internet can reach in. So B does all the talking — it connects to A and pulls. A only has to open its filer and S3 ports to B’s outbound IP and nothing else. No port-forwarding at home, no VPN needed just for the sync.
One box per site, on purpose
The “proper” design is two nodes per site so SeaweedFS mirrors between them. I skipped that. A two-node master group needs both halves alive to agree, and the second they can’t see each other you get split-brain. One node per site kills that whole class of problem — a lone master is always its own quorum.
The trade is honest: each site holds one copy. My safety net is two independent sites plus normal backups, not redundancy inside a box. If a disk dies, that site’s copy is gone until I restore or re-pull. For photos that already live in two places, I’m fine with that.
What flows where
Content only goes A → B. Files land on A, show up on B a moment later. Dropping a file straight onto B does nothing to A — it’s a mirror, not a second uploader.
The one thing that travels back upstream is deletions. Delete a photo on B and the copy on A goes too. So B doubles as a cleanup surface: prune the local copy and the public box gets pruned with it. Deletes I make on A still come down to B as usual.
Why the split? filer.sync replays a whole changelog per direction — creates and deletes together — so it can’t do “content one way, deletes the other.” Content stays on a single pull stream, and a tiny watcher handles the deletes going back the other way.
The moving parts
Two small things run on the home box:
weed filer.syncsubscribes to A’s change stream and writes every new file into B. Pull-only, so NAT stays out of the way.- A tiny delete watcher tails B’s own log and, when I delete something on B, deletes the same key on A. It only reacts to deletions I make on B — not to creates, and not to the deletes coming down from A — so it never loops, and it can’t nuke a file that just hasn’t synced yet.
Both sites also fire a webhook when a new file lands (S3’s ObjectCreated, basically), handy if you want to trigger something when a photo shows up.
The full build — configs, firewall, rc scripts, and pointing PhotoSync at it — is in Part 2: The Build.