The why is in Part 1: The Idea. This is just the build.

  • Public site A (the phone’s write target) + internal NATed site B (pull mirror).
  • Content flows A→B; deletions made on B are replayed to A.
  • Bucket: data.
  • Done on SeaweedFS 4.x and FreeBSD 15.1-RELEASE.

Hosts

Host IP Role
swa 10.10.0.11 public write target, s3a.example.org
swb 192.168.20.11 home box, NAT egress 203.0.113.50

Ports

9333/19333 master 8080/18080 volume — localhost only 8888/18888 filer — swa allows 203.0.113.50 only 9090 S3 HTTPS 8333 blocked 8181 disabled

1. Both boxes

pkg install seaweedfs rclone jq
sysrc ntpd_enable=YES ntpd_sync_on_start=YES
service ntpd start
mkdir -p /var/db/seaweedfs/data
chown -R seaweedfs:seaweedfs /var/db/seaweedfs

Configs go in /usr/local/etc/seaweedfs/. Drop a TLS cert/key per site at seaweedfs.crt / .key.

2. swa (public)

# /etc/rc.conf
seaweedfs_enable="YES"
seaweedfs_user="seaweedfs"
seaweedfs_group="seaweedfs"
seaweedfs_options="server \
  -ip=10.10.0.11 \
  -dir=/var/db/seaweedfs/data \
  -volume.max=200 \
  -s3 \
  -s3.config=/usr/local/etc/seaweedfs/s3.json \
  -s3.cert.file=/usr/local/etc/seaweedfs/seaweedfs.crt \
  -s3.key.file=/usr/local/etc/seaweedfs/seaweedfs.key \
  -s3.port.https 9090 \
  -s3.port.iceberg=0"
seaweedfs_syslog_output_enable="YES"

service seaweedfs start
echo "s3.bucket.create -name data" | weed shell -master=10.10.0.11:9333

3. swb (home)

Same block, IP 192.168.20.11.

service seaweedfs start
echo "s3.bucket.create -name data" | weed shell -master=192.168.20.11:9333

4. S3 identities

⚠️ MAKE SURE TO CHANGE THE SECRET KEYS! ⚠️

s3.json on swa — syncuser needs Write:data (delete is part of Write):

{
  "identities": [
    { "name": "admin", "credentials": [{ "accessKey": "AdminAccessA", "secretKey": "AdminSecretA" }],
      "actions": [ "Admin", "Read", "List", "Tagging", "Write" ] },
    { "name": "syncuser", "credentials": [{ "accessKey": "SyncAccessA", "secretKey": "SyncSecretA" }],
      "actions": [ "List:data", "Read:data", "Write:data" ] }
  ]
}

s3.json on swb — deletions on B done with admin:

{
  "identities": [
    { "name": "admin", "credentials": [{ "accessKey": "AdminAccessB", "secretKey": "AdminSecretB" }],
      "actions": [ "Admin", "Read", "List", "Tagging", "Write" ] },
    { "name": "syncuser", "credentials": [{ "accessKey": "SyncAccessB", "secretKey": "SyncSecretB" }],
      "actions": [ "List:data", "Read:data" ] }
  ]
}

Restart seaweedfs on both.

5. New-file webhook (both sites)

weed scaffold -config=notification > /usr/local/etc/seaweedfs/notification.toml

[notification.webhook]
enabled         = true
endpoint        = "https://events.example.org/seaweed-hook"
bearer_token    = "change-me-per-site"
event_types     = ["create"]
path_prefixes   = ["/buckets/data"]
timeout_seconds = 10
max_retries     = 3

Restart the filer. Have the receiver ignore keys with /.uploads/ in them (multipart parts).

6. pf on swa

ext_if   = "vtnet0"
b_egress = "203.0.113.50"

pass in quick on $ext_if proto tcp from $b_egress to ($ext_if) port { 8888, 18888 }
pass in quick on $ext_if proto tcp from $b_egress to ($ext_if) port 9090
pass in quick on $ext_if proto tcp to ($ext_if) port 9090            # S3 for the phone / apps
block in quick on $ext_if proto tcp to ($ext_if) port { 8333, 9333, 19333, 8080, 18080 }

# from swb
nc -vz s3a.example.org 8888
nc -vz s3a.example.org 18888

7. Content sync A→B (on swb)

#!/bin/sh
# /usr/local/etc/rc.d/seaweedsync
# PROVIDE: seaweedsync
# REQUIRE: NETWORKING seaweedfs
# KEYWORD: shutdown

. /etc/rc.subr
name=seaweedsync
rcvar=seaweedsync_enable
load_rc_config $name
: ${seaweedsync_enable:="NO"}
: ${seaweedsync_user:="seaweedfs"}
: ${seaweedsync_options:=""}
pidfile="/var/run/${name}.pid"
command="/usr/sbin/daemon"
command_args="-P ${pidfile} -R 15 -S -T ${name} -u ${seaweedsync_user} \
    /usr/local/bin/weed filer.sync ${seaweedsync_options}"
run_rc_command "$1"

chmod 755 /usr/local/etc/rc.d/seaweedsync

# /etc/rc.conf on swb, appended
seaweedsync_enable="YES"
seaweedsync_options="-isActivePassive \
  -a s3a.example.org:8888 -a.filerProxy \
  -b 127.0.0.1:8888 \
  -a.path /buckets/data -b.path /buckets/data"

Want files to stick around on B after you delete them on A? Append -b.doDeleteFiles=false. Already have photos on A? Add -a.fromTsMs=1 on the first run, or run rclone copy primary:data secondary:data once.

service seaweedsync start
tail -f /var/log/messages | grep seaweedsync

8. Delete replay B→A (on swb)

⚠️ This isn’t the most solid way of cleaning up your node A ⚠️ If you have more solid ideas, let me know and I will create an updated version of this how-to.

/usr/local/etc/seaweedfs/rclone.conf (chmod 600, owner seaweedfs):

[primary]
type = s3
provider = SeaweedFS
endpoint = https://s3a.example.org:9090
access_key_id = SyncAccessA
secret_access_key = SyncSecretA

#!/bin/sh
# /usr/local/sbin/seaweedfs-delprop.sh
set -eu
RCLONE=/usr/local/bin/rclone
CONF=/usr/local/etc/seaweedfs/rclone.conf

/usr/local/bin/weed filer.meta.tail -filer=127.0.0.1:8888 -pathPrefix=/buckets/data -timeAgo=10m \
| jq --unbuffered -r '
    select(.eventNotification.oldEntry != null
       and .eventNotification.newEntry == null
       and .eventNotification.isFromOtherCluster == false
       and .eventNotification.oldEntry.isDirectory == false)
    | .directory + "/" + .eventNotification.oldEntry.name' \
| while IFS= read -r path; do
    key=${path#/buckets/data/}
    [ "$key" = "$path" ] && continue
    "$RCLONE" deletefile "primary:data/${key}" --config "$CONF" --s3-no-check-bucket || true
  done

Self-signed cert on A: add --no-check-certificate to the deletefile line. The isFromOtherCluster == false filter means it only fires on deletions you make on B; -timeAgo=10m replays recent events on restart (deletes are idempotent).

#!/bin/sh
# /usr/local/etc/rc.d/seaweeddelprop
# PROVIDE: seaweeddelprop
# REQUIRE: NETWORKING seaweedfs
# KEYWORD: shutdown

. /etc/rc.subr
name=seaweeddelprop
rcvar=seaweeddelprop_enable
load_rc_config $name
: ${seaweeddelprop_enable:="NO"}
: ${seaweeddelprop_user:="seaweedfs"}
pidfile="/var/run/${name}.pid"
command="/usr/sbin/daemon"
command_args="-P ${pidfile} -R 15 -S -T ${name} -u ${seaweeddelprop_user} \
    /usr/local/sbin/seaweedfs-delprop.sh"
run_rc_command "$1"

chmod 755 /usr/local/etc/rc.d/seaweeddelprop /usr/local/sbin/seaweedfs-delprop.sh
sysrc seaweeddelprop_enable=YES
service seaweeddelprop start

9. Check it works

primaryadmin / secondaryadmin are your own admin S3 remotes for A and B.

rclone copyto ./hi.txt primaryadmin:data/hi.txt
rclone ls secondary:data                 # appears on B

rclone deletefile secondaryadmin:data/hi.txt
rclone ls primaryadmin:data              # gone from A within seconds

weed filer.sync.verify -isActivePassive \
  -a s3a.example.org:8888 -b 127.0.0.1:8888 \
  -a.path /buckets/data -b.path /buckets/data \
  -modifiedTimeAgo 15m

10. Point PhotoSync at it

On the phone, add an S3 target in PhotoSync:

  • Endpoint: https://s3a.example.org:9090
  • Bucket: data
  • Access / secret key: one with Write:data on A — use admin, or add a dedicated identity in s3.json
  • SSL: on

Hit sync. Photos upload to the public box from wherever you are, and the home box pulls a full copy down in the background.

11. Harden

mTLS: weed scaffold -config=security on both, fill [grpc.*], add -a.security=/path/to/securityA.toml to filer.sync.

Or WireGuard swb → swa; point filer.sync and the primary rclone endpoint at the tunnel address and close 8888/18888 on the public interface.

12. When something’s off

  • sockstat -46l | grep weed for listeners
  • weed shell → cluster.check, volume.list, fs.du /buckets/data
  • Sync connection loops → check pf/NAT with nc -vz
  • Deletions on B not reaching A → service seaweeddelprop status, run weed filer.meta.tail -filer=127.0.0.1:8888 -pathPrefix=/buckets/data by hand to see the delete events, check swb syslog and that A’s syncuser has Write:data
  • Signature errors → check chronyc tracking
  • Upgrades: same release on both, home box first