Magpie

Magpie is a small object store for app-owned files. It stores objects on local disk and exposes a minimal S3-compatible path-style API.

It is intended to run behind your application or edge proxy, on localhost, a private Docker network, or a VPN-only address. Do not expose write-capable endpoints directly to the public internet.

Status

Magpie supports common application object-storage operations:

• PUT, GET, HEAD, and DELETE objects
• path-style S3 routes: /:bucket/:key
• SigV4 header authentication for writes and private reads
• presigned GET and HEAD URLs
• prefix listing with continuation tokens
• batch delete
• copy object
• multipart upload
• bucket allowlist and opt-in public-read buckets
• local metadata index in SQLite
• async static-peer replication
• backup, restore, scrub, reindex, stats, and repair commands

It does not implement bucket creation, ACLs, bucket policies, object versioning, lifecycle rules, quorum writes, or consensus replication.

Quick Start

export MAGPIE_ADDR='127.0.0.1:8090'
export MAGPIE_DATA_DIR='/var/lib/magpie'
export MAGPIE_S3_ACCESS_KEY_ID='magpie'
export MAGPIE_S3_SECRET_ACCESS_KEY='replace-with-a-long-random-secret'
export MAGPIE_ALLOWED_BUCKETS='app-uploads'

magpie serve

For local development from source:

go run . serve

Configuration

Required authentication config:

MAGPIE_S3_ACCESS_KEY_ID=magpie
MAGPIE_S3_SECRET_ACCESS_KEY=replace-with-a-long-random-secret

Or define multiple scoped keys:

MAGPIE_S3_KEYS=app:secret1:read,write;auditor:secret2:read;admin:secret3:admin

Supported scopes are read, write, and admin. admin implies all scopes.

Common runtime config:

MAGPIE_ADDR=127.0.0.1:8090
MAGPIE_DATA_DIR=/var/lib/magpie
MAGPIE_ALLOWED_BUCKETS=app-uploads
# MAGPIE_PUBLIC_PREFIXES=app-uploads/avatars
MAGPIE_RATE_LIMIT_PER_MINUTE=600
MAGPIE_TRUST_PROXY_HEADERS=false
MAGPIE_MAX_OBJECT_SIZE=1073741824
MAGPIE_MIN_FREE_BYTES=1073741824
MAGPIE_PRESIGNED_MAX_EXPIRY=24h
MAGPIE_MULTIPART_MAX_AGE=24h
MAGPIE_REINDEX_ON_START=false
MAGPIE_MAX_RESTORE_BYTES=10737418240

MAGPIE_ALLOWED_BUCKETS should be set in production. Leave it empty only if you intentionally want any bucket name accepted.

Buckets are private by default. Set MAGPIE_PUBLIC_PREFIXES to comma-separated bucket/prefix values, such as app-uploads/avatars, only when an app intentionally serves unsigned public media through an edge proxy. Set MAGPIE_PUBLIC_PREFIXES=none or leave it unset to require signed reads.

Public reads are hardened against stored-XSS from uploader-controlled content types: responses carry Content-Security-Policy: default-src 'none'; sandbox, and any object that is not inert inline media (images, audio, video — SVG excluded) is served with Content-Disposition: attachment so browsers download it instead of rendering it. For untrusted user uploads, still prefer serving public objects from an origin separate from your application.

Application Integration

Use any S3-compatible client that supports path-style endpoints and custom hosts.

If Magpie runs on the same host as your application outside Docker:

S3_ACCESS_KEY_ID=magpie
S3_SECRET_ACCESS_KEY=replace-with-a-long-random-secret
S3_ENDPOINT=127.0.0.1:8090
S3_BUCKET_NAME=app-uploads
S3_PUBLIC_URL=http://127.0.0.1:8090/app-uploads
S3_SCHEME=http://
S3_PORT=8090

If your app and Magpie are in the same Docker Compose network, use the Magpie service name:

S3_ENDPOINT=magpie:8090
S3_BUCKET_NAME=app-uploads
S3_PUBLIC_URL=http://magpie:8090/app-uploads
S3_SCHEME=http://
S3_PORT=8090

Keep Magpie write access private. Your application should enforce user permissions before it writes to Magpie. Public buckets are intended for already-public objects such as avatars and timeline media.

API Notes

S3 object routes use path-style URLs:

PUT    /:bucket/:key
GET    /:bucket/:key
HEAD   /:bucket/:key
DELETE /:bucket/:key
POST   /:bucket?delete

Objects are stored under:

MAGPIE_DATA_DIR/<bucket>/<key>

Internal names are reserved and cannot be used as object keys:

• .multipart
• .magpie.db
• .magpie.db-*
• *.meta.json

Presigned URLs are read-only. Magpie accepts presigned GET and HEAD requests, but rejects presigned writes and deletes.

Normal SigV4 write requests must include a real X-Amz-Content-Sha256 payload hash. UNSIGNED-PAYLOAD is reserved for authenticated replication traffic.

Health And Metrics

Health and metrics require authentication.

curl -H "Authorization: Bearer $MAGPIE_AUTH_TOKEN" \
  http://127.0.0.1:8090/health

curl -H "Authorization: Bearer $MAGPIE_AUTH_TOKEN" \
  http://127.0.0.1:8090/metrics

The health endpoint returns 503 when free disk space is below MAGPIE_MIN_FREE_BYTES. Writes are rejected in that state.

If you only use S3 credentials and do not set MAGPIE_AUTH_TOKEN, sign health and metrics requests with an admin S3 key instead.

CLI

magpie serve
magpie stats
magpie scrub
magpie reindex
magpie repair
magpie backup /backup/magpie.tar.gz
magpie restore /backup/magpie.tar.gz
magpie smoke
magpie version

Command summary:

• stats: object count, logical bytes, disk status, replication queue status
• scrub: verifies metadata against files and reports missing, corrupt, and orphaned objects
• reindex: rebuilds SQLite metadata from files on disk
• repair: queues known objects for replication to configured peers
• backup: creates a tar/gzip archive of the data directory
• restore: restores a tar/gzip archive into the data directory
• smoke: runs a local write/read/list/scrub/delete check

Restore rejects unsafe archive paths, symlinks, unsupported archive entries, and archives whose expanded file content exceeds MAGPIE_MAX_RESTORE_BYTES. Backup archives are created with owner-only permissions. For consistent backups, stop Magpie or use a filesystem snapshot before running magpie backup.

Replication

Replication is asynchronous and best-effort. Local writes complete before peers acknowledge them.

Configure every node with matching S3 credentials and a shared replication secret:

MAGPIE_REPLICATION_PEERS=https://magpie-b.internal,https://magpie-c.internal
MAGPIE_REPLICATION_SECRET=replace-with-a-long-random-peer-secret
MAGPIE_REPLICATION_MAX_JOBS=10000
MAGPIE_REPLICATION_MAX_ATTEMPTS=100

MAGPIE_REPLICATION_SECRET is required when peers are configured. Replication-marked requests without the shared secret are rejected.

Replication peer URLs must use https:// by default because Magpie sends SigV4 authorization and the replication secret to peers. Set MAGPIE_ALLOW_INSECURE_REPLICATION=true only for trusted local/private networks where plaintext HTTP is acceptable.

Use magpie repair after outages to queue a full object repair pass to all configured peers.

Deployment

Docker Compose example:

cp .env.production.example .env.production

services:
  magpie:
    image: magpie:latest
    container_name: magpie
    build: .
    restart: unless-stopped
    env_file:
      - .env.production
    ports:
      - "127.0.0.1:8090:8090"
    volumes:
      - /var/lib/magpie:/var/lib/magpie

The default Compose file is standalone and binds Magpie to localhost only.

If another Compose project needs to reach Magpie by service name, use the optional network override. It attaches Magpie to an external shared network with the magpie network alias:

docker network create elektrine-magpie-shared
docker compose -f docker-compose.yml -f docker-compose.network.yml up -d --build

Then attach your application container to the same external elektrine-magpie-shared network and use:

S3_ENDPOINT=magpie:8090
S3_PUBLIC_URL=http://magpie:8090/app-uploads
S3_SCHEME=http://
S3_PORT=8090

You can rename the shared network without editing the override file:

MAGPIE_DOCKER_NETWORK=my-existing-shared-network \
  docker compose -f docker-compose.yml -f docker-compose.network.yml up -d --build

Bare-metal installs can use deploy/systemd/magpie.service. The systemd unit is optional; use it only if you want Magpie managed directly by systemd.

Back up /var/lib/magpie or whatever directory you set as MAGPIE_DATA_DIR.

Build And Test

go test ./...
go build ./...

Release artifacts:

make test
make build
make release

Artifacts are written to dist/, with checksums in dist/SHA256SUMS.