magpie/README.md

# Magpie

Magpie is a small object store for app-owned files. It stores objects on local disk and exposes a minimal S3-compatible path-style API.

It is intended to run behind your application or edge proxy, on localhost, a private Docker network, or a VPN-only address. Do not expose write-capable endpoints directly to the public internet.

## Status

Magpie supports common application object-storage operations:

- `PUT`, `GET`, `HEAD`, and `DELETE` objects
- path-style S3 routes: `/:bucket/:key`
- SigV4 header authentication for writes and private reads
- presigned `GET` and `HEAD` URLs
- prefix listing with continuation tokens
- batch delete
- copy object
- multipart upload
- bucket allowlist and opt-in public-read buckets
- local metadata index in SQLite
- async static-peer replication
- backup, restore, scrub, reindex, stats, and repair commands

It does not implement bucket creation, ACLs, bucket policies, object versioning, lifecycle rules, quorum writes, or consensus replication.

## Quick Start

```sh
export MAGPIE_ADDR='127.0.0.1:8090'
export MAGPIE_DATA_DIR='/var/lib/magpie'
export MAGPIE_S3_ACCESS_KEY_ID='magpie'
export MAGPIE_S3_SECRET_ACCESS_KEY='replace-with-a-long-random-secret'
export MAGPIE_ALLOWED_BUCKETS='app-uploads'

magpie serve
```

For local development from source:

```sh
go run . serve
```

## Configuration

Required authentication config:

```env
MAGPIE_S3_ACCESS_KEY_ID=magpie
MAGPIE_S3_SECRET_ACCESS_KEY=replace-with-a-long-random-secret
```

Or define multiple scoped keys:

```env
MAGPIE_S3_KEYS=app:secret1:read,write;auditor:secret2:read;admin:secret3:admin
```

Supported scopes are `read`, `write`, and `admin`. `admin` implies all scopes.

Common runtime config:

```env
MAGPIE_ADDR=127.0.0.1:8090
MAGPIE_DATA_DIR=/var/lib/magpie
MAGPIE_ALLOWED_BUCKETS=app-uploads
# MAGPIE_PUBLIC_PREFIXES=app-uploads/avatars
MAGPIE_RATE_LIMIT_PER_MINUTE=600
MAGPIE_TRUST_PROXY_HEADERS=false
MAGPIE_MAX_OBJECT_SIZE=1073741824
MAGPIE_MIN_FREE_BYTES=1073741824
MAGPIE_PRESIGNED_MAX_EXPIRY=24h
MAGPIE_MULTIPART_MAX_AGE=24h
MAGPIE_REINDEX_ON_START=false
MAGPIE_MAX_RESTORE_BYTES=10737418240
```

`MAGPIE_ALLOWED_BUCKETS` should be set in production. Leave it empty only if you intentionally want any bucket name accepted.

Buckets are private by default. Set `MAGPIE_PUBLIC_PREFIXES` to comma-separated `bucket/prefix` values, such as `app-uploads/avatars`, only when an app intentionally serves unsigned public media through an edge proxy. Set `MAGPIE_PUBLIC_PREFIXES=none` or leave it unset to require signed reads.

Public reads are hardened against stored-XSS from uploader-controlled content types: responses carry `Content-Security-Policy: default-src 'none'; sandbox`, and any object that is not inert inline media (images, audio, video — SVG excluded) is served with `Content-Disposition: attachment` so browsers download it instead of rendering it. For untrusted user uploads, still prefer serving public objects from an origin separate from your application.

## Application Integration

Use any S3-compatible client that supports path-style endpoints and custom hosts.

If Magpie runs on the same host as your application outside Docker:

```env
S3_ACCESS_KEY_ID=magpie
S3_SECRET_ACCESS_KEY=replace-with-a-long-random-secret
S3_ENDPOINT=127.0.0.1:8090
S3_BUCKET_NAME=app-uploads
S3_PUBLIC_URL=http://127.0.0.1:8090/app-uploads
S3_SCHEME=http://
S3_PORT=8090
```

If your app and Magpie are in the same Docker Compose network, use the Magpie service name:

```env
S3_ENDPOINT=magpie:8090
S3_BUCKET_NAME=app-uploads
S3_PUBLIC_URL=http://magpie:8090/app-uploads
S3_SCHEME=http://
S3_PORT=8090
```

Keep Magpie write access private. Your application should enforce user permissions before it writes to Magpie. Public buckets are intended for already-public objects such as avatars and timeline media.

## API Notes

S3 object routes use path-style URLs:

```text
PUT    /:bucket/:key
GET    /:bucket/:key
HEAD   /:bucket/:key
DELETE /:bucket/:key
POST   /:bucket?delete
```

Objects are stored under:

```text
MAGPIE_DATA_DIR/<bucket>/<key>
```

Internal names are reserved and cannot be used as object keys:

- `.multipart`
- `.magpie.db`
- `.magpie.db-*`
- `*.meta.json`

Presigned URLs are read-only. Magpie accepts presigned `GET` and `HEAD` requests, but rejects presigned writes and deletes.

Normal SigV4 write requests must include a real `X-Amz-Content-Sha256` payload hash. `UNSIGNED-PAYLOAD` is reserved for authenticated replication traffic.

## Health And Metrics

Health and metrics require authentication.

```sh
curl -H "Authorization: Bearer $MAGPIE_AUTH_TOKEN" \
  http://127.0.0.1:8090/health
```

```sh
curl -H "Authorization: Bearer $MAGPIE_AUTH_TOKEN" \
  http://127.0.0.1:8090/metrics
```

The health endpoint returns `503` when free disk space is below `MAGPIE_MIN_FREE_BYTES`. Writes are rejected in that state.

If you only use S3 credentials and do not set `MAGPIE_AUTH_TOKEN`, sign health and metrics requests with an admin S3 key instead.

## CLI

```sh
magpie serve
magpie stats
magpie scrub
magpie reindex
magpie repair
magpie backup /backup/magpie.tar.gz
magpie restore /backup/magpie.tar.gz
magpie smoke
magpie version
```

Command summary:

- `stats`: object count, logical bytes, disk status, replication queue status
- `scrub`: verifies metadata against files and reports missing, corrupt, and orphaned objects
- `reindex`: rebuilds SQLite metadata from files on disk
- `repair`: queues known objects for replication to configured peers
- `backup`: creates a tar/gzip archive of the data directory
- `restore`: restores a tar/gzip archive into the data directory
- `smoke`: runs a local write/read/list/scrub/delete check

Restore rejects unsafe archive paths, symlinks, unsupported archive entries, and archives whose expanded file content exceeds `MAGPIE_MAX_RESTORE_BYTES`. Backup archives are created with owner-only permissions. For consistent backups, stop Magpie or use a filesystem snapshot before running `magpie backup`.

## Replication

Replication is asynchronous and best-effort. Local writes complete before peers acknowledge them.

Configure every node with matching S3 credentials and a shared replication secret:

```env
MAGPIE_REPLICATION_PEERS=https://magpie-b.internal,https://magpie-c.internal
MAGPIE_REPLICATION_SECRET=replace-with-a-long-random-peer-secret
MAGPIE_REPLICATION_MAX_JOBS=10000
MAGPIE_REPLICATION_MAX_ATTEMPTS=100
```

`MAGPIE_REPLICATION_SECRET` is required when peers are configured. Replication-marked requests without the shared secret are rejected.

Replication peer URLs must use `https://` by default because Magpie sends SigV4 authorization and the replication secret to peers. Set `MAGPIE_ALLOW_INSECURE_REPLICATION=true` only for trusted local/private networks where plaintext HTTP is acceptable.

Use `magpie repair` after outages to queue a full object repair pass to all configured peers.

## Deployment

Docker Compose example:

```sh
cp .env.production.example .env.production
```

```yaml
services:
  magpie:
    image: magpie:latest
    container_name: magpie
    build: .
    restart: unless-stopped
    env_file:
      - .env.production
    ports:
      - "127.0.0.1:8090:8090"
    volumes:
      - /var/lib/magpie:/var/lib/magpie
```

The default Compose file is standalone and binds Magpie to localhost only.

If another Compose project needs to reach Magpie by service name, use the optional network override. It attaches Magpie to an external shared network with the `magpie` network alias:

```sh
docker network create elektrine-magpie-shared
docker compose -f docker-compose.yml -f docker-compose.network.yml up -d --build
```

Then attach your application container to the same external `elektrine-magpie-shared` network and use:

```env
S3_ENDPOINT=magpie:8090
S3_PUBLIC_URL=http://magpie:8090/app-uploads
S3_SCHEME=http://
S3_PORT=8090
```

You can rename the shared network without editing the override file:

```sh
MAGPIE_DOCKER_NETWORK=my-existing-shared-network \
  docker compose -f docker-compose.yml -f docker-compose.network.yml up -d --build
```

Bare-metal installs can use `deploy/systemd/magpie.service`. The systemd unit is optional; use it only if you want Magpie managed directly by systemd.

Back up `/var/lib/magpie` or whatever directory you set as `MAGPIE_DATA_DIR`.

## Build And Test

```sh
go test ./...
go build ./...
```

Release artifacts:

```sh
make test
make build
make release
```

Artifacts are written to `dist/`, with checksums in `dist/SHA256SUMS`.