Maxfield Luke
aa43ce2562
Add scripts/deploy.sh to build/start Magpie and ensure it joins elektrine-magpie-shared (alias magpie) so Elektrine can reach magpie:8090. Forgejo Actions rsyncs and runs the script on every main push.
293 lines
9 KiB
Markdown
293 lines
9 KiB
Markdown
# Magpie
|
|
|
|
Magpie is a small object store for app-owned files. It stores objects on local disk and exposes a minimal S3-compatible path-style API.
|
|
|
|
It is intended to run behind your application or edge proxy, on localhost, a private Docker network, or a VPN-only address. Do not expose write-capable endpoints directly to the public internet.
|
|
|
|
## Status
|
|
|
|
Magpie supports common application object-storage operations:
|
|
|
|
- `PUT`, `GET`, `HEAD`, and `DELETE` objects
|
|
- path-style S3 routes: `/:bucket/:key`
|
|
- SigV4 header authentication for writes and private reads
|
|
- presigned `GET` and `HEAD` URLs
|
|
- prefix listing with continuation tokens
|
|
- batch delete
|
|
- copy object
|
|
- multipart upload
|
|
- bucket allowlist and opt-in public-read buckets
|
|
- local metadata index in SQLite
|
|
- async static-peer replication
|
|
- backup, restore, scrub, reindex, stats, and repair commands
|
|
|
|
It does not implement bucket creation, ACLs, bucket policies, object versioning, lifecycle rules, quorum writes, or consensus replication.
|
|
|
|
## Quick Start
|
|
|
|
```sh
|
|
export MAGPIE_ADDR='127.0.0.1:8090'
|
|
export MAGPIE_DATA_DIR='/var/lib/magpie'
|
|
export MAGPIE_S3_ACCESS_KEY_ID='magpie'
|
|
export MAGPIE_S3_SECRET_ACCESS_KEY='replace-with-a-long-random-secret'
|
|
export MAGPIE_ALLOWED_BUCKETS='app-uploads'
|
|
|
|
magpie serve
|
|
```
|
|
|
|
For local development from source:
|
|
|
|
```sh
|
|
go run . serve
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Required authentication config:
|
|
|
|
```env
|
|
MAGPIE_S3_ACCESS_KEY_ID=magpie
|
|
MAGPIE_S3_SECRET_ACCESS_KEY=replace-with-a-long-random-secret
|
|
```
|
|
|
|
Or define multiple scoped keys:
|
|
|
|
```env
|
|
MAGPIE_S3_KEYS=app:secret1:read,write;auditor:secret2:read;admin:secret3:admin
|
|
```
|
|
|
|
Supported scopes are `read`, `write`, and `admin`. `admin` implies all scopes.
|
|
|
|
Common runtime config:
|
|
|
|
```env
|
|
MAGPIE_ADDR=127.0.0.1:8090
|
|
MAGPIE_DATA_DIR=/var/lib/magpie
|
|
MAGPIE_ALLOWED_BUCKETS=app-uploads
|
|
# MAGPIE_PUBLIC_PREFIXES=app-uploads/avatars
|
|
MAGPIE_RATE_LIMIT_PER_MINUTE=600
|
|
MAGPIE_TRUST_PROXY_HEADERS=false
|
|
MAGPIE_MAX_OBJECT_SIZE=1073741824
|
|
MAGPIE_MIN_FREE_BYTES=1073741824
|
|
MAGPIE_PRESIGNED_MAX_EXPIRY=24h
|
|
MAGPIE_MULTIPART_MAX_AGE=24h
|
|
MAGPIE_REINDEX_ON_START=false
|
|
MAGPIE_MAX_RESTORE_BYTES=10737418240
|
|
```
|
|
|
|
`MAGPIE_ALLOWED_BUCKETS` should be set in production. Leave it empty only if you intentionally want any bucket name accepted.
|
|
|
|
Buckets are private by default. Set `MAGPIE_PUBLIC_PREFIXES` to comma-separated `bucket/prefix` values, such as `app-uploads/avatars`, only when an app intentionally serves unsigned public media through an edge proxy. Set `MAGPIE_PUBLIC_PREFIXES=none` or leave it unset to require signed reads.
|
|
|
|
Public reads are hardened against stored-XSS from uploader-controlled content types: responses carry `Content-Security-Policy: default-src 'none'; sandbox`, and any object that is not inert inline media (images, audio, video — SVG excluded) is served with `Content-Disposition: attachment` so browsers download it instead of rendering it. For untrusted user uploads, still prefer serving public objects from an origin separate from your application.
|
|
|
|
## Application Integration
|
|
|
|
Use any S3-compatible client that supports path-style endpoints and custom hosts.
|
|
|
|
If Magpie runs on the same host as your application outside Docker:
|
|
|
|
```env
|
|
S3_ACCESS_KEY_ID=magpie
|
|
S3_SECRET_ACCESS_KEY=replace-with-a-long-random-secret
|
|
S3_ENDPOINT=127.0.0.1:8090
|
|
S3_BUCKET_NAME=app-uploads
|
|
S3_PUBLIC_URL=http://127.0.0.1:8090/app-uploads
|
|
S3_SCHEME=http://
|
|
S3_PORT=8090
|
|
```
|
|
|
|
If your app and Magpie are in the same Docker Compose network, use the Magpie service name:
|
|
|
|
```env
|
|
S3_ENDPOINT=magpie:8090
|
|
S3_BUCKET_NAME=app-uploads
|
|
S3_PUBLIC_URL=http://magpie:8090/app-uploads
|
|
S3_SCHEME=http://
|
|
S3_PORT=8090
|
|
```
|
|
|
|
Keep Magpie write access private. Your application should enforce user permissions before it writes to Magpie. Public buckets are intended for already-public objects such as avatars and timeline media.
|
|
|
|
## API Notes
|
|
|
|
S3 object routes use path-style URLs:
|
|
|
|
```text
|
|
PUT /:bucket/:key
|
|
GET /:bucket/:key
|
|
HEAD /:bucket/:key
|
|
DELETE /:bucket/:key
|
|
POST /:bucket?delete
|
|
```
|
|
|
|
Objects are stored under:
|
|
|
|
```text
|
|
MAGPIE_DATA_DIR/<bucket>/<key>
|
|
```
|
|
|
|
Internal names are reserved and cannot be used as object keys:
|
|
|
|
- `.multipart`
|
|
- `.magpie.db`
|
|
- `.magpie.db-*`
|
|
- `*.meta.json`
|
|
|
|
Presigned URLs are read-only. Magpie accepts presigned `GET` and `HEAD` requests, but rejects presigned writes and deletes.
|
|
|
|
Normal SigV4 write requests must include a real `X-Amz-Content-Sha256` payload hash. `UNSIGNED-PAYLOAD` is reserved for authenticated replication traffic.
|
|
|
|
## Health And Metrics
|
|
|
|
Health and metrics require authentication.
|
|
|
|
```sh
|
|
curl -H "Authorization: Bearer $MAGPIE_AUTH_TOKEN" \
|
|
http://127.0.0.1:8090/health
|
|
```
|
|
|
|
```sh
|
|
curl -H "Authorization: Bearer $MAGPIE_AUTH_TOKEN" \
|
|
http://127.0.0.1:8090/metrics
|
|
```
|
|
|
|
The health endpoint returns `503` when free disk space is below `MAGPIE_MIN_FREE_BYTES`. Writes are rejected in that state.
|
|
|
|
If you only use S3 credentials and do not set `MAGPIE_AUTH_TOKEN`, sign health and metrics requests with an admin S3 key instead.
|
|
|
|
## CLI
|
|
|
|
```sh
|
|
magpie serve
|
|
magpie stats
|
|
magpie scrub
|
|
magpie reindex
|
|
magpie repair
|
|
magpie backup /backup/magpie.tar.gz
|
|
magpie restore /backup/magpie.tar.gz
|
|
magpie smoke
|
|
magpie version
|
|
```
|
|
|
|
Command summary:
|
|
|
|
- `stats`: object count, logical bytes, disk status, replication queue status
|
|
- `scrub`: verifies metadata against files and reports missing, corrupt, and orphaned objects
|
|
- `reindex`: rebuilds SQLite metadata from files on disk
|
|
- `repair`: queues known objects for replication to configured peers
|
|
- `backup`: creates a tar/gzip archive of the data directory
|
|
- `restore`: restores a tar/gzip archive into the data directory
|
|
- `smoke`: runs a local write/read/list/scrub/delete check
|
|
|
|
Restore rejects unsafe archive paths, symlinks, unsupported archive entries, and archives whose expanded file content exceeds `MAGPIE_MAX_RESTORE_BYTES`. Backup archives are created with owner-only permissions. For consistent backups, stop Magpie or use a filesystem snapshot before running `magpie backup`.
|
|
|
|
## Replication
|
|
|
|
Replication is asynchronous and best-effort. Local writes complete before peers acknowledge them.
|
|
|
|
Configure every node with matching S3 credentials and a shared replication secret:
|
|
|
|
```env
|
|
MAGPIE_REPLICATION_PEERS=https://magpie-b.internal,https://magpie-c.internal
|
|
MAGPIE_REPLICATION_SECRET=replace-with-a-long-random-peer-secret
|
|
MAGPIE_REPLICATION_MAX_JOBS=10000
|
|
MAGPIE_REPLICATION_MAX_ATTEMPTS=100
|
|
```
|
|
|
|
`MAGPIE_REPLICATION_SECRET` is required when peers are configured. Replication-marked requests without the shared secret are rejected.
|
|
|
|
Replication peer URLs must use `https://` by default because Magpie sends SigV4 authorization and the replication secret to peers. Set `MAGPIE_ALLOW_INSECURE_REPLICATION=true` only for trusted local/private networks where plaintext HTTP is acceptable.
|
|
|
|
Use `magpie repair` after outages to queue a full object repair pass to all configured peers.
|
|
|
|
## Deployment
|
|
|
|
Docker Compose example:
|
|
|
|
```sh
|
|
cp .env.production.example .env.production
|
|
```
|
|
|
|
```yaml
|
|
services:
|
|
magpie:
|
|
image: magpie:latest
|
|
container_name: magpie
|
|
build: .
|
|
restart: unless-stopped
|
|
env_file:
|
|
- .env.production
|
|
ports:
|
|
- "127.0.0.1:8090:8090"
|
|
volumes:
|
|
- /var/lib/magpie:/var/lib/magpie
|
|
```
|
|
|
|
The default Compose file is standalone and binds Magpie to localhost only.
|
|
|
|
If another Compose project needs to reach Magpie by service name, use the optional network override. It attaches Magpie to an external shared network with the `magpie` network alias:
|
|
|
|
```sh
|
|
docker network create elektrine-magpie-shared
|
|
docker compose -f docker-compose.yml -f docker-compose.network.yml up -d --build
|
|
```
|
|
|
|
Then attach your application container to the same external `elektrine-magpie-shared` network and use:
|
|
|
|
```env
|
|
S3_ENDPOINT=magpie:8090
|
|
S3_PUBLIC_URL=http://magpie:8090/app-uploads
|
|
S3_SCHEME=http://
|
|
S3_PORT=8090
|
|
```
|
|
|
|
You can rename the shared network without editing the override file:
|
|
|
|
```sh
|
|
MAGPIE_DOCKER_NETWORK=my-existing-shared-network \
|
|
docker compose -f docker-compose.yml -f docker-compose.network.yml up -d --build
|
|
```
|
|
|
|
Bare-metal installs can use `deploy/systemd/magpie.service`. The systemd unit is optional; use it only if you want Magpie managed directly by systemd.
|
|
|
|
Back up `/var/lib/magpie` or whatever directory you set as `MAGPIE_DATA_DIR`.
|
|
|
|
## Build And Test
|
|
|
|
```sh
|
|
go test ./...
|
|
go build ./...
|
|
```
|
|
|
|
Release artifacts:
|
|
|
|
```sh
|
|
make test
|
|
make build
|
|
make release
|
|
```
|
|
|
|
Artifacts are written to `dist/`, with checksums in `dist/SHA256SUMS`.
|
|
|
|
## Production deploy (same host as Elektrine)
|
|
|
|
Magpie runs as a separate Compose project on the Elektrine host and joins the
|
|
shared Docker network `elektrine-magpie-shared` so Elektrine services can reach
|
|
`http://magpie:8090`.
|
|
|
|
```sh
|
|
# on the server (linuxuser)
|
|
cd /opt/magpie
|
|
./scripts/deploy.sh
|
|
```
|
|
|
|
Forgejo Actions (push to `main`) rsyncs to `linuxuser@elektrine.com:/opt/magpie`
|
|
and runs that script. Elektrine deploy re-attaches Magpie to the shared network
|
|
every time via `ensure_magpie_shared_network`.
|
|
|
|
Required on the host:
|
|
|
|
- `/opt/magpie/.env.production` (not in git)
|
|
- Docker (or passwordless `sudo docker`) for `linuxuser`
|
|
- Shared network name: `MAGPIE_DOCKER_NETWORK` (default `elektrine-magpie-shared`)
|