Pipely™ - single-purpose, single-tenant CDN

Based on Varnish Cache. This started as the simplest CDN running on fly.io for changelog.com

You are welcome to fork this and build your own - OSS FTW 💚

How it started

🧢 Jerod Santo - March 29, 2024 - Changelog & Friends #38

The Roadmap to v2.0

• Tag & ship v1.1
  • ✅ Log & forward original fly-request-id header - PR #42
  • ✅ Support websocket connections - PR #43
  • ✅ Store MP3s in file cache + HOT & COLD instances - PR #44
  • ✅ Update deps to latest stable (hold Varnish at v7.7.3) - PR #45
  • Run periodic MP3 & feed checks against all regions
  • Add nightly.changelog.com backend
• Add logging acceptance tests
• Keep Dagger version in .github/workflows/_namespace.yaml in sync with just/dagger.just

What went into v1.0

• ✅ Static backend, 1 day stale, stale on error, x-headers - Initial commit
• ✅ Dynamic backend, cache-status header - PR #1
• ✅ Add tests - PR #3
• ✅ Make it easy to develop locally - PR #7
• ✅ Add support for TLS backends, publish & deploy to production - PR #8
• ✅ Add Feeds backend - PR #10
• ✅ Add Assets backend - PR #11
• ✅ Send Varnish logs to Honeycomb.io - PR #12
• ✅ Enrich Varnish logs with GeoIP data - PR #13
• ✅ Supervisor restarts crashed processes - PR #14
• ✅ Auth PURGE requests - PR #16
• ✅ Add redirects from Fastly VCL - PR #19
• ✅ Send Varnish logs to S3 - PR #27
• ✅ All contributors review & clean-up
  • Is the VCL as clean & efficient as it could be?
  • Does everything work as expected?
  • Anything that can be removed?
  • How do we make this friendlier to new users?
  • What would make this more contribution-friendly?
  • How easy is this to use as your own deployment?
• ✅ Tag & ship v1.0-rc.1
  • ✅ Update documentation and do some local dev tests - PR #22
  • ✅ Add debug welcome message and prompt - PR #25
  • ✅ Avoid using home_dir() due to Windows issues - PR #26
  • ✅ Add troubleshooting and misc to local dev docs - PR #29
• ✅ Tag & ship v1.0-rc.2
  • ✅ Prepare for 20% of the production traffic - PR #30
  • Route 20% of the production traffic through
  • Observe cold cache behaviour
• ✅ Tag & ship v1.0-rc.3
  • ✅ Fix feeds URL rewrite - PR #31
  • ✅ Increase instance size - PR #32
• ✅ Tag & ship v1.0-rc.4
  • ✅ Limit Varnish memory to 66% (out of 3200M out of 4000M) - 3553723
• ✅ Tag & ship v1.0-rc.5
  • ✅ Handle varnish-json-response failing on startup - PR #33
  • ✅ Bump the instance size to performance-1x with 8GB of RAM - PR #34
  • Route 50% of the production traffic through
• ✅ Tag & ship v1.0-rc.6
  • ✅ Add more locations - PR #35
  • ✅ Increase backend timeout - PR #36
• ✅ Tag & ship v1.0-rc.7
  • ✅ Update to Varnish v7.7.3 & Vector v0.49.0 - PR #38
  • ✅ Support MP3 uploads - PR #39
• ✅ Tag & ship v1.0
  • ✅ Update all dependencies to latest (hold Varnish at v7.7.3) - PR #40
  • ✅ Route 100% of the production traffic through v1.0

Local development and testing

While it's fun watching other people experiment with digital resin (varnish 😂), it's a whole lot more fun when you can repeat those experiments yourself, understand more how it works, and make your own modifications.

Prerequisites

• 🐳 Docker
• 🤖 Just version 1.35.0 or higher

And that's about it. Everything else is containerized with Dagger.

Note

For Windows Developers: The project's toolchain is made for Linux-like systems. On a Windows machine you will need to have the Windows Subsystem for Linux (WSL) installed in addition to Docker. just should be installed inside your WSL Linux operating system. You might be able to run Just natively from Windows, but there are some known bugs related to home directory filenames, so better to avoid that altogether and work directly in WSL.

just
Available recipes:
    docker-run *ARGS                          # Run container in Docker (works on remote servers too): http://<DOCKER_HOST>:9000
    how-many-lines                            # How many lines of Varnish config?
    how-many-lines-raw                        # How many lines of Varnish config?
    http-profile url="https://changelog.com/" # Observe all HTTP timings - https://blog.cloudflare.com/a-question-of-timing
    local-debug                               # Debug container locally
    local-run                                 # Run container locally: available on http://localhost:9000
    test                                      # Test VTC + acceptance locally
    test-acceptance-local                     # Test local setup
    test-reports                              # Open test reports
    test-reports-rm                           # Clear test reports
    test-vtc                                  # Test VCL config

    [team]
    envrc-secrets                             # Create .envrc.secrets with credentials from 1Password
    local-debug-production                    # Debug production container locally - assumes envrc-secrets has already run
    local-run-production                      # Run production container locally - assumes envrc-secrets has already run - available on http://localhost:9000
    publish tag=_DEFAULT_TAG                  # Publish container image - assumes envrc-secrets was already run
    tag tag sha discussion                    # Tag a new release
    test-acceptance-production *ARGS          # Test production acceptance

# Run the tests
just test

How can you help

If you have any ideas on how to improve this, please open an issue or go straight for a pull request. We make this as easy as possible:

• All commits emphasize good commit messages (more text for humans)
• This repository is kept small & simple (single purpose: build the simplest CDN on Fly.io)
• Slow & thoughtful approach - join our journey via audio with transcripts or written

See you in our Zulip Chat 👋

Note

Join from https://changelog.com/~ . It requires signing up and requesting an invite before you can Log in

Contributors

• Nabeel Sulieman
• Matt Johnson
• James A Rosen
• Gerhard Lazu