Logbook - 2022-05-05

Features and fixes are flying like dodgeballs in a school gym, and the Fly.io Changelog Enforcer could probably have done a better job patrolling—but let’s have a look at our haul of updates since our first Logbook post.

There’s a fair amount of protein in this week’s mix. Let’s kick it off with improved remote builders you can activate for your organization!

• [Feature] We updated remote builders for faster first deployments and fewer full disks. Builders will now:
  1. Use regional Docker Hub mirrors, which especially improved initial Buildpack deployments
  2. Clean up unused build caches created by RUN --mount=type=cache along with standard image cleanup
  3. Run cleanup tasks on startup and shutdown.
  This behavior is the default for new builders. Existing builders must be destroyed with fly apps destroy builder-app-name to get the new behavior.
• [Feature] Deployed some new security logging for our API that should both help us check a box for SOC2, and also start to give us a way to track down key security-related events for customers. More to come there.
• [Feature] Shipped a self-service payment retry feature on our dashboard. Official announcement
• [Feature] Shipped our new /dashboard page. Here’s the official announcement.
• [Feature] Thomas wrote about 10,000 words of security policy for SOC2 and policy will make us all safer, shut up, it will.
• [Feature] Improved performance of our existing WireGuard gateways (by fixing load issues) and added more capacity in each region. This does two things: 1) fly deploy and ssh commands will work more reliably with less timeouts and 2) it lays the groundwork for gateway redundancy. gateway redundancy is important for people who want to peer with other private networks for server communications.
• [Feature] Spun up some new gateways so that customers have a better experience creating new peers.
• [Feature] Merged some internal dev-setup Quality of Life things, to make it easier for new and old team members to jump into our API and be productive quickly.
• [Feature] Changed the default size for new persistent volumes from 10GB to 3GB. This makes it less likely that new users will accidentally provision (and be charged for) more storage than they need for small projects.
• [Feature] Re-enabled OpenTelemetry, which should give us deeper visibility into how our API is running, allow us to better target problem areas, and catch issues earlier, with higher confidence.
• [Feature] Major speedup to our proxy build pipeline, enabling faster features and fixes deployment.
• [Feature] Stood up new servers to increase capacity in Hong Kong.
• [Feature] Updated the reset password page styling to make it easier to use.
• [Feature] Added fly.toml validation and inline documentation to the VSCode Even Better TOML extension, to help users read and edit their fly.toml more easily.
• [Feature] Added another wireguard gateway in IAD. Customers connect through gateways to their 6pn networks, IAD is our busiest region for customer network peers because most CI tools run there. This should make CI (github actions especially) more reliable.
• [Feature] Deployed some initial changes to how we collect our database models for our GraphQL APIs that should help us manage API performance as GraphQL queries increase in complexity and depth. Also removed some unnecessary metric collection that should reduce our baseline database load.
• [Fix] Deployed some fixes to the GraphQL API for edge cases folks were hitting, mostly with some of our older nodeproxy apps.
• [Fix] Fixed organization dashboard not showing the latest invoice total. It only showed “-”.
• [Fix] Fixed fly monitor: Ctrl-C sets off an infinite loop of error messages in fly monitor by porting it to the new CLI format and reusing the new deployment monitoring code.
• [Fix] Personal Organizations are a subclass of Organization. Both of these models automatically encrypt/decrypt SSH key information. Last week we noticed that something that wasn’t accessing those encrypted attributes was still throwing errors due to Vault connection issues. This is due to a mix of a discrepancy with how Ruby does inheritance/class attributes and an issue in our use of the library that does encryption/decryption for us. This should prevent us from loading those attributes unless they’re actually used going forward.
• [Fix] Updated an expired token that would have broken automatic bumps of flyctl in Homebrew. We didn’t have any releases to bump while this was broken, so this should be an invisible fix. PR
• [Docs] Updated our list of built-in metrics for completeness and included details about the more complex ones. https://fly.io/docs/reference/metrics/#built-in-metrics
• [Docs] Updated regions doc, linking to other related doc pages. It’s now easier to find the commands for working with regions. Added missing Madrid region. https://fly.io/docs/reference/regions/