This is a post about the most dangerous vulnerability most web applications face, one step that we took at Fly to mitigate it, and how you can do the same.
Server-side request forgery (SSRF) is application security jargon for “attackers can get your app server to make HTTP requests on their behalf”. Compared to other high severity vulnerabilities like SQL injection, which allows attackers to take over your database, or filesystem access or remote code injection, SSRF doesn’t sound that scary. But it is, and you should be nervous about it.
The deceptive severity of SSRF is one of two factors that makes SSRF so insidious. The reason is simple: your app server is behind a security perimeter, and can usually reach things an ordinary Internet user can’t. Because HTTP is a relatively flexible protocol, and URLs are so expressive, attackers can often use SSRF to reach surprising places; in fact, leveraging HTTP SSRF to reach non-SSRF protocols has become a sport among security researchers. A meaty, complicated example of this is Joshua Maddux’s TLS SSRF trick from last August. Long story short: in serious applications, SSRF is usually a game-over vulnerability, meaning attackers can use it to gain full control over an application’s hosting environment.
The other factor that makes SSRF nerve-wracking is its prevalence. As an industry, we’ve managed to drastically reduce instances of vulnerabilities like SQL injection by updating our libraries and changing best practices; for instance, it would be weird to see a mainstream SQL library that didn’t use parameterized queries to keep attacker meta-characters out of query parsing. But applications of all shapes and sizes make server-side HTTP queries; in fact, if anything, that’s becoming more common as we adopt more and more web APIs.
There are two common patterns of SSRF vulnerabilities. The first, simplest, and most dangerous comprises features that allow users to provide URLs for the web server to call directly; for instance, your app might offer “web hooks” to call back to customer web servers. The second pattern involves features that incorporate user data into URLs. In both cases, an attacker will try to leverage whatever control you offer over URLs to trick your server into hitting unexpected URLs.
Fortunately, there’s a mitigation that frustrates attackers trying to exploit either pattern of SSRF vulnerabilities: SSRF proxies.
You should know about Smokescreen
Imagine if your application code didn’t have to be relentlessly vigilant about every URL it reached out to, and could instead assume that a basic security control existed to make sure that no server-side HTTP query would be able to touch internal resources? It’s easy if you try! What you want is to run your server-side HTTP through a proxy.
We’ve been putting Smokescreen to work at Fly, and it’s so useful, we thought we should share. Smokescreen is an egress proxy that was built at Stripe to help manage outgoing connections in a sensible and safe way.
Smokescreen’s job is to make sure your outgoing requests are sane, sensible and safe. SmokeScreen was created by Stripe to ensure that they knew where all their outgoing requests were going. Specifically, it makes sure that the IP address requested is a publicly routed IP and that means checking any request isn’t destined for 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, or fc00::/7 (the IPv6 “ULA” space, which includes Fly’s 6PN private addresses.
Out of the box
There’s more SmokeScreen can do, but before we get to that, let’s talk about how Smokescreen determines who you are. By default, Smokescreen uses the client certificate from a TLS connection, extracts the common name of the certificate and uses that as the role. There is another mechanism documented for non-TLS connections using a header but doesn’t seem to be actually wired up Smokescreen (probably because it’s way too simple to present to be another system). So you’ll have to use TLS CA certs for all the systems connecting through Smokescreen and that is an administrative pain.
Getting Basic
We wanted Smokescreen to be simpler to enable, and with Fly we have the advantage of supporting Secrets for all applications. Rather than repurposing TLS CAs to provide a name, we can store a secret with the Smokescreen proxy and with the app that sends requests to the outside world. That secret? For the example, we’ve gone with a PROXY_PASSWORD that we can distribute to all inside the Fly network.
Here’s the Fly Github repository for the Fly Smokescreen.
I’m on the list…
In all cases, what Smokescreen does is turn the identity of an incoming request into a role. That role is then looked up in the acl.yaml file. Here’s the Fly example ACL:
---
version: v1
services:
- name: authed
project: users
action: report
default:
project: other
action: enforce
We’ve gone super simple on the roles here. There’s one and that’s authed. You’re either authed or you fall through to default. The project field is there to make logging more meaningful by associating roles with projects.
The control of what happens with requests comes from the action field; this has three settings: open lets all traffic through, report lets all traffic through but logs the request if it’s not on the list, and enforce only lets through traffic on the list. The list in this example isn’t there, so report logs all requests and enforce blocks all requests.
Adding allowed-domains and a list of domains lets you fine tune these options. For a general purpose block-or-log egress proxy, this example is enough. Smokescreen has more ACL control options, including global allow and deny lists if you want to maintain simple but specfic rules but want to block a long list of sites.
Smokescreen inside
If you are interested in how this modified Smokescreen works, look in the main.go file. This is where the smokescreen code is loaded as a Go package. The program creates a new configuration for Smokescreen with a alternative RoleFromRequest function. It’s this function that extracts the Proxy-Authorization password and checks it against the PROXY_PASSWORD environment variable. If it passes that test, it returns authed as a role. Otherwise, it returns an empty string, denoting no role. It’s this function that you may want to customize to create your own mappings from username and password combinations to Smokescreen roles.
Deploy now
Fly
This is where we show how to deploy on Fly first:
fly init mysmokescreen --import source.fly.toml --org personal
fly set secret PROXY_PASSWORD="somesecret"
fly deploy
And that’s it for Fly; there’ll be a mysmokescreen app set up with Fly’s internal private networking DNS (and external DNS if we needed that, which we don’t here), and it’ll be up and running. Turn on your Fly 6PN (Private Networking) VPN and test it with:
curl -U anyname:somesecret -x mysmokescreen.internal:4750 https://fly.io
And that will return the Fly homepage to you. Run fly logs and you’ll see entries for the opening and closing of the proxy’s connection to fly.io. What’s neat with the Fly deployment is that with just two commands you can deploy the same application globally.
Docker - locally
If you’re on another platform, you should be able to reuse the Dockerfile. Running locally, you just need to do:
docker build -t smokescreen .
docker run -p 4750:4750 --env PROXY_PASSWORD=somesecret smokescreen
And to test, in another session, do:
curl -U anyname:somesecret -x localhost:4750 https://fly.io
You’ll see the log output appearing in the session where you did the docker run. We leave it as an exercise to readers to deploy the application to their own Cloud.
Using a Proxy from an app
To wrap up this article, we present two code examples, one in Go and one in Node, that take from the environment a PROXYURL pointed at our smokescreen and a PROXYPASSWORD for that smokescreen and issue a simple GET for an https: URL.
On Fly, the PROXY_URL can be as simple as “http://mysmokescreen.internal:4750/”. Fly’s 6PN network automatically maps deployed applications’ names and instances into the the .internal TLD for DNS. On other platforms, you’ll have to configure a hostname for your smokescreen and make sure you change it everywhere if you move your proxy.
Calling through an authenticated Proxy from Go
This example uses only the system libraries. There are no extra modules needed.
package main
import (
"encoding/base64"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
)
func main() {
// Set the proxy's URL
proxy, ok := os.LookupEnv("PROXY_URL")
if !ok {
log.Fatal("Set PROXY_URL environment variable")
}
// And parse it.
proxyURL, err := url.Parse(proxy)
if err != nil {
log.Fatal("The proxyURL is unparsable: " + proxy)
}
proxyPASS, ok := os.LookupEnv("PROXY_PASSWORD")
if !ok {
log.Fatal("Set PROXY_PASSWORD environment variable")
}
// Get you a transport that understands Proxies and Proxy authentication
transport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
// Create a usename:password string
auth := "anyname:" + proxyPASS
// Base64 that string with "Basic " prepended to it
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
// Put a header into the proxy connect header
transport.ProxyConnectHeader = http.Header{}
// And then add in the Proxy-Authorization header with our auth string
transport.ProxyConnectHeader.Add("Proxy-Authorization", basicAuth)
// Now we are ready to get pages, just create HTTP clients which use the
// Proxy transport.
client := &http.Client{Transport: transport}
rawURL := "https://fly.io"
request, err := http.NewRequest("GET", rawURL, nil)
if err != nil {
fmt.Println(err)
return
}
response, err := client.Do(request)
if err != nil {
fmt.Println(err)
return
}
fmt.Println("Read ok")
if response.StatusCode != 200 {
fmt.Println(response.Status)
}
bs, err := ioutil.ReadAll(response.Body)
fmt.Println(string(bs))
}
Calling through an authenticated Proxy from Node.js
This example uses the https-proxy-agent package.
var url = require('url');
var https = require('https');
var HttpsProxyAgent = require('https-proxy-agent');
// Create a URL for our proxy from the env var
var proxyOpts = new URL(process.env.PROXY_URL);
// Get a password from the environment too
var proxyPass= process.env.PROXY_PASSWORD;
// Inject a Proxy-Authorization header into the proxy using the password
proxyOpts.headers = {
'Proxy-Authorization': 'Basic ' + (`anyname:${proxyPass}`).toString('base64')
};
// Create an HTTPS Proxy Agent with our accumulated options
var agent = new HttpsProxyAgent(proxyOpts);
var options = new URL("https://fly.io");
options.agent = agent;
https.get(options, function (res) {
console.log('"response" event!', res.headers);
res.pipe(process.stdout);
});
Smokescreen summarized
We’ve shown you examples of setting up a custom Smokescreen with password authentication. You’ll find all the code for setting that up at the Fly Github repository for this Smokescreen. Have fun sanitizing your outgoing web requests.