Engineering

Authenticating S3 Proxy

FINBOURNE

01/03/2017

Using nginx and oauth2_proxy to deliver S3 based content (and azure blob store)

tl;dr

Making oauth2_proxy work with nginx, and redirecting the output was hard.
It is possible to have an oauth2 authenticated website that serves static content from Azure Blob Stores, or S3 buckets.

S3 static sites

AWS S3 buckets allows you to serve up static html as a website – see here. But what if you want to control access to the website?

Our setup has us using AWS for everything except our authentication, which we are doing via Azure Active Directory. So the way to secure a static website would be to host it behind a proxy, and force authentication via AzureAD. For the authentication piece, we chosen to use oauth2_proxy. This could also be the proxy, but we already use nginx in a number of places, so we leverage that.

To set it up in AWS, we use Elastic Beanstalk to host a multi-container setup. The deployment basically takes a json’ified version of a docker-compose file. So what does the docker-compose.yml look like:

replace any items with appropriate values for you.

version: '2'
services:

  oauth2_proxy:
    image: 0/oauth2_proxy:latest
    environment:
    - OAUTH2_PROXY_CLIENT_ID=0
    - OAUTH2_PROXY_CLIENT_SECRET=0
    - OAUTH2_PROXY_COOKIE_SECRET=0
    - OAUTH2_PROXY_EMAIL_DOMAIN=finbourne.com
    - OAUTH2_PROXY_TENANT_ID=0
    command:

  nginx:
    image: openresty/openresty:alpine
    ports:
      - "80:80"
      - "443:443"
    links:
      - oauth2_proxy
    volumes:
     - ./aws/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf
     - ./.certs:/etc/ssl/certs

Since Elastic Beanstalk multidocker container deployments can’t pass command arguments to containers, we’ve had to roll our own version of oauth2_proxy. This requires the dockerfile, and a startup.sh script:

dockerfile

FROM alpine:latest

ENV OAUTH2_PROXY_VERSION 2.1
ENV GO_VERSION 1.6

RUN apk --update add curl

RUN curl -sL -o oauth2_proxy.tar.gz \
    "https://github.com/bitly/oauth2_proxy/releases/download/v$OAUTH2_PROXY_VERSION/oauth2_proxy-$OAUTH2_PROXY_VERSION.linux-amd64.go$GO_VERSION.tar.gz" \
  && tar xzvf oauth2_proxy.tar.gz \
  && mv oauth2_proxy-*/oauth2_proxy /bin/ \
  && chmod +x /bin/oauth2_proxy \
  && rm -r oauth2_proxy*

EXPOSE 4180

ADD startup.sh .

CMD ./startup.sh

startup.sh

#!/bin/ash

oauth2_proxy --email-domain=$OAUTH2_PROXY_EMAIL_DOMAIN \
             --provider=azure \
             --azure-tenant=$OAUTH2_PROXY_TENANT_ID \
             --upstream=https://0.0.0.0 \
             --http-address=0.0.0.0:4180

That just leaves getting nginx to play nicely with S3. In Azure it was dead simple, just proxy pass to a SAS tokenised url. S3 on the other hand is quite a bit different. But a bit of googling later, many failed attempts and then eventually thanks to https://github.com/lovelysystems/nginx-examples, I managed to cobble together the following:

nginx.conf – once again, substitute your own values into vars.

#user  nginx;
worker_processes  1;

error_log  /usr/local/openresty/nginx/logs/error.log warn;
pid        /usr/local/openresty/nginx/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /usr/local/openresty/nginx/conf/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /usr/local/openresty/nginx/logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;


server {
    listen 80;

    location /health {
      return 200 'Healthy!';
      add_header Content-Type text/plain;
    }

    location / {
      return 301 https://$host$request_uri;
    }
}

server {
    listen 443;

  proxy_intercept_errors off;
  proxy_send_timeout 120;
  proxy_read_timeout 300;

    client_max_body_size 3G;

    ssl_certificate           /etc/ssl/certs/cert.crt;
    ssl_certificate_key       /etc/ssl/certs/cert.key;

    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    access_log            /usr/local/openresty/nginx/logs/access.log;

    keepalive_timeout  5 5;
    proxy_buffering    off;

    location = /oauth2/auth {
        internal;
        proxy_pass http://oauth2_proxy:4180;
    }

    location /oauth2/ {
        proxy_pass http://oauth2_proxy:4180;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
    }

    location / {
        auth_request /oauth2/auth;
        error_page 401 = /oauth2/start?rd=$request_uri;  # required to get redirection working correctly after login

        set $s3_bucket '';

        # Setup AWS Authorization header
        set $aws_signature '';

        # the only reason we need lua is to get the current date
        set_by_lua $now "return ngx.cookie_time(ngx.time())";

        #the  access key
        set $aws_access_key '';
        set $aws_secret_key '';

        # the actual string to be signed
        # see: http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html
        set $string_to_sign "$request_method\n\n\n\nx-amz-date:$now\n/$s3_bucket/test/unit-results$request_uri";

        # create the hmac signature
        set_hmac_sha1 $aws_signature $aws_secret_key $string_to_sign;
        # encode the signature with base64
        set_encode_base64 $aws_signature $aws_signature;
        proxy_set_header x-amz-date $now;
        proxy_set_header Authorization "AWS $aws_access_key:$aws_signature";

        proxy_http_version     1.1;
        proxy_set_header       Host $s3_bucket.s3.amazonaws.com;
        proxy_redirect off;
        proxy_set_header       X-Real-IP $remote_addr;
        proxy_set_header       X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_hide_header      Content-Type;
        proxy_hide_header      x-amz-id-2;
        proxy_hide_header      x-amz-request-id;

        resolver               8.8.8.8 valid=300s;
        proxy_pass             https://$s3_bucket.s3.amazonaws.com/test/unit-results$request_uri;
        proxy_read_timeout     90;
    }
  }
}

since this requires the use of some non-standard nginx modules, we had to use the openresty nginx docker container.

And that’s it for S3.

And what about Azure??

Azure was very similar except for the nginx.config, the salient difference outlined here:

  location / {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/start?rd=$request_uri;  # required to get redirection working correctly after login

      proxy_hide_header Content-Type;
      set $args $args&st=2017-01-12T17%3A13%3A00Z&se=2017-03-13T17%3A13%3A00Z&sp=r&sv=2015-12-11&sr=c&sig=;
      proxy_pass          https://.blob.core.windows.net/tests/;
      proxy_read_timeout  90;
  }

Conclusion

The devil is almost always in the detail. I didn’t have redirect working after sign-in, and desperately wanted it to work. I couldn’t find any help online with it. I did open an issue on the oauth2_proxy but there was no immediate help, so I soldiered on by myself, eventually posting back the solution to fulfil my duties as an upstanding netizen.

In order to test it with AzureAD, the code had to be deployed. That doesn’t make for fast iteration devops. I must have burned a day and a half getting that one line right. For me, worth it. I burned a few hours getting the S3 authentication bit right too. Also worth it.

FINBOURNE

01/03/2017