Content tagged haproxy

If you're using Amazon's ELB service with WebSockets, you probably already know you'll need to set the instance protocol as TCP or else the WebSocket will not be able to route messages between the frontend and the backend. This is an inherent limitation to Amazon's ELB that doesn't seem to be changing any time soon since this ticket has been open for over four years at the time of writing, we can safely assume that Amazon does not deem this issue worth fixing.

This poses several problems:

• You cannot have the X-Forwarded-For header for TCP routes
• You are required to enable the proxy protocol to retrieve client connection information

In this post I'll enumerate several issues I found when implementing the proxy protocol support for an Erlang application. Despite being for an Erlang application, the fundamentals will be applicable to other platforms.

I chose to be able to run my application and integration tests with proxy protocol support. I do this so my local environment matches the EC2 environment as closely as possible. To achieve this I opted to use HAProxy locally because it natively supports the proxy protocol.

Your local HAProxy configuration should contain, at a minimum, the following:

global
    stats timeout 30s
    maxconn 1024
    crt-base /path/to/your/crt/base
    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
    log global
    mode http
    timeout connect  5000
    timeout client  50000
    timeout server  50000

frontend device
    mode http
    bind 0.0.0.0:8543 ssl crt cert.pem
    reqadd X-Forwarded-Proto:\ https
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket hdr_beg(Host) -i ws
    use_backend backend-ws if is_websocket
    default_backend backend

backend backend
    mode http
    server srv-1 0.0.0.0:8180 check

backend backend-ws
    mode http
    server srv-1 0.0.0.0:8180 check send-proxy

I'll break this down in parts to explain.

Here we just set up some simple defaults which all HAProxy:

global
    stats timeout 30s
    maxconn 1024
    crt-base /path/to/your/crt/base
    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
    log global
    mode http
    timeout connect  5000
    timeout client  50000
    timeout server  50000

Here is the meat of the configuration for HAProxy:

frontend device
    mode http
    bind 0.0.0.0:8543 ssl crt cert.pem
    reqadd X-Forwarded-Proto:\ https
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket hdr_beg(Host) -i ws
    use_backend backend-ws if is_websocket
    default_backend backend

backend backend
    mode http
    server srv-1 0.0.0.0:8180 check

backend backend-ws
    mode http
    server srv-1 0.0.0.0:8180 check send-proxy

Be aware that all backends must contain the mode http line or else hdr* functions will not work!

The bulk of the routing is done in these two ACLs:

    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket hdr_beg(Host) -i ws
    use_backend backend-ws if is_websocket

First we check if the request has an Upgrade header with the content WebSocket (case insensitively matched). We also route to this backend if the Host header begins with ws. This is enough to route to the correct backend.

The next part of the configuration sets up the backends so they can optionally provide the proxy protocol to downstream servers:

backend backend
    mode http
    server srv-1 0.0.0.0:8180 check

backend backend-ws
    mode http
    server srv-1 0.0.0.0:8180 check send-proxy

The first backend is a simple HTTP backend with no proxy protocol support, this means that if you are connecting to this backend without the Upgrade: WebSocket or Host: ws* header then you will be forwarded on without the proxy protocol. The benefit here is that the backend support for proxy protocol only needs to be exactly where it is needed. This simplifies a later step.

The last backend contains the send-proxy option, this adds proxy protocol support to all downstream requests. The server running here must support the proxy protocol. If not, all client requests will fail with a 400 Bad Request.

Now, we're all set to start forwarding requests to the backend and implementing proxy protocol support in our backend. I'll outline what the backend needs to do and then implement it in Erlang.

The HAProxy specification for the proxy protocol is a great resource but the gist of it is as follows: dumb proxies generally lose client-specific information, such as the IP, when tunneling through one or more proxy servers. This poses a problem when you need that information, for example for analytical or client-tracking purposes.

What the proxy protocol does is insert a header containing the client's information into the beginning of a request, the client is none-the-wiser and downstream servers simply need to parse a single extra line before processing the underlying protocol's request.

All this comes down to is something akin to this:

"PROXY TCP4 255.255.255.255 255.255.255.255 65535 65535\r\n"

Before all client requests, they will contain PROXY, the protocol (in this case TCP4) next comes the layer 3 source address, a space, the layer 3 destination address, a space, the TCP source port, another space, the TCP destination port and finally, CRLF.

Using just this line, you can easily track the client IP through multiple proxies.

The HAProxy set up is so that when testing locally or in your CI server you can inject the proxy protocol lines. This makes it simpler to test since you don't have to optionally enable/disable it depending on environment. HAProxy acts as a "local ELB".

What does this mean for WebSockets through ELB?

Since the ELB cannot add the X-Forwarded-For header we must use this proxy protocol to retain client information. Amazon does not provide an interface for enabling this. You must use the CLI:

This is outlined in the ELB documentation for proxy protocol support, but I will reiterate here for completeness:

Step 1: Create the policy:

aws elb create-load-balancer-policy            \
    --load-balancer-name my-loadbalancer       \
    --policy-name my-ProxyProtocol-policy      \
    --policy-type-name ProxyProtocolPolicyType \
    --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true

You'll need to change the attributes my-loadbalancer to point at the loadbalancer you want to create this policy for. Don't worry about messing this step up, since at this point, you haven't changed how your loadbalancer works. We are simply making the policy available should we wish to enable it.

Step 2: Assign the policy:

aws elb set-load-balancer-policies-for-backend-server \
    --load-balancer-name my-loadbalancer              \
    --instance-port 80 --policy-names my-ProxyProtocol-policy my-existing-policy

Again you'll need to change the my-loadbalancer to point at the loadbalancer you want to change. The --instance-port also needs to be the port you want to enable this proxy policy on. Make sure to also use the same name for --policy-names for both the proxy policy we just created (my-ProxyProtocol-policy if you are following on from the previous example) and any other policies which already exist. This is important since if you have any other policies, calling set-load-balancer-policies-for-backend-server will override the list. SSL certificates appear as policies, so list the policies by using:

aws elb describe-load-balancers --load-balancer my-loadbalancer

Now that we have enabled proxy protocol support on the load balancer, the application will now not be able to service requests through this port unless proxy protocol support is also added there.

Let's do that now.

In our Erlang/Cowboy application we can override the protocol which is used. Typically you would create a cowboy HTTP listener by calling:

AccCount = 10,
TransOpts = [{port, 8080}],
ProtoOpts = [],
cowboy:start_http(custom_name_of_listener, AccCount, TransOpts, ProtoOpts)
What this calls underneath is:

ranch:start_listener(custom_name_of_listener, AccCount,
                     ranch_tcp, TransOpts,
                     cowboy_protocol, ProtoOpts)

Cowboy comes with a typical TCP/HTTP protocol parser, a module named cowboy_protocol. This is what you want to use for normal HTTP listeners. However, since our protocol is the proxy protocol, cowboy will fail to parse incoming requests.

Next I'll implement the proxy protocol request parser that allows us to grab the proxy protocol header and inject it into our cowboy handlers:

The whole file is available in a gist a lot of it is copy and pasted from the cowboy_protocol module since we just need to inject a little bit of code before we parse our request and hand off to the original module. This saves us implementing the entirety of TCP/HTTP handling ourselves. From line 105 onwards is where we parse out the proxy protocol.

Now in your handlers you can call:

ProxyInfo = get(proxy_info)

And you'll get the proxy information the client connected with!

Alternatives If you don't want to go through the rigmarole of implementing proxy protocol support in your application and having custom policies on your load balancers. What can you do?

It depends on the application. If you control both the client code and the backend code, then you could implement a simple HTTP endpoint that associates a client token (username, device_id, whatever) to the IP address they are connecting with. This means that the IP association is done over HTTP and allows the ELB to inject the X-Forwarded-For header.

If you don't control both the client-side code and the backend-side or you're completely tied to using WebSockets. Then I am very interested in any ideas of how you could get around this limitation.

Sources and thanks * https://github.com/ninenines/cowboy/ * https://github.com/ninenines/cowboy/issues/708 * https://github.com/ninenines/cowboy/pull/912 * http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt