Content tagged haproxy
If you're using Amazon's ELB service with WebSockets, you probably already know you'll need to set the instance protocol as TCP or else the WebSocket will not be able to route messages between the frontend and the backend. This is an inherent limitation to Amazon's ELB that doesn't seem to be changing any time soon since this ticket has been open for over four years at the time of writing, we can safely assume that Amazon does not deem this issue worth fixing.
This poses several problems:
- You cannot have the X-Forwarded-For header for TCP routes
- You are required to enable the proxy protocol to retrieve client connection information
In this post I'll enumerate several issues I found when implementing the proxy protocol support for an Erlang application. Despite being for an Erlang application, the fundamentals will be applicable to other platforms.
I chose to be able to run my application and integration tests with proxy protocol support. I do this so my local environment matches the EC2 environment as closely as possible. To achieve this I opted to use HAProxy locally because it natively supports the proxy protocol.
Your local HAProxy configuration should contain, at a minimum, the following:
global
stats timeout 30s
maxconn 1024
crt-base /path/to/your/crt/base
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
defaults
log global
mode http
timeout connect 5000
timeout client 50000
timeout server 50000
frontend device
mode http
bind 0.0.0.0:8543 ssl crt cert.pem
reqadd X-Forwarded-Proto:\ https
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend backend-ws if is_websocket
default_backend backend
backend backend
mode http
server srv-1 0.0.0.0:8180 check
backend backend-ws
mode http
server srv-1 0.0.0.0:8180 check send-proxy
I'll break this down in parts to explain.
Here we just set up some simple defaults which all HAProxy:
global
stats timeout 30s
maxconn 1024
crt-base /path/to/your/crt/base
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
defaults
log global
mode http
timeout connect 5000
timeout client 50000
timeout server 50000
Here is the meat of the configuration for HAProxy:
frontend device
mode http
bind 0.0.0.0:8543 ssl crt cert.pem
reqadd X-Forwarded-Proto:\ https
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend backend-ws if is_websocket
default_backend backend
backend backend
mode http
server srv-1 0.0.0.0:8180 check
backend backend-ws
mode http
server srv-1 0.0.0.0:8180 check send-proxy
Be aware that all backends must contain the mode http line or else hdr* functions will not work!
The bulk of the routing is done in these two ACLs:
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend backend-ws if is_websocket
First we check if the request has an Upgrade header with the content WebSocket (case insensitively matched). We also route to this backend if the Host header begins with ws. This is enough to route to the correct backend.
The next part of the configuration sets up the backends so they can optionally provide the proxy protocol to downstream servers:
backend backend
mode http
server srv-1 0.0.0.0:8180 check
backend backend-ws
mode http
server srv-1 0.0.0.0:8180 check send-proxy
The first backend is a simple HTTP backend with no proxy protocol
support, this means that if you are connecting to this backend without
the Upgrade: WebSocket
or Host: ws*
header then you will be forwarded
on without the proxy protocol. The benefit here is that the backend
support for proxy protocol only needs to be exactly where it is
needed. This simplifies a later step.
The last backend contains the send-proxy
option, this adds proxy
protocol support to all downstream requests. The server running here
must support the proxy protocol. If not, all client requests will
fail with a 400 Bad Request.
Now, we're all set to start forwarding requests to the backend and implementing proxy protocol support in our backend. I'll outline what the backend needs to do and then implement it in Erlang.
The HAProxy specification for the proxy protocol is a great resource but the gist of it is as follows: dumb proxies generally lose client-specific information, such as the IP, when tunneling through one or more proxy servers. This poses a problem when you need that information, for example for analytical or client-tracking purposes.
What the proxy protocol does is insert a header containing the client's information into the beginning of a request, the client is none-the-wiser and downstream servers simply need to parse a single extra line before processing the underlying protocol's request.
All this comes down to is something akin to this:
"PROXY TCP4 255.255.255.255 255.255.255.255 65535 65535\r\n"
Before all client requests, they will contain PROXY, the protocol (in this case TCP4) next comes the layer 3 source address, a space, the layer 3 destination address, a space, the TCP source port, another space, the TCP destination port and finally, CRLF.
Using just this line, you can easily track the client IP through multiple proxies.
The HAProxy set up is so that when testing locally or in your CI server you can inject the proxy protocol lines. This makes it simpler to test since you don't have to optionally enable/disable it depending on environment. HAProxy acts as a "local ELB".
What does this mean for WebSockets through ELB?
Since the ELB cannot add the X-Forwarded-For header we must use this proxy protocol to retain client information. Amazon does not provide an interface for enabling this. You must use the CLI:
This is outlined in the ELB documentation for proxy protocol support, but I will reiterate here for completeness:
Step 1: Create the policy:
aws elb create-load-balancer-policy \
--load-balancer-name my-loadbalancer \
--policy-name my-ProxyProtocol-policy \
--policy-type-name ProxyProtocolPolicyType \
--policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
You'll need to change the attributes my-loadbalancer to point at the loadbalancer you want to create this policy for. Don't worry about messing this step up, since at this point, you haven't changed how your loadbalancer works. We are simply making the policy available should we wish to enable it.
Step 2: Assign the policy:
aws elb set-load-balancer-policies-for-backend-server \
--load-balancer-name my-loadbalancer \
--instance-port 80 --policy-names my-ProxyProtocol-policy my-existing-policy
Again you'll need to change the my-loadbalancer
to point at the
loadbalancer you want to change. The --instance-port
also needs to
be the port you want to enable this proxy policy on. Make sure to also
use the same name for --policy-names
for both the proxy policy we
just created (my-ProxyProtocol-policy
if you are following on from
the previous example) and any other policies which already
exist. This is important since if you have any other policies,
calling set-load-balancer-policies-for-backend-server
will override
the list. SSL certificates appear as policies, so list the policies by
using:
aws elb describe-load-balancers --load-balancer my-loadbalancer
Now that we have enabled proxy protocol support on the load balancer, the application will now not be able to service requests through this port unless proxy protocol support is also added there.
Let's do that now.
In our Erlang/Cowboy application we can override the protocol which is used. Typically you would create a cowboy HTTP listener by calling:
AccCount = 10,
TransOpts = [{port, 8080}],
ProtoOpts = [],
cowboy:start_http(custom_name_of_listener, AccCount, TransOpts, ProtoOpts)
What this calls underneath is:
ranch:start_listener(custom_name_of_listener, AccCount,
ranch_tcp, TransOpts,
cowboy_protocol, ProtoOpts)
Cowboy comes with a typical TCP/HTTP protocol parser, a module named cowboy_protocol. This is what you want to use for normal HTTP listeners. However, since our protocol is the proxy protocol, cowboy will fail to parse incoming requests.
Next I'll implement the proxy protocol request parser that allows us to grab the proxy protocol header and inject it into our cowboy handlers:
The whole file is available in a gist a lot of it is copy and pasted from the cowboy_protocol module since we just need to inject a little bit of code before we parse our request and hand off to the original module. This saves us implementing the entirety of TCP/HTTP handling ourselves. From line 105 onwards is where we parse out the proxy protocol.
Now in your handlers you can call:
ProxyInfo = get(proxy_info)
And you'll get the proxy information the client connected with!
Alternatives If you don't want to go through the rigmarole of implementing proxy protocol support in your application and having custom policies on your load balancers. What can you do?
It depends on the application. If you control both the client code and the backend code, then you could implement a simple HTTP endpoint that associates a client token (username, device_id, whatever) to the IP address they are connecting with. This means that the IP association is done over HTTP and allows the ELB to inject the X-Forwarded-For header.
If you don't control both the client-side code and the backend-side or you're completely tied to using WebSockets. Then I am very interested in any ideas of how you could get around this limitation.
Sources and thanks * https://github.com/ninenines/cowboy/ * https://github.com/ninenines/cowboy/issues/708 * https://github.com/ninenines/cowboy/pull/912 * http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
This blog covers amazon, distributed computing, ec2, erlang, haproxy, kubernetes, websockets