Forum Discussion

hc_andy_35682's avatar
hc_andy_35682
Icon for Nimbostratus rankNimbostratus
Mar 22, 2010

OneConnect and Proxy/Squid Load Balancing

Hi All,

 

 

We provide proxy services (http/https) to a large number of schools (each with their own static IP). Each school can have 1000's of connections to the virtual server at any time.

 

 

What we have is a a large squid (proxy) cluster and in front of that is the F5 doing least conns with source address persistence. Each pool member also has a 4000 max conns limit. What we're finding is that several of our squid servers are reaching their max conn limit.

 

 

1/ When the max conn limit is reached, will the client (school) request be served by another available pool member taking into consideration that we have source address pesistence enabled so the client (school) is expecting to hit the same pool member all the time? If the client (school) is served by another pool member, wouldn't this then break the source address persistence?

 

 

2/ Would something like a OneConnect profile help us reduce the number of connections from the LTM to the squid cluster. I've read up on it but not 100% sure how it applies in a squid/proxy environment.

 

 

Is this like having one big pipe (tcp connection) from the LTM to the squid box say for school 1's incoming connections, so that all new connections from school 1 uses that big pipe. What about the return traffic from the squid box back to the school? Is it still returning traffic via this big pipe or does it need to create a new tcp connection for the return traffic?

 

 

Thanks.

 

 

Andy
  • Note to self: When using "SSL Proxy" in your webbrowser it means that the request is sent in cleartext to the forward https-proxy.

     

     

    It turned out that oneconnect (using /32 mask) seems to be working just fine but http profile can only be attached to the vserver for the forward http-proxy and not the forward https-proxy.

     

     

    Perhaps any of you in here who previously have stumbled upon this behaviour and know a workaround other than setting http profile to none?

     

     

    Because when set to none I will also miss AVR (statistics) and ability to use ASM :-(
  • without private key, i do not think we can use either asm or avr with https proxy since you know we are not able to see clear text traffic.
  • I can use a private key because I do ssl-termination further out in the chain, but will that work since the http profile doesnt seem to allow CONNECT as method?

     

     

    When I tested with the http profile enabled for VS_FORWARD-HTTPS the F5 will just send the client a FIN-ACK as soon as it receives the CONNECT request (the F5 doesnt even forward the traffic to the forward-proxy).

     

     

    Can I somehow alter the http profile to allow CONNECT?
  • i am trying to setup in lab but i am a bit confused about your setting. would you mind posting the VS_FORWARD-HTTPS virtual and its pool configuration here?

     

     

    and how did you test? what proxy setting did you set in browser?
  • Sure, I hope the attached code is enough to reproduce this?

     

     

    The setup is:

     

     

    Physical address of the forward http(s)-proxy is 10.1.1.254 which listens to 3128 for http and 3129 for https (in this test-setup I just use one box, in true scenario there will more than one).

     

     

    This is verified to work when you in your browser (firefox) sets the following settings:

     

     

    Manual proxy configuration:

     

    HTTP Proxy: 10.1.1.254:3128

     

    SSL Proxy: 10.1.1.254:3129

     

     

    Virtual address of the forward http(s)-proxy is 10.2.1.1 which also listens to 3128 for http and 3129 for https.

     

     

    Which means that you need these settings in your browser to let F5 do its magic (both as loadbalancer but also as protocol inspector):

     

     

    Manual proxy configuration:

     

    HTTP Proxy: 10.2.1.1:3128

     

    SSL Proxy: 10.2.1.1:3129

     

     

    The VS_PROXY_HTTP uses http profile, http class, analytics, oneconnect and a small tcp-lan-optimized tweak and works like a charm.

     

     

    The VS_PROXY_HTTPS doesnt so I (in the uploaded code) have disabled http profile which also means that I am forced to disable the http class AND analytics 😞 in order to make it work. If you enable the http profile you will notice that the F5 will just send FIN-ACK in return as soon as the client sends its "CONNECT https://www.example.com:443/".

     

     

    As a sidenote I have also tried to disable oneconnect (even if its enabled in the uploaded code).

     

     

    The configuration is pretty straight forward (sure I have tweaked the monitors but thats just to make it less aggressive because each ping etc generates log-entries in the forward-proxy).

     

     

    The logical flow for the packets is more or less:

     

     

    client -> F5 -> forward-http(s)-proxy -> application firewall -> external net

     

     

    The application firewall is set to do SSL-termination so I could load the private key used for this in the F5 and use as server-ssl if needed, but then again - I believe the problem is in how the http profile functions (my best guess so far is that the http profile doesnt allow "CONNECT" as method) simply because the VS_PROXY_HTTPS works when I disable the http profile.

     

     

    There is also a new setting (I think when you setup client/server-ssl) named "proxy ssl" which I wonder if it would help in my case?

     

     

    Oh and the firmware used is v11.1.

     

     

     
    ltm default-node-monitor {
        rule /Common/ICMP_PROXY 
    }
    ltm node /Common/PROXY1 {
        address 10.1.1.254
    }
    ltm pool /Common/POOL_PROXY_HTTP {
        load-balancing-mode fastest-node
        members {
            /Common/PROXY1:3128 {
                address 10.1.1.254
            }
        }
        monitor /Common/inband and /Common/TCP_PROXY 
        service-down-action reset
    }
    ltm pool /Common/POOL_PROXY_HTTPS {
        load-balancing-mode fastest-node
        members {
            /Common/PROXY1:3129 {
                address 10.1.1.254
            }
        }
        monitor /Common/inband and /Common/TCP_PROXY 
        service-down-action reset
    }
    ltm virtual /Common/VS_PROXY_HTTP {
        destination /Common/10.2.1.1:3128
        http-class {
            /Common/HTTPCLASS_PROXY
        }
        ip-protocol tcp
        mask 255.255.255.255
        persist {
            /Common/SOURCE_ADDR_PROXY {
                default yes
            }
        }
        pool /Common/POOL_PROXY_HTTP
        profiles {
            /Common/ANALYTICS_PROXY { }
            /Common/HTTP_PROXY { }
            /Common/ONECONNECT_PROXY { }
            /Common/TCP-LAN-OPTIMIZED_PROXY { }
        }
        vlans-disabled
    }
    ltm virtual /Common/VS_PROXY_HTTPS {
        destination /Common/10.2.1.1:3129
        ip-protocol tcp
        mask 255.255.255.255
        persist {
            /Common/SOURCE_ADDR_PROXY {
                default yes
            }
        }
        pool /Common/POOL_PROXY_HTTPS
        profiles {
            /Common/ONECONNECT_PROXY { }
            /Common/TCP-LAN-OPTIMIZED_PROXY { }
        }
        vlans-disabled
    }
    ltm virtual-address /Common/10.2.1.1 {
        address 10.2.1.1
        mask 255.255.255.255
        traffic-group /Common/traffic-group-1
    }
    ltm monitor icmp /Common/ICMP_PROXY {
        defaults-from /Common/icmp
        destination *
        interval 15
        time-until-up 0
        timeout 46
    }
    ltm monitor tcp /Common/TCP_PROXY {
        defaults-from /Common/tcp
        destination *:*
        interval 15
        time-until-up 0
        timeout 46
    }
    ltm persistence source-addr /Common/SOURCE_ADDR_PROXY {
        app-service none
        defaults-from /Common/source_addr
        timeout 3600
    }
    ltm profile analytics /Common/ANALYTICS_PROXY {
        captured-traffic-external-logging disabled
        captured-traffic-internal-logging enabled
        collect-geo enabled
        collect-http-throughput enabled
        collect-ip enabled
        collect-methods enabled
        collect-page-load-time enabled
        collect-response-codes enabled
        collect-server-latency enabled
        collect-url enabled
        collect-user-agent enabled
        collect-user-sessions enabled
        collected-stats-external-logging disabled
        collected-stats-internal-logging enabled
        defaults-from /Common/analytics
        description none
        notification-by-email disabled
        notification-by-snmp disabled
        notification-by-syslog disabled
        remote-server-ip any6
        remote-server-port 514
        remote-server-syslog-facility local0
        session-timeout 300
        session-timeout-minutes 5
        traffic-capture {
            capturing-for-ANALYTICS_PROXY {
                captured-protocols all
            }
        }
        trust-xff enabled
    }
    ltm profile http /Common/HTTP_PROXY {
        app-service none
        defaults-from /Common/http
        security enabled
    }
    ltm profile httpclass /Common/HTTPCLASS_PROXY {
        app-service none
        asm enabled
        defaults-from /Common/httpclass
    }
    ltm profile one-connect /Common/ONECONNECT_PROXY {
        app-service none
        defaults-from /Common/oneconnect
        max-age 3600
        source-mask 255.255.255.255
    }
    ltm profile tcp /Common/TCP-LAN-OPTIMIZED_PROXY {
        app-service none
        defaults-from /Common/tcp-lan-optimized
        proxy-buffer-high 280000
        proxy-buffer-low 277000
    }
    
  • what proxy are you using? is it squid? may i also have squid configuration?

     

  • i am not much familiar with squid. wondering if there is any special when configuring https proxy i.e. port 3129. is it just another instance like port 3128?

    [root@ve1023:Active] config  b virtual bar1 list
    virtual bar1 {
       snat automap
       pool foo1
       destination 172.28.19.79:3128
       ip protocol 6
       profiles {
          http {}
          tcp {}
       }
    }
    [root@ve1023:Active] config  b pool foo1 list
    pool foo1 {
       members 172.28.19.251:3128 {}
    }
    [root@ve1023:Active] config  b virtual bar2 list
    virtual bar2 {
       snat automap
       pool foo2
       destination 172.28.19.79:3129
       ip protocol 6
       profiles {
          http {}
          tcp {}
       }
    }
    [root@ve1023:Active] config  b pool foo2 list
    pool foo2 {
       members 172.28.19.251:3129 {}
    }
    
    
     curl -Ik https://www.google.com/ -x 172.28.19.79:3129
    HTTP/1.0 200 Connection established
    
    HTTP/1.1 200 OK
    Date: Sun, 15 Jan 2012 04:40:45 GMT
    Expires: -1
    Cache-Control: private, max-age=0
    Content-Type: text/html; charset=ISO-8859-1
    Set-Cookie: PREF=ID=7d5331ea4cb18cc1:FF=0:TM=1326602445:LM=1326602445:S=XqerRof5YDSF_HmF; expires=Tue, 14-Jan-2014 04:40:45 GMT; path=/; domain=.google.com
    Set-Cookie: NID=55=RAt7S2S7r0ObTTPxyMjBDgD32anR8fjyt-4_syPbnTIVIZNXsDp05oB0IvLlm-7Crd7Djrmw5ZJdWyKV5ADooEWHwEj6yzJGDADudQfx8i5mU6PvRa7JVEzdSFjzR2Cs; expires=Mon, 16-Jul-2012 04:40:45 GMT; path=/; domain=.google.com; HttpOnly
    P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
    Server: gws
    X-XSS-Protection: 1; mode=block
    X-Frame-Options: SAMEORIGIN
    Transfer-Encoding: chunked
    
    
    [root@ve1023:Active] config  ssldump -Aed -nni 0.0 port 3129
    New TCP connection 1: 172.28.19.253(35339) <-> 172.28.19.79(3129)
    1326602789.8656 (0.0030)  C>S
    ---------------------------------------------------------------
    CONNECT www.google.com:443 HTTP/1.0
    Host: www.google.com:443
    User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8r zlib/1.2.3 libidn/0.6.5
    Proxy-Connection: Keep-Alive
    
    ---------------------------------------------------------------
    
    New TCP connection 2: 172.28.19.80(35339) <-> 172.28.19.251(3129)
    1326602789.8667 (0.0010)  C>S
    ---------------------------------------------------------------
    CONNECT www.google.com:443 HTTP/1.0
    Host: www.google.com:443
    User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8r zlib/1.2.3 libidn/0.6.5
    Proxy-Connection: Keep-Alive
    
    ---------------------------------------------------------------
    
  • No its not squid and which portnumber one use doesnt really matter. You can use 1234 in your tests if you like.

     

     

    And in this case the F5 doesnt even forward the traffic to the proxyservers when I enable http profile for the VS_PROXY_HTTPS so the "error" is in the F5 and not somewhere else (verified by tcpdump on the F5 unit for the interface connected to the proxyserver).

     

     

    But as soon as I disable the http profile (and also disables http class and analytics since they depend on the http profile) then I can use VS_PROXY_HTTPS.

     

     

    But I can try to redo the tests on monday/tuesday and get back with the failing config (unless someone else got some tips to test at the same time?).
  • Hi nitass,

     

     

    We are configuring F5 to load balance squid web cache proxy:

     

     

    This is the flow:

     

    Client >> F5(rewrite) >>F5 forward to cache proxy servers>>Cache proxy servers >> F5(transparent) >> Client

     

     

    Inbound and outbound traffic falls on the same VLAN. Using same interface

     

    INBOUND: From internet to firewall to F5

     

    OUTBOUND: F5 to Cache servers

     

    VS is configured for the 2 cache servers

     

    Used iRule to rewrite URL

     

     

    Objective:

     

    To be able to rewrite the URL and then gets passed to the cache servers after the rewrite.

     

    Maintain the session

     

    So traffic gets cached.

     

     

     

    Client initiates HTTP request with URL: www.google.com.example.com.ph

     

    With iRule,example.com.ph gets truncated, then returns back www.google.com to the client browser.

     

    This is working, and F5 rewrites it

     

    However, upon rewriting F5 must forward it to the cache servers.

     

     

    Problem:

     

     

    traffic is not getting a hit on the cache servers.

     

    From the iRule, it seems that once URL is rewritten traffic gets redirected and doesn’t go to cache.

     

     

    virtual LB-Voyager {

     

    snat automap

     

    pool LB_Voyager-MIB_3128

     

    destination 10.128.142.60:http

     

    ip protocol tcp

     

    rules LB_Voyager6

     

    persist beta_mysandbox_cookie_insert

     

    profiles {

     

    http_beta_mysandbox {}

     

    tcp {}

     

    }

     

    }

     

    }

     

    pool LB_Voyager-MIB_3128 {

     

    monitor all gateway_icmp and tcp_3128

     

    members {

     

    10.128.142.58:squid {}

     

    10.128.142.61:squid {}

     

    }

     

    }

     

    rule LB_Voyager6 {

     

    when HTTP_REQUEST {

     

    set host [string tolower [HTTP::host]]

     

    set newhost [string map {".example.com.ph" ""} $host]

     

    if {$host ne $newhost} {

     

    HTTP::redirect "http://$newhost[HTTP::uri]"

     

    }

     

    }

     

     

    but we are not hitting the servers.

     

     

    • Spidey_29396's avatar
      Spidey_29396
      Icon for Nimbostratus rankNimbostratus
      Hi Mahmoud ElSoury, Unfortunate client didn't push thru with the project. Thanks! Ferdz