How to Prepare Your Network Infrastructure to Add HPC Clusters for AI to Your Data Center
HPC AI clusters are getting deployed as highly-engineered 'lego blocks' which are opaque to established data center operations and standards. By taking advantage of established Kubernetes based networking solutions that provide high-speed intelligent networking, you can save yourself from expensive cost overruns, data center re-auditing, and delays. By using Kubernetes based solutions which take advantage of the high-speed networking solutions already required by HP AI deployments, you further optimize your investment in AI.High Performance HMAC Cookie Signing
Problem this snippet solves: General Information The outlined iRule implements a Hash-based Message Authentication Code (see RFC 2104) cookie signing functionality, to provide an additional security layer for sensitive session cookie information. The iRule computes an HMAC verification token based on X509 certificate- or TCP-Connection information and a given session cookie and then passes the results as an additional HTTP cookie to the client. If the HMAC cookie is missing on subsequent requests, or a mismatch between the X509 certificate-, TCP-Connection- or session cookie information and the HMAC verification token is identified, then the given session cookie would become silently removed from the HTTP request to the backend system. Thus would then result in re-initialization and/or a re-authentication of the user session on the backend. Security benefits of HMAC cookie signing The provided HMAC based cookie signing mechanism has some very unique advantages. Providing tamper resistance to cookie information without utilizing CPU intensive per-request cookie decryption. Transparently binding of X509 certificate information to sensitive session cookies, so that ANY KNOWN FORM of cookie hijacking on smartcard authenticated connections can be securely mitigated. Even those between two legitimate smartcard users! Transparently binding of client IP address or F5 GeoLocation information to sensitive session cookies, so that an additional protection layer against cookie hijacking or general misuse can be provided. Performance consideration The iRule is highly performance optimized to provide an as fast as possible request throughput. The iRule utilizes a RAM cache for previously calculated HMAC tokens, to shrink the required CPU cycles for the per-request cookie verification to an absolute minimum. The RAM caches are based on run-time modified $static::variable(arrays) to avoid any form of cross TMM communications and TMM connection parking situations like the [table] command would do, while still being fully CMP-compliant. The RAM caches are built and maintained on each TMM core independently and do support a RULE_INIT driven and also configurable garbage collection interval using the [after -periodic] syntax and a maximum cache size limiter to optimize and protect the memory/health of the plattform. When a garbage collection occurs, then all previously cached HMAC tokens are getting flushed as a whole and recalculated on demand without interrupting the active sessions. The performance test used to measure and optimize the performance of the iRule was built on a two core LTM-1600 unit, with synthetically pre-filled RAM caches containing 100.000 unique HMAC tokens on each TMM core. The cookies where based on random ASP.NET session ids in combination with X509 CNAME information and resulted to a memory footprint of roughly 17,5 Mbyte per TMM core. The test scenario contains a single user session requesting consecutive 25 page impressions on a website with 40 web objects (1000 requests) using four independent HTTP keep-alive connections. -------------------------------------------- Ltm::Rule Event: iRule_2_Delete:HTTP_REQUEST -------------------------------------------- Priority 500 Executions Total 1.0K Failures 0 Aborts 0 CPU Cycles on Executing Average 67.6K Maximum 189.4K Minimum 27.2K --------------------------------------------- Ltm::Rule Event: iRule_2_Delete:HTTP_RESPONSE --------------------------------------------- Priority 500 Executions Total 1.0K Failures 0 Aborts 0 CPU Cycles on Executing Average 29.9K Maximum 212.6K Minimum 19.4K Note: The rather high maximum CPU values are cause by a single Set-Cookie HMAC calculation in the HTTP_RESPONSE event, and an additional HMAC calculation on the second TMM core on its first HTTP_REQUEST event. Based on the differences of the Average and Maximum CPU cycles needed for the different code paths of the HMAC verification, the offloading factor of the RAM cache can be specified at round about 65% saved CPU cycles per HTTP_REQUEST event. Credits Special thanks goes to the DevCentral member Devon Twesten from "Booz Allen Hamilton", which came up with the idea of HMAC signing session cookies based on X509 certificate information as an additional protection layer. You may read his original posting here. Cheers, Kai How to use this snippet: Usage: Create a new iRule and copy/paste the provided iRule into it. Identify your session cookie name and modify the RULE_INIT section to reflect your environment and configure the desired cache garbage collection interval and maximum cache size limits as needed. Modify the $hmac_input variable creation examples as needed to include X509 information, Client IP address or F5 GeoLocation information. Attach the iRule to your Vitual Server. Test the iRule by trying to modify the HMAC or session cookie. Keep an eye on the provided HMAC cache garbage collection log entires and tweak the garbage collection interval as needed. Code : # # Deployment specific iRule events # # Note: Use just one of the outlined deployment specific iRule events. Combining them would be possible by changing [set hmac_input] to [append hmac_input], but it wouldn't make much sense to do so. when CLIENT_ACCEPTED { # log -noname local0.debug "HMAC: CLIENT_ACCEPTED: Setting client IP addr as HMAC token input." set hmac_input [IP::client_addr] } when CLIENT_ACCEPTED { # log -noname local0.debug "HMAC: CLIENT_ACCEPTED: Setting GeoLocation country data as HMAC token input." set hmac_input [whereis [IP::client_addr] country] } when CLIENTSSL_HANDSHAKE { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: Checking if SSL handshake used client certificate." if { ( [SSL::cert count] > 0 ) and ( [set x509_subject [X509::subject [SSL::cert 0]]] ne "" ) } then { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: Setting the X509 subject name as HMAC token input." if { [set hmac_input [X509::subject [SSL::cert 0]]] eq "" } then { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: The certificate does not contain a X509 subject name. Rejecting the session..." reject } } else { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: The SSL handshake does not used any client certificate. Rejecting the session..." reject } } # # HMAC verification related iRule events # when RULE_INIT { # Modify the settings below to reflect your application and insert a strong enougth HMAC signing key. set static::hmac_signing_key "AzaZ5678901234567XYZ34567XYZ" ;# The HMAC256 signing key should be ideally a 256bit random key set static::session_cookie "ASP.NET_SessionId";# Name of the Session Cookie set static::hmac_cookie "ASP.NET_SessionId_HMAC";# Name of the HMAC Cookie set static::hmac_cache_maxsize 100000;# Maximum number of HMAC cache entries per TMM core # Initialize the HMAC cache array unset -nocomplain static::hmac_cache set static::hmac_cache(count) 0 # Tweak the -periodic interval of the HMAC garbage collection as needed (3600000 msec = 1hour, 86400000 msec = 1day) after 86400000 -periodic { if { [array exists static::hmac_cache] } then { log -noname local0.debug "HMAC: Cache Array Recycling: Clearing the HMAC cache array on TMM [TMM::cmp_unit] with currently $static::hmac_cache(count) entries" unset -nocomplain static::hmac_cache set static::hmac_cache(count) 0 } } } when HTTP_REQUEST { if { [catch { # log -noname local0.debug "HMAC: HTTP_REQUEST: Query the HMAC cache array on TMM[TMM::cmp_unit] for existence of pre-computed HMAC values." if { $static::hmac_cache([HTTP::cookie value $static::session_cookie]:$hmac_input) eq [HTTP::cookie value $static::hmac_cookie] } then { # log -noname local0.debug "HMAC: HTTP_REQUEST: HMAC cookie is verified using a cached HMAC token on TMM[TMM::cmp_unit]. Allowing the session cookie \"$static::session_cookie\" to pass." } else { # log -noname local0.debug "HMAC: HTTP_REQUEST: HMAC cookie is NOT verified using a cached HMAC token on TMM[TMM::cmp_unit]. Removing session cookie \"$static::session_cookie\" from current request." HTTP::cookie remove $static::session_cookie } }]} then { # log -noname local0.debug "HMAC: HTTP_REQUEST: The cache on TMM[TMM::cmp_unit] didn't contain a pre-computed HMAC value. Extracting received session_cookie and hmac_cookie values." if { ( [set session_cookie [HTTP::cookie value $static::session_cookie]] ne "" ) and ( [set hmac_cookie [HTTP::cookie value $static::hmac_cookie]] ne "" ) } then { # log -noname local0.debug "HMAC: HTTP_REQUEST: The request contains the app cookie \"$static::session_cookie\" = \"$session_cookie\" and HMAC cookie \"$static::hmac_cookie\" = \"$hmac_cookie\"." # log -noname local0.debug "HMAC: HTTP_REQUEST: Compute a fresh HMAC token for message \"$session_cookie:$hmac_input\" using Key \"$static::hmac_signing_key\"." set hmac_token [b64encode [CRYPTO::sign -alg hmac-sha256 -key $static::hmac_signing_key "$session_cookie:$hmac_input"]] # log -noname local0.debug "HMAC: HTTP_REQUEST: Compare the computed HMAC token \"$hmac_token\" with received HMAC cookie \"$static::hmac_cookie\" = \"$hmac_cookie\"." if { $hmac_token eq $hmac_cookie } then { # log -noname local0.debug "HMAC: HTTP_REQUEST: The HMAC cookie $static::hmac_cookie is verified using the computed HMAC token. Allowing the session cookie \"$static::session_cookie\" to pass." if { [incr static::hmac_cache(count)] > $static::hmac_cache_maxsize } then { # log -noname local0.debug "HMAC: HTTP_REQUEST: The HMAC cache has reached the maximum size of $static::hmac_cache_maxsize. Clearing the HMAC cache on TMM [TMM::cmp_unit]." unset -nocomplain static::hmac_cache set static::hmac_cache(count) 1 } # log -noname local0.debug "HMAC: HTTP_REQUEST: Storing the computed HMAC token \"$hmac_token\" into the HMAC cache array." set static::hmac_cache($session_cookie:$hmac_input) $hmac_token } else { # log -noname local0.debug "HMAC: HTTP_REQUEST: The HMAC cookie $static::hmac_cookie is NOT verified using the computed HMAC token. Removing session cookie \"$static::session_cookie\" from current request." HTTP::cookie remove $static::session_cookie } } else { # log -noname local0.debug "HMAC: HTTP_REQUEST: Current request didn't contain session_cookie and/or hmac_cookie values. Removing session cookie \"$static::session_cookie\" from current request." HTTP::cookie remove $static::session_cookie } } } when HTTP_RESPONSE { # log -noname local0.debug "HMAC: HTTP_RESPONSE: Checking if HTTP response sets the app cookie \"$static::session_cookie\"." if { [set session_cookie [HTTP::cookie value $static::session_cookie]] ne "" } then { # log -noname local0.debug "HMAC: HTTP_RESPONSE: The response contains the app cookie \"$static::session_cookie\" = \"$session_cookie\"." if { [catch { # log -noname local0.debug "HMAC: HTTP_RESPONSE: Inserting cached HMAC cookie \"$static::hmac_cookie\" = \"$static::hmac_cache($session_cookie:$hmac_input)\" to the response." HTTP::header insert "Set-Cookie" "$static::hmac_cookie=$static::hmac_cache($session_cookie:$hmac_input); HttpOnly; Secure; Path=/" }]} then { # log -noname local0.debug "HMAC: HTTP_RESPONSE: The cache on TMM[TMM::cmp_unit] didn't contain a pre-computed HMAC value." if { [incr static::hmac_cache(count)] > $static::hmac_cache_maxsize } then { # log -noname local0.debug "HMAC: HTTP_RESPONSE: The HMAC cache has reached the maximum size of $static::hmac_cache_maxsize. Clearing the HMAC cache on TMM [TMM::cmp_unit]." unset -nocomplain static::hmac_cache set static::hmac_cache(count) 1 } # log -noname local0.debug "HMAC: HTTP_RESPONSE: Compute a fresh HMAC token for message \"$session_cookie:$hmac_input\" using Key \"$static::hmac_signing_key\". HTTP::header insert "Set-Cookie" "$static::hmac_cookie=[set static::hmac_cache($session_cookie:$hmac_input) [b64encode [CRYPTO::sign -alg hmac-sha256 -key $static::hmac_signing_key "$session_cookie:$hmac_input"]]]; HttpOnly; Secure; Path=/" # log -noname local0.debug "HMAC: HTTP_RESPONSE: Inserting computed HMAC cookie \"$static::hmac_cookie\" = \"$static::hmac_cache($session_cookie:$hmac_input)\" to the response and storing the computed HMAC token into the HMAC cache array." } } } Tested this on version: 12.0High Performance HMAC Cookie Signing (for v10.x)
Problem this snippet solves: General Information The outlined iRule implements a Hash-based Message Authentication Code (see RFC 2104) cookie signing functionality, to provide an additional security layer for sensitive session cookie information. The iRule computes an HMAC verification token based on X509 certificate- or TCP-Connection information and a given session cookie and then passes the results as an additional HTTP cookie to the client. If the HMAC cookie is missing on subsequent requests, or a mismatch between the X509 certificate-, TCP-Connection- or session cookie information and the HMAC verification token is identified, then the given session cookie would become silently removed from the HTTP request to the backend system. Thus would then result in re-initialization and/or a re-authentication of the user session on the backend. Security benefits of HMAC cookie signing The provided HMAC based cookie signing mechanism has some very unique advantages. Providing tamper resistance to cookie information without utilizing CPU intensive per-request cookie decryption. Transparently binding of X509 certificate information to sensitive session cookies, so that ANY KNOWN FORM of cookie hijacking on smartcard authenticated connections can be securely mitigated. Even those between two legitimate smartcard users! Transparently binding of client IP address or F5 GeoLocation information to sensitive session cookies, so that an additional protection layer against cookie hijacking or general misuse can be provided. Performance consideration The iRule is highly performance optimized to provide an as fast as possible request throughput. The iRule utilizes a RAM cache for previously calculated HMAC tokens, to shrink the required CPU cycles for the per-request cookie verification to an absolute minimum. The RAM caches are based on run-time modified $static::variable(arrays) to avoid any form of cross TMM communications and TMM connection parking situations like the [table] command would do, while still being fully CMP-compliant. The RAM caches are built and maintained on each TMM core independently and do support a RULE_INIT driven and also configurable garbage collection interval using the [after -periodic] syntax and a maximum cache size limiter to optimize and protect the memory/health of the plattform. When a garbage collection occurs, then all previously cached HMAC tokens are getting flushed as a whole and recalculated on demand without interrupting the active sessions. The performance test used to measure and optimize the performance of the iRule was built on a two core LTM-1600 unit, with synthetically pre-filled RAM caches containing 100.000 unique HMAC tokens on each TMM core. The cookies where based on random ASP.NET session ids in combination with X509 CNAME information and resulted to a memory footprint of roughly 17,5 Mbyte per TMM core. The test scenario contains a single user session requesting consecutive 25 page impressions on a website with 40 web objects (1000 requests) using four independent HTTP keep-alive connections. -------------------------------------------- Ltm::Rule Event: iRule_2_Delete:HTTP_REQUEST -------------------------------------------- Priority 500 Executions Total 1.0K Failures 0 Aborts 0 CPU Cycles on Executing Average 69.7K Maximum 486.0K Minimum 32.8K --------------------------------------------- Ltm::Rule Event: iRule_2_Delete:HTTP_RESPONSE --------------------------------------------- Priority 500 Executions Total 1.0K Failures 0 Aborts 0 CPU Cycles on Executing Average 30.6K Maximum 564.6K Minimum 17.7K Note: The rather high maximum CPU values are cause by a single Set-Cookie HMAC calculation in the HTTP_RESPONSE event, and an additional HMAC calculation on the second TMM core on its first HTTP_REQUEST event. Based on the differences of the Average and Maximum CPU cycles needed for the different HMAC verification code paths, the offloading factor of the RAM cache can be specified at round about 85% saved CPU cycles per HTTP_REQUEST event. Warning: TMOS version 10.x don't have a specialized iRule command to compute HMAC tokens. The used HMAC token generator of this iRule is purely written in TCL and provided by F5 as a workaround for v10.x platforms. The performance of this TCL code is somewhat slower than the on-purpose HMAC token generator which is available in v11.x and later platforms. Although the RAM caches are doing a really great job to offload the fresh HMAC generation to a great extent , the interval based garbage collection of the RAM cache or certain flooding attacks may still cause some CPU spikes. Credits Special thanks goes to the DevCentral member Devon Twesten from "Booz Allen Hamilton", which came up with the idea of HMAC signing session cookies based on X509 certificate information as an additional protection layer. You may read his original posting here. Cheers, Kai How to use this snippet: Usage: Create a new iRule and copy/paste the provided iRule into it. Identify your session cookie name and modify the RULE_INIT section to reflect your environment and configure the desired cache garbage collection interval and maximum cache size limits as needed. Modify the $hmac_input variable creation examples as needed to include X509 information, Client IP address or F5 GeoLocation information. Attach the iRule to your Vitual Server. Test the iRule by trying to modify the HMAC or session cookie. Keep an eye on the provided HMAC cache garbage collection log entires and tweak the garbage collection interval as needed. Code : # # Deployment specific iRule events # # Note: Use just one of the outlined deployment specific iRule events. Combining them would be possible by changing [set hmac_input] to [append hmac_input], but it wouldn't make much sense to do so. when CLIENT_ACCEPTED { # log -noname local0.debug "HMAC: CLIENT_ACCEPTED: Setting client IP addr as HMAC token input." set hmac_input [IP::client_addr] } when CLIENT_ACCEPTED { # log -noname local0.debug "HMAC: CLIENT_ACCEPTED: Setting GeoLocation country data as HMAC token input." set hmac_input [whereis [IP::client_addr] country] } when CLIENTSSL_HANDSHAKE { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: Checking if SSL handshake used client certificate." if { ( [SSL::cert count] > 0 ) and ( [set x509_subject [X509::subject [SSL::cert 0]]] ne "" ) } then { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: Setting the X509 subject name as HMAC token input." if { [set hmac_input [X509::subject [SSL::cert 0]]] eq "" } then { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: The certificate does not contain a X509 subject name. Rejecting the session..." reject } } else { # log -noname local0.debug "HMAC: CLIENTSSL_HANDSHAKE: The SSL handshake does not used any client certificate. Rejecting the session..." reject } } # # HMAC verification related iRule events # when RULE_INIT { # Modify the settings below to reflect your application and insert a strong enougth HMAC signing key. set static::hmac_signing_key "AzaZ5678901234567XYZ34567XYZ" ;# The HMAC256 signing key should be ideally a 256bit random key set static::session_cookie "ASP.NET_SessionId";# Name of the Session Cookie set static::hmac_cookie "ASP.NET_SessionId_HMAC";# Name of the HMAC Cookie set static::hmac_cache_maxsize 100000;# Maximum number of HMAC cache entries per TMM core # Initialize the HMAC cache array unset -nocomplain static::hmac_cache set static::hmac_cache(count) 0 # Tweak the -periodic interval of the HMAC garbage collection as needed (3600000 msec = 1hour, 86400000 msec = 1day) after 86400000 -periodic { if { [array exists static::hmac_cache] } then { log -noname local0.debug "HMAC: Cache Array Recycling: Clearing the HMAC cache array on TMM [TMM::cmp_unit] with currently $static::hmac_cache(count) entries" unset -nocomplain static::hmac_cache set static::hmac_cache(count) 0 } } # TCL Macro to support HMAC token generation on v10.X plattforms (see https://devcentral.f5.com/s/articles/hmac) set static::genarate_hmac_token { set bsize 64 if { [string length $static::hmac_signing_key] > $bsize } { set key [sha256 $static::hmac_signing_key] } else { set key $static::hmac_signing_key } set ipad "" set opad "" for { set j 0 }{ $j < [string length $key] }{ incr j }{ binary scan $key @${j}H2 k set o [expr 0x$k ^ 0x5c] set i [expr 0x$k ^ 0x36] append ipad [format %c $i] append opad [format %c $o] } for { }{ $j < $bsize }{ incr j }{ append ipad 6 append opad \\ } set hmac_token [b64encode [sha256 $opad[sha256 "${ipad}$session_cookie:$hmac_input"]]] } } when HTTP_REQUEST { if { [catch { # log -noname local0.debug "HMAC: HTTP_REQUEST: Query the HMAC cache array on TMM[TMM::cmp_unit] for existence of pre-computed HMAC values." if { $static::hmac_cache([HTTP::cookie value $static::session_cookie]:$hmac_input) eq [HTTP::cookie value $static::hmac_cookie] } then { # log -noname local0.debug "HMAC: HTTP_REQUEST: HMAC cookie is verified using a cached HMAC token on TMM[TMM::cmp_unit]. Allowing the session cookie \"$static::session_cookie\" to pass." } else { # log -noname local0.debug "HMAC: HTTP_REQUEST: HMAC cookie is NOT verified using a cached HMAC token on TMM[TMM::cmp_unit]. Removing session cookie \"$static::session_cookie\" from current request." HTTP::cookie remove $static::session_cookie } }]} then { # log -noname local0.debug "HMAC: HTTP_REQUEST: The cache on TMM[TMM::cmp_unit] didn't contain a pre-computed HMAC value. Extracting received session_cookie and hmac_cookie values." if { ( [set session_cookie [HTTP::cookie value $static::session_cookie]] ne "" ) and ( [set hmac_cookie [HTTP::cookie value $static::hmac_cookie]] ne "" ) } then { # log -noname local0.debug "HMAC: HTTP_REQUEST: The request contains the app cookie \"$static::session_cookie\" = \"$session_cookie\" and HMAC cookie \"$static::hmac_cookie\" = \"$hmac_cookie\"." # log -noname local0.debug "HMAC: HTTP_REQUEST: Compute a fresh HMAC token for message \"$session_cookie:$hmac_input\" using Key \"$static::hmac_signing_key\"." eval $static::genarate_hmac_token # log -noname local0.debug "HMAC: HTTP_REQUEST: Compare the computed HMAC token \"$hmac_token\" with received HMAC cookie \"$static::hmac_cookie\" = \"$hmac_cookie\"." if { $hmac_token eq $hmac_cookie } then { # log -noname local0.debug "HMAC: HTTP_REQUEST: The HMAC cookie $static::hmac_cookie is verified using the computed HMAC token. Allowing the session cookie \"$static::session_cookie\" to pass." if { [incr static::hmac_cache(count)] > $static::hmac_cache_maxsize } then { log -noname local0.debug "HMAC: HTTP_REQUEST: The HMAC cache has reached the maximum size of $static::hmac_cache_maxsize. Clearing the HMAC cache on TMM [TMM::cmp_unit]." unset -nocomplain static::hmac_cache set static::hmac_cache(count) 1 } # log -noname local0.debug "HMAC: HTTP_REQUEST: Storing the computed HMAC token \"$hmac_token\" into the HMAC cache array." set static::hmac_cache($session_cookie:$hmac_input) $hmac_token } else { # log -noname local0.debug "HMAC: HTTP_REQUEST: The HMAC cookie $static::hmac_cookie is NOT verified using the computed HMAC token. Removing session cookie \"$static::session_cookie\" from current request." HTTP::cookie remove $static::session_cookie } } else { # log -noname local0.debug "HMAC: HTTP_REQUEST: Current request didn't contain session_cookie and/or hmac_cookie values. Removing session cookie \"$static::session_cookie\" from current request." HTTP::cookie remove $static::session_cookie } } } when HTTP_RESPONSE { # log -noname local0.debug "HMAC: HTTP_RESPONSE: Checking if HTTP response sets the app cookie \"$static::session_cookie\"." if { [set session_cookie [HTTP::cookie value $static::session_cookie]] ne "" } then { # log -noname local0.debug "HMAC: HTTP_RESPONSE: The response contains the app cookie \"$static::session_cookie\" = \"$session_cookie\"." if { [catch { HTTP::header insert "Set-Cookie" "$static::hmac_cookie=$static::hmac_cache($session_cookie:$hmac_input); HttpOnly; Secure; Path=/" # log -noname local0.debug "HMAC: HTTP_RESPONSE: Inserting cached HMAC cookie \"$static::hmac_cookie\" = \"$static::hmac_cache($session_cookie:$hmac_input)\" to the response." }]} then { # log -noname local0.debug "HMAC: HTTP_RESPONSE: The cache on TMM[TMM::cmp_unit] didn't contain a pre-computed HMAC value." if { [incr static::hmac_cache(count)] > $static::hmac_cache_maxsize } then { log -noname local0.debug "HMAC: HTTP_RESPONSE: The HMAC cache has reached the maximum size of $static::hmac_cache_maxsize. Clearing the HMAC cache on TMM [TMM::cmp_unit]." unset -nocomplain static::hmac_cache set static::hmac_cache(count) 1 } # log -noname local0.debug "HMAC: HTTP_RESPONSE: Compute a fresh HMAC token for message \"$session_cookie:$hmac_input\" using Key \"$static::hmac_signing_key\"." eval $static::genarate_hmac_token # log -noname local0.debug "HMAC: HTTP_RESPONSE: Storing the computed HMAC token \"$hmac_token\" into the HMAC cache array." set static::hmac_cache($session_cookie:$hmac_input) $hmac_token # log -noname local0.debug "HMAC: HTTP_RESPONSE: Inserting computed HMAC cookie \"$static::hmac_cookie\" = \"$static::hmac_cache($session_cookie:$hmac_input)\" to the response." HTTP::header insert "Set-Cookie" "$static::hmac_cookie=$hmac_token; HttpOnly; Secure; Path=/" } } } Tested this on version: 12.0
Handle Over 100 Gbps With a Single BIG-IP Virtual Edition
Cloud computing is an inescapable term. The general public knows that their cat pictures, videos, and memes go to the cloud somehow. Companies and application developers go cloud-first, or often cloud-only, when they develop a new service. They think of the cloud as the group of resources and APIs offered by cloud providers. Large enterprises and service providers have a bifurcated view of cloud computing: they see a public cloud and a private cloud. A service provider might mandate that any new software or services must run within their orchestrated virtualization or bare metal environment, actively discouraging or simply disallowing new vendor-specific hardware purchases. This has the effect of causing traditional networking vendors to improve the efficiency of their software offerings, and take advantage of the available server hardware in an opportunistic fashion. Behold the early fruits of our labor. 100+ Gbps L4 From a Single VE?! We introduced the high performance license option for BIG-IP Virtual Edition with BIG-IP v13.0.0 HF1. This means that rather than having a throughput capped license, you can purchase a license that is only restricted by the maximum number of vCPUs that can be assigned. This license allows you to optimize the utilization of the underlying hypervisor hardware. BIG-IP v13.0.0 HF1 introduced a limit of 16 vCPUs per VE. BIG-IP v13.1.0.1 raised the maximum to 24 vCPUs. Given that this is a non-trivial amount of computing capacity for a single VM, we decided to see what kind of performance can be obtained when you use the largest VE license on recent hypervisor hardware. The result is decidedly awesome. I want to show you precisely how we achieved 100+ Gbps in a single VE. Test Harness Overview The hypervisor for this test was KVM running on an enterprise grade rack mount server. The server had two sockets, and each socket had an Intel processor with 24 physical cores / 48 hyperthreads. We exposed 3 x 40 Gbps interfaces from Intel XL710 NICs to the guest via SR-IOV. Each NIC utilized a PCI-E 3.0x8 slot. There was no over-subscription of hypervisor resources. Support for "huge pages", or memory pages much larger than 4 KB, was enabled on the hypervisor. It is not a tuning requirement, but it proved beneficial on our hypervisor. See: Ubuntu community - using hugepages. The VE was configured to run BIG-IP v13.1.0.1 with 24 vCPU and 48 GB of RAM in an "unpacked" configuration. This means that we dedicated a single vCPU per physical core. This was done to prevent hyperthread contention within each physical core. Additionally, all of the physical cores were on the same socket. This eliminated any inter-socket communication latency and bus limitations. The VE was provisioned with LTM only, and all test traffic utilized a single FastL4 virtual server. There were two logical VLANs. The 3 x 40 Gbps interfaces were logically trunked. The VE only has two L3 presences, one for the client network and one for the server network. In direct terms, this is a single application deployment achieving 100+ Gbps with a single BIG-IP Virtual Edition. Result The network load was generated using Ixia IxLoad and Ixia hardware appliances. The traffic was legitimate HTTP traffic with full TCP handshakes and graceful TCP teardowns. A single 512 kB HTTP transaction was completed for every TCP connection. We describe this scenario as one request per connection, or 1-RPC. It's worth noting that 1-RPC is the worst case for an ADC. For every Ixia client connection: Three-way TCP handshake HTTP request (less than 200 B) delivered to Ixia servers HTTP response (512 kB, multiple packets) from Ixia servers Three-way TCP termination The following plot shows the L7 throughput in Gbps during the "sustained" period of a test, meaning that the device is under constant load and new connections are being established immediately after a previous connection is satisfied. If you work in the network testing world, you'll probably note how stupendously smooth this graph is... The average for the sustained period ends up around 108 Gbps. Note that, as hardware continues to improve, this performance will only go up. Considerations Technical forums love car analogies and initialisms, like "your mileage may vary" as YMMV. This caveat applies to the result described above. You should consider these factors when planning a high performance VE deployment: Physical hardware layout of the hypervisor - Non-uniform memory access (NUMA) architectures are ubiquitous in today's high density servers. In very simple terms, the implication of NUMA architectures is that the physical locality of a computational core matters. All of the work for a given task should be confined to a single NUMA node when possible. The slot placement of physical NICs can be a factor as well. Your server vendor can guide you in understanding the physical layout of your hardware. Example: you have a hypervisor with two sockets, and each socket has 20c / 40t. You have 160 Gbps of connectivity to the hypervisor. The recommended deployment would be two 20 vCPU high performance VE guests, one per socket, with each receiving 80 Gbps of connectivity. Spanning a 24 vCPU guest across both sockets would result in more CPU load per unit of work done, as the guest would be communicating between both sockets rather than within a single socket. Driver support - The number of drivers that BIG-IP supports for SR-IOV access is growing. See: https://support.f5.com/csp/article/K17204. Do note that we also have driver support for VMXNET3, virtio, and OvS-DPDK via virtio. Experimentation and an understanding of the available hypervisor configurations will allow you to select the proper deployment. Know the workload - This result was generated with a pure L4 configuration using simple load balancing, and no L5-L7 inspection or policy enforcement. The TMM CPU utilization was at maximum during this test. Additional inspection and manipulation of network traffic requires more CPU cycles per unit of work.
