Announcing Security Insights
A small but mighty update to the DevCentral Article pages... Security Insights. Brought to you by the estimable engineers and analysts from our own F5 SIRT and F5 Labs teams the articles in Security Insights bring security research, threat analysis, and years of experience to bear on the cyber security industry into one destination on your DevCentral community. To begin, I have moved all of the This Week In Security #TWIS articles out of Technical Articles and into this new location. In the coming days I will also move select other pieces and going forward you can expect to see the Labs and SIRT team members share their content directly in this new location. Currently we expect about five or so new pieces each month.Click the bell icon at the top right to follow (and be notified) of any new posts. Any questions or feedback - you know where I work. LiefZimmerman
How to test APM access policy locally?
Very new to BIG-IP. I have downloaded the BIG-IP VE and imported it into VirtualBox. Followed the basic configuration mentioned here https://f5.bravais.com/s/vf4XckakhbTbCRUwa2sb I created an APM access policy to use AD. But is there any way to test it? Some internal URL which simulates the real world behavior? As I am on VBox, any virtual server I create, is directly accessible from local. How do I get BIG-IP into the picture?Implementing SSL Orchestrator - Validation & Troubleshooting
Introduction This article is part of a series on implementing BIG-IP SSL Orchestrator. It includes high availability and central management with BIG-IQ. Implementing SSL/TLS Decryption is not a trivial task. There are many factors to keep in mind and account for, from the network topology and insertion point, to SSL/TLS keyrings, certificates, ciphersuites and on and on. This article focuses on verification that the solution is working. This article is divided into the following high level sections: How to check if content is being decrypted How to check if content is being blocked How to check if content is being bypassed Logging and Troubleshooting Please forgive me for using SSL and TLS interchangeably in this article. Software versions used in this article: BIG-IP Version: 14.1.2 SSL Orchestrator Version: 5.5 BIG-IQ Version: 7.0.1 How to check if content is being decrypted Go to a computer that connects to the internet through SSL Orchestrator.Go to an encrypted website like www.cisco.com.The page should load without error.Click the padlock icon in the Address Bar. Notice that the connection is secure and the certificate is valid.Click the Certificate to view more details.The certificate was issued to www.cisco.com but has been issued by subrsa.f5labs.com.The connection to this web site was decrypted and re-encrypted by SSL Orchestrator. Switch to the Palo Alto UI and go to Monitor > Traffic.It should look like this. Click the link on the left in the red box. This drills into more details. Notice the IP address of 72.163.10.10 and port 80 (not port 443).A quick lookup of that IP shows it belongs to Cisco. How to check if content is being blocked Now go to the website www.eicar.org.This site hosts a harmless test malware file so you can see if your security measures are effective.Click Download Anti Malware Testfile on the right. Click the Download link on the left. Scroll down on the next page.You should see a section with 8 different downloads. There are 4 different file types.The first group uses HTTP.The second group uses HTTPS.Click any of the HTTPS links to verify it is detected and blocked. You should get a block page like the following. Notice the URL indicates an attempt to download one of the compressed .zip files. How to check if content is being bypassed Next try a Banking website and make sure it is bypassed.On the client open a browser and go to www.chase.com.Click the Padlock icon in the address bar and it should look like this. Click the highlighted section for more details about this certificate. This certificate was issued by Entrust CA.The SSL Orchestrator is not decrypting this connection.We can check the Palo Alto logs to make sure it doesn’t have a record of this transaction. The image below shows ping traffic generated by SSL Orchestrator to test the health of the L2 Service.Notice there is no web-browsing or port 80 traffic. Logging and Troubleshooting To enable logging: -Connect to the WebUI -Go to SSL Orchestrator and click on Logs -Enable the required logging levels Note: By default, logs are stored locally on the BIG-IP. Below is a reasonably-ordered list of troubleshooting steps. ·If the SSL Orchestrator deployment process fails, review the ensuing error message. It would be impossible to list here all of the possible error messages and their meanings, but often enough the messages will reveal the issue. ·Re-review the lab steps for any missing or misconfigured settings. ·Enable debug logging in the SSL Orchestrator configuration. Tail the APM log from a BIG-IP command line or from the logs page in the management UI. Debug logging will very often reveal important issues. Specifically, it will indicate traffic classification matches, mismatches or deployment issues. tail –f /var/log/apm tail -f /var/log/restnoded/restnoded.log tail -f /var/log/restjavad.0.log ·If the SSL Orchestrator deployment process succeeds, but traffic isn’t flowing through the environment made evident by lack of access to remote sites from the client: o Never enable debug logging on the Per-Request Policy logging option in a production environment. If needed, be sure to turn it back off after capture logs. The PRP logging is extremely verbose and WILL affect system performance. o /var/log/apm is used to log data plane traffic flow o /var/log/restnoded/restnoded.log is used to log control plane (when SSLO configurations are deployed) o /var/log/restjavad is similarly used to log control plane restjava functions o It’s also sometimes useful to tail /var/log/ltm, which is where SSL and generic data plane issues will show up. oEnsure that the client is properly configured to either default route to the ingress VLAN and self-IP of the BIG-IP for transparent proxy access or has the correct browser proxy settings defined for explicit proxy access. oEnsure that traffic is flowing to the BIG-IP from the client with a tcpdump capture at the ingress interface. oReview the LTM configuration created by the SSL Orchestrator. Specifically, look at the pools and respective monitors for any failures. oIsolate service chain services. If at least one service chain has been created, and debug logging indicates that traffic is matching this chain, remove all but one service from that chain and test. Add one service back at a time until traffic flow stops. If a single added service breaks traffic flow, this service will typically be the culprit. oIf a broken service is identified, insert probes to verify inbound and outbound traffic flow. Inline services will have a source (S) VLAN and destination (D) VLAN, and ICAP and receive only services will each have a single source VLAN. Insert a tcpdump capture at each VLAN in order to determine if traffic is getting to the device, and if traffic is leaving the device through its outbound interface. o How to insert probes. The services VLANs are wrapped in application services so must be addressed accordingly in the tcpdump. Note that each inline service will create two separate VLANs – one for inbound and one for outbound, so it becomes easy to surgically insert captures at specific points in the flow (ie. to the service, coming from the service). So a tcpdump capture of a service named “FEYE” might look like this: tcpdump -lnni ssloS_FEYE.app/ssloS_FEYE Where “ssloS_FEYE.app” is the application service container, which contains the “ssloS_FEYE” VLAN. oBy default the ‘all traffic’ rule doesn’t attach a service chain, so it can be as simple as removing service chains from all of the security rules. If traffic doesn’t flow with no service chains attached anywhere, then the problem is in SSLO proper, likely a routing or decryption issue. If the traffic only fails with a service chain applied, this is when it’s useful to isolate the services. If a broken service is identified, insert tcpdump probes as described above. oIf traffic is flowing through all of the security devices, insert a tcpdump probe at the egress point to verify traffic is leaving the BIG-IP to the gateway router. oIf traffic is flowing to the gateway router, perform a more extensive packet analysis to determine if SSL if failing between the BIG-IP egress point and the remote server. tcpdump –i 0.0:nnn –nn –Xs0 –vv –w <file.pcap> <any additional filters> Then either export this capture to WireShark or send to ssldump: ssldump –nr <file.pcap> -H –S crypto > text-file.txt oIf the WireShark or ssldump analysis verifies an SSL issue: oPlug the site’s address into the SSLLabs.com server test site at: www.ssllabs.com/ssltest/ This report will indicate any specific SSL requirements that this site has. oVerify that the SSL Orchestrator server SSL profiles (two of them) have the correct cipher string to match the requirements of this site. To do that, perform the following command at the BIG-IP command line: tmm --clientciphers ‘CIPHER STRING AS DISPLAYED IN SERVER SSL PROFILES’ Further SSL/TLS issues are beyond the depth of this guide. Seek assistance. • If all else fails, seek assistance. Summary In this article you learned how to verify that SSL Orchestrator is decrypting SSL and passing it to an inline security device. You can verify that SSL is decrypted simply by viewing web site certificates. You can also use logging or reporting capabilities of an inline security device to verify that SSL is decrypted. You learned how to test whether the inline security device was blocking malicious content. You also learned how to verify that a policy to bypass SSL decryption is working. Then you learned how to enable logging and perform troubleshooting steps. Next Steps Click Next to proceed to the next article in the series.
Throughput per virtual server
Hi! Not quite sure if my question fits this forum, but I'm trying anyway :) I'm trying to measure performance through our LTMs. Is there any way to get throughput numbers per virtual server? The default graph in the dashboard gives a good indication of the total traffic, but it would be nice to be able to drill down to a specific vip. Is it perhaps possible to query with snmp? /Niklas
Performance of an external datagroup versus a local datagroup?
Hi, Is there any expected difference in performance of a ~10k list of networks/hosts if the datagroup is defined in an external file versus locally, directly in the bigip.conf? This is for 9.4.8 if it matters. I'm guessing that the lookup will be handled the same once the datagroup is loaded in memory regardless if whether it's coming from an external file versus the bigip.conf. But I figure I'd ask. I'll also try testing to see if I notice any difference in the timings for the two different datagroups. Thanks, Aaron
iRule test & analysis tool
I'm looking for an iRule analyzer that would allow testing simple traffic scenarios independent of configuring a BIG-IP device. The user would specify values for system variables such as [HTTP:uri] and [HTTP:cookie] and the analyzer would output rewrite url and name of pool routed to , or redirect url , etc. Also would be nice if the tool would also analyzed the iRule for optimum code and performance load testingBIG-IP : simplest way to ensure request is made on new tcp connection
F5 BIG-IP Virtual Edition v11.4.1 (Build 635.0) LTM on ESXi When testing from client browser or shell ( curl ) , what is simplest way to ensure a request initiates a new tcp connection with the virtual-server ( is not sent on any pre-existing connection ) ? I want to ensure my tests are on a fresh connection and irule stack processing starts fresh.
Bare Metal Blog: Throughput Sometimes Has Meaning
#BareMetalBlog Knowing what to test is half the battle. Knowing how it was tested the other. Knowing what that means is the third. That’s some testing, real clear numbers. In most countries, top speed is no longer the thing that auto manufacturers want to talk about. Top speed is great if you need it, but for the vast bulk of us, we’ll never need it. Since the flow of traffic dictates that too much speed is hazardous on the vast bulk of roads, automobile manufacturers have correctly moved the conversation to other things – cup holders (did you know there is a magic number of them for female purchasers? Did you know people actually debate not the existence of such a number, but what it is?), USB/bluetooth connectivity, backup cameras, etc. Safety and convenience features have supplanted top speed as the things to discuss. The same is true of networking gear. While I was at Network Computing, focus was shifting from “speeds and feeds” as the industry called it, to overall performance in a real enterprise environment. Not only was it getting increasingly difficult and expensive to push ever-larger switches until they could no longer handle the throughput, enterprise IT staff was more interested in what the capabilities of the box were than how fast it could go. Capabilities is a vague term that I used on purpose. The definition is a moving target across both time and market, with a far different set of criteria for, say, an ADC versus a WAP. There are times, however, where you really do want to know about the straight-up throughput, even if you know it is the equivalent of a professional driver on a closed course, and your network will never see the level of performance that is claimed for the device. There are actually several cases where you will want to know about the maximum performance of an ADC, using the tools I pay the most attention to at the moment as an example. WAN optimization is a good one. In WANOpt, the goal is to shrink the amount of data being transferred between two dedicated points to try and maximize the amount of throughput. When “maximize the amount of throughput” is in the description, speeds and feeds matter. WANOpt is a pretty interesting example too, because there’s more than just “how much data did I send over the wire in that fifteen minute window”. It’s more complex than that (isn’t it always?). The best testing I’ve seen for WANOpt starts with “how many bytes were sent by the originating machine”, then measures that the same number of bytes were received by the WANOpt device, then measures how much is going out the Internet port of the WANOpt device – to measure compression levels and bandwidth usage – then measures the number of bytes the receiving machine at the remote location receives to make sure it matches the originating machine. So even though I say “speeds and feeds matter”, there is a caveat. You want to measure latency introduced with compression and dedupe, and possibly with encryption since WANOpt is almost always over the public Internet these days, throughput, and bandwidth usage. All technically “speeds and feeds” numbers, but taken together giving you an overall picture of what good the WANOpt device is doing. There are scenarios where the “good” is astounding. I’ve seen the numbers that range as high as 95x the performance. If you’re sending a ton of data over WANOpt connections, even 4x or 5x is a huge savings in connection upgrades, anything higher than that is astounding. This is an (older) diagram of WAN Optimization I’ve marked up to show where the testing took place, because sometimes a picture is indeed worth a thousand words. And yeah, I used F5 gear for the example image… That really should not surprise you . So basically, you count the bytes the server sends, the bytes the WANOpt device sends (which will be less for 99.99% of loads if compression and de-dupe are used), and the total number of bytes received by the target server. Then you know what percentage improvement you got out of the WANOpt device (by comparing server out bytes to WANOpt out bytes), that the WANOpt devices functioned as expected (server received bytes == server sent bytes), and what the overall throughput improvement was (server received bytes/time to transfer). There are other scenarios where simple speeds and feeds matter, but less of them than their used to be, and the trend is continuing. When a device designed to improve application traffic is introduced, there are certainly few. The ability to handle a gazillion connections per second I’ve mentioned before is a good guardian against DDoS attacks, but what those connections can do is a different question. Lots of devices in many networking market spaces show little or even no latency introduction on their glossy sales hand-outs, but make those devices do the job they’re purchased for and see what the latency numbers look like. It can be ugly, or you could be pleasantly surprised, but you need to know. Because you’re not going to use it in a pristine lab with perfect conditions, you’re going to slap it into a network where all sorts of things are happening and it is expected to carry its load. So again, I’ll wrap with acknowledgement that you all are smart puppies and know where speeds and feeds matter, make sure you have realistic performance numbers for those cases too. Technorati Tags: Testing,Application Delivery Controller,WAN Optimization,throughput,latency,compression,deduplication,Bare Metal Blog,F5 Networks,Don MacVittie The Whole Bare Metal Blog series: Bare Metal Blog: Introduction to FPGAs | F5 DevCentral Bare Metal Blog: Testing for Numbers or Performance? | F5 ... Bare Metal Blog: Test for reality. | F5 DevCentral Bare Metal Blog: FPGAs The Benefits and Risks | F5 DevCentral Bare Metal Blog: FPGAs: Reaping the Benefits | F5 DevCentral Bare Metal Blog: Introduction | F5 DevCentralBare Metal Blog: Test for reality.
#BareMetalBlog #F5 Test results provided by vendors and “independent testing labs” often test for things that just don’t matter in the datacenter. Know what you’re getting. When working in medicine, you don’t glance over a patient, then ask them “so how do you feel when you’re at your best?” You ask them what is wrong, then run a ton of tests – even if the patient thinks they know what is wrong – then let the evidence determine the best course of treatment. Sadly, when determining the best tools for your IT staff to use, we rarely follow the same course. We invite a salesperson in, ask them “so, what do you do?”, and let them tell us with their snippets of reality or almost-reality why their product rocks the known world. Depending upon the salesperson and the company, their personal moral code or corporate standards could limit them to not bringing up the weak points of their products to outright lying about its capabilities. “But Don!”, you say, “you’re being a bit extreme, aren’t you?” Not in my experience I am not. From being an enterprise architect to doing comparative reviews, I have seen it all. Vendor culture seems to infiltrate how employees interact with the world outside their HQ – or more likely (though I’ve never seen any research on it), vendors tend to hire to fit their culture, which ranges from straight-up truth about everything to wild claims that fall apart the first time the device is put into an actual production-level network. The most common form of disinformation that is out there is to set up tests so they simply show the device operating at peak efficiency. This is viewed as almost normal by most vendors – why would you showcase your product in less than its best light? and as a necessary evil by most of the few who don’t have that view – every other vendor in the space is using this particular test metric, we’d better too or we’ll look bad. Historically, in network gear, nearly empty communications streams have been the standard for high connection rates, very large window sizes the standard for manipulating throughput rates. While there are many other games vendors in the space play to look better than they are, it is easy to claim you handle X million connections per second if those connections aren’t actually doing anything. It is also easier to claim you handle a throughput of Y Mbps if you set the window size larger than reality would ever see it. Problem with this kind of testing is that it seeps into the blood, after a while, those test results start to be sold as actual… And then customers put the device into their network, and needless to say, they see nothing of the kind. You would be surprised the number of times when we were testing for Network Computing that a vendor downright failed to operate as expected when put into a live network, let alone met the numbers the vendor was telling customers about performance. One of the reasons I came to F5 way back when was that they did not play these games. They were willing to make the marketing match the product and put a new item on the roadmap for things that weren’t as good as they wanted. We’re still out there today helping IT staff understand testing, and what testing will show relevant numbers to the real world. By way of example, there is the Testing Configuration Guide on F5 DevCentral. As Application Delivery Controllers have matured and solidified, there has been much change in how they approach network traffic. This has created an area we are now starting to talk more about, which is the validity of throughput testing in regards to ADCs in general. The thing is, we’ve progressed to the point that simply “we can handle X Mbps!” is no longer a valid indication of the workloads an ADC will be required to handle in production scenarios. The real measure for application throughput that matters is requests per second. Vendors generally avoid this kind of testing, because response is also limited by the capacity of the server doing the actual responding, so it is easy to get artificially low numbers. At this point in the evolution of the network, we have reached the reality of that piece of utility computing. Your network should be like electricity. You should be able to expect that it will be on, and that you will have enough throughput to handle incoming load. Mbps is like measuring amperage… When you don’t have enough, you’ll know it, but you should, generally speaking, have enough. It is time to focus more on what uses you put that bandwidth to, and how to conserve it. Switching to LED bulbs is like putting in an ADC that is provably able to improve app performance. LEDs use less electricity, the ADC reduces bandwidth usage… Except that throughput or packets per second isn’t measuring actual improvements of bandwidth usage. It’s more akin to turning off your lights after installing LED bulbs, and then saying “lookie how much electricity those new bulbs saved!” Seriously, do you care if your ADC can delivery 20 million Megabits per second in throughput, or that it allows your servers to respond to requests in a timely manner? Seriously, the purpose of an Application Delivery Controller is to facilitate and accelerate the delivery of applications, which means responses to requests. If you’re implementing WAN Optimization functionality, throughput is still a very valid test. If you’re looking at the Application Delivery portion of the ADC though, it really has no basis in reality, because requests and responses are messy, not “as large a string of ones as I can cram through here”. From an application perspective – particularly from a web application perspective – there is a lot of “here’s a ton of HTML, hold on, sending images, wait, I have a video lookup…” Mbps or MBps just doesn’t measure the variety of most web applications. But users are going to feel requests per second just as much as testing will show positive or negative impacts. To cover the problem of application servers actually having a large impact on testing, do what you do with everything else in your environment, control for change. When evaluating ADCs, simply use the same application infrastructure and change only the ADC out. Then you are testing apples-to-apples, and the relative values of those test results will give you a gauge for how well a given ADC will perform in your environment. In short, of course the ADC looks better if it isn’t doing anything. But ADCs do a ton in production networks, and that needs to be reflected in testing. If you’re fortunate enough to have time and equipment, get a test scheduled with your prospective vendor, and make sure that test is based upon the usage your actual network will expose the device to. If you are not, then identify your test scenarios to stress what’s most important to you, and insist that your vendor give you test results in those scenarios. In the end, you know your network far better than they ever will, and you know they’re at least not telling you the whole story, make sure you can get it. Needless to say, this is a segue into the next segment of our #BareMetalBlog series, but first I’m going to finish educating myself about our use of FPGAs and finish that segment up.
