Demystifying iControl REST Part 7 - Understanding Transactions
iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5. Several articles on basic usage have been written about the rest interface so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. A few months ago, a question in Q&A from community member spirrello asking how to update a tcp profile on a virtual. He was using bigsuds, the python wrapper for the soap interface. For the rest interface on this particular object, this is easy; just use the put method and supply the payload mapping the updated profile. But for soap, this requires a transaction. There are some changes to BIG-IP via the rest interface, however, like updating an ssl cert or key, that likewise will require a transaction to accomplish. In this article, I’ll show you how to use transactions with the rest interface. The Fine Print From the iControl REST user guide, the life cycle of a transaction progresses through three phases: Creation - This phase occurs when the transaction is created using a POST command. Modification - This phase occurs when commands are added to the transaction, or changes are made to the sequence of commands in the transaction. Commit - This phase occurs when iControl REST runs the transaction. To create a transaction, post to /tm/transaction POST https://192.168.25.42/mgmt/tm/transaction {} Response: { "transId":1389812351, "state":"STARTED", "timeoutSeconds":30, "kind":"tm:transactionstate", "selfLink":"https://localhost/mgmt/tm/transaction/1389812351?ver=11.5.0" } Note the transId, the state, and the timeoutSeconds. You'll need the transId to add or re-sequence commands within the transaction, and the transaction will expire after 30 seconds if no commands are added. You can list all transactions, or the details of a specific transaction with a get request. GET https://192.168.25.42/mgmt/tm/transaction GET https://192.168.25.42/mgmt/tm/transaction/transId To add a command to the transaction, you use the normal method uris, but include the X-F5-REST-Coordination-Id header. This example creates a pool with a single member. POST https://192.168.25.42/mgmt/tm/ltm/pool X-F5-REST-Coordination-Id:1389812351 { "name":"tcb-xact-pool", "members": [ {"name":"192.168.25.32:80","description":"First pool for transactions"} ] } Not a great example because there is no need for a transaction here, but we'll roll with it! There are several other option methods for interrogating the transaction itself, see the user guide for details. Now we can commit the transaction. To do that, you reference the transaction id in the URI, remove the X-F5-REST-Coordination-Id header and use the patch method with payload key/value state: VALIDATING . PATCH https://localhost/mgmt/tm/transaction/1389812351 { "state":"VALIDATING" } That's all there is to it! Now that you've seen the nitty gritty details, let's take a look at some code samples. Roll Your Own In this example, I am needing to update and ssl key and certificate. If you try to update the cert or the key, it will complain that they do not match, so you need to update both at the same time. Assuming you are writing all your code from scratch, this is all it takes in python. Note on line 21 I post with an empty payload, and then on line 23, I add the header with the transaction id. I make my modifications and then in line 31, I remove the header, and finally on line 32, I patch to the transaction id with the appropriate payload. import json import requests btx = requests.session() btx.auth = (f5_user, f5_password) btx.verify = False btx.headers.update({'Content-Type':'application/json'}) urlb = 'https://{0}/mgmt/tm'.format(f5_host) domain = 'mydomain.local_sslobj' chain = 'mychain_sslobj try: key = btx.get('{0}/sys/file/ssl-key/~Common~{1}'.format(urlb, domain)) cert = btx.get('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, domain)) chain = btx.get('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, 'chain')) if (key.status_code == 200) and (cert.status_code == 200) and (chain.status_code == 200): # use a transaction txid = btx.post('{0}/transaction'.format(urlb), json.dumps({})).json()['transId'] # set the X-F5-REST-Coordination-Id header with the transaction id btx.headers.update({'X-F5-REST-Coordination-Id': txid}) # make modifications modkey = btx.put('{0}/sys/file/ssl-key/~Common~{1}'.format(urlb, domain), json.dumps(keyparams)) modcert = btx.put('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, domain), json.dumps(certparams)) modchain = btx.put('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, 'le-chain'), json.dumps(chainparams)) # remove header and patch to commit the transaction del btx.headers['X-F5-REST-Coordination-Id'] cresult = btx.patch('{0}/transaction/{1}'.format(urlb, txid), json.dumps({'state':'VALIDATING'})).json() A Little Help from a Friend The f5-common-python library was released a few months ago to relieve you of a lot of the busy work with building requests. This is great, especially for transactions. To simplify the above code just to the transaction steps, consider: # use a transaction txid = btx.post('{0}/transaction'.format(urlb), json.dumps({})).json()['transId'] # set the X-F5-REST-Coordination-Id header with the transaction id btx.headers.update({'X-F5-REST-Coordination-Id': txid}) # do stuff here # remove header and patch to commit the transaction del btx.headers['X-F5-REST-Coordination-Id'] cresult = btx.patch('{0}/transaction/{1}'.format(urlb, txid), json.dumps({'state':'VALIDATING'})).json() With the library, it's simplified to: tx = b.tm.transactions.transaction with TransactionContextManager(tx) as api: # do stuff here api.do_stuff Yep, it's that simple. So if you haven't checked out the f5-common-python library, I highly suggest you do! I'll be writing about how to get started using it next week, and perhaps a follow up on how to contribute to it as well, so stay tuned!
Connection list via iControlREST API
Problem this snippet solves: This python script will retrieve a filtered list of active connections via the iControl REST API. It supports both the default connection list as well as the detailed list you get by specifying 'all-properties'. It has options to generate output on STDOUT as displayed by TMSH (raw), as a JSON formatted dictionary, or written to an Excel spreadsheet. Currently tested in python 2.7 and python 3.5 against BIGIP 11.6.0. How to use this snippet: $ ./conn-list.py -h usage: conn-list.py [-h] [-a] [-x FILE | -j | -r] -f P=V host positional arguments: host Host name to connect. Specified as [<username>@]<hostname> optional arguments: -h, --help show this help message and exit -a, --all-properties Get detailed connection information -f P=V, --filter P=V You must have at least one filter argument, but may have multiple. Output Type: -x FILE, --xlout FILE Excel workbook to be created. -j, --json JSON formatted output to STDOUT -r, --raw RAW ouptut from API request (default) P=V: P = Connection Property (below), V = Value to match. Multiple -f options are joined as logical AND. age Specifies the age, in seconds, of a connection cs-client-addr Specifies the clientside remote address of the active connections cs-client-port Specifies the clientside remote port of the active connections cs-server-addr Specifies the clientside local address of the active connections cs-server-port Specifies the clientside local port of the active connections protocol Specifies the protocol used for specified connections (for example: tcp, udp) ss-client-addr Specifies the serverside local address of the active connections ss-client-port Specifies the serverside local port of the active connections ss-server-addr Specifies the serverside remote address of the active connections ss-server-port Specifies the serverside remote port of the active connections type Specifies the connnection type used for specified connections (for example: any, mirror, self) $ ./conn-list.py -r -f cs-server-port=80 admin@192.0.2.45 Password: Sys::Connections 192.0.2.31:55345 192.0.2.20:80 192.0.2.31:55345 198.51.100.66:80 tcp 6 (tmm: 1) none Total records returned: 1 $ ./conn-list.py -j -f cs-server-port=80 admin@192.0.62.45 Password: [{"acceleration": "none", "cs-server": "192.0.2.20:80", "protocol": "tcp", "cs-client": "192.0.2.31:55613", "idle": 1, "ss-server": "198.51.100.66:80", "tmm": 1, "ss-client": "192.0.2.31:55613"}] Code : #!/usr/bin/env python """ This script will use F5's iControl REST API to collect current connection data. It enforces the use a at least one filter criteria. This has only been tested on 11.6.0 so far. Should be extendable to 11.5.x and 12.x by adding to or duplicating Record Definitions and RE sections. """ import json import sys import getpass import argparse import re try: # Py3 from urllib.parse import urlparse, parse_qs except ImportError: # Py2 from urlparse import urlparse, parse_qs import requests from openpyxl import Workbook RE_LAST = re.compile('(\w+)$') RE_LAST_TWO = re.compile('([\w/:\.]+)\s+([\w/:\.]+)$') DETAILED_DEF = { '11.6.0': {'rec-sep': '---', # actually second line in record. 'fields': {'Slot': {'re': RE_LAST, 'id': ['slot']}, 'TMM': {'re': RE_LAST, 'id': ['tmm']}, 'Acceleration': {'re': RE_LAST, 'id': ['acceleration']}, 'Protocol': {'re': RE_LAST, 'id': ['protocol']}, 'Idle Time ': {'re': RE_LAST, 'id': ['idle']}, 'Idle Timeout': {'re': RE_LAST, 'id': ['idle_timeout']}, 'Lasthop': {'re': RE_LAST_TWO, 'id': ['lasthop-vlan', 'lasthop-mac']}, 'Client Addr': {'re': RE_LAST_TWO, 'id': ['cs-client', 'ss-client']}, 'Server Addr': {'re': RE_LAST_TWO, 'id': ['cs-server', 'ss-server']}, 'Bits In': {'re': RE_LAST_TWO, 'id': ['cs-bits-in', 'ss-bits-in']}, 'Bits Out': {'re': RE_LAST_TWO, 'id': ['cs-bits-out', 'ss-bits-out']}, 'Packets In': {'re': RE_LAST_TWO, 'id': ['cs-packets-in', 'ss-packets-in']}, 'Packets Out': {'re': RE_LAST_TWO, 'id': ['cs-packets-out', 'ss-packets-out']}, } } } SHRT_RE_AP = \ re.compile( '([\w:\.]+)\s+([\w:\.]+)\s+([\w:\.]+)\s+([\w:\.]+)\s+(\w+)\s+(\d+)\s+\(tmm: (\d+)\)\s+(\w+)') SHRT_RE_CH = \ re.compile( '([\w:\.]+)\s+([\w:\.]+)\s+([\w:\.]+)\s+([\w:\.]+)\s+(\w+)\s+(\d+)\s+\(slot/tmm: (\d+)/(\d+)\)\s+(\w+)') SHORT_DEF = { '11.6.0': {'chs-indc': 'slot', 'fields': {'appliance': {'re': SHRT_RE_AP, 'id': ['cs-client', 'cs-server', 'ss-client', 'ss-server', 'protocol', 'idle', 'tmm', 'acceleration']}, 'chassis': {'re': SHRT_RE_CH, 'id': ['cs-client', 'cs-server', 'ss-client', 'ss-server', 'protocol', 'idle', 'slot', 'tmm', 'acceleration']}, } } } requests.packages.urllib3.disable_warnings() try: dict.iteritems except AttributeError: # Py3 def itervalues(d): return iter(d.values()) def iteritems(d): return iter(d.items()) else: # Py2 def itervalues(d): return d.itervalues() def iteritems(d): return d.iteritems() def main(): parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument('-a', '--all-properties', dest='detail', action='store_true', help='Get detailed connection information') outype = parser.add_argument_group("Output Type") outputs = outype.add_mutually_exclusive_group() outputs.add_argument('-x', '--xlout', action='store', metavar="FILE", help='Excel workbook to be created.') outputs.add_argument('-j', '--json', action='store_true', help='JSON formatted output to STDOUT') outputs.add_argument('-r', '--raw', action='store_true', help='RAW ouptut from API request (default)') parser.add_argument('-f', '--filter', required=True, action='append', metavar='P=V', help='You must have at least one filter argument, but may have multiple.') parser.add_argument('host', help='Host name to connect. Specified as [@]') parser.epilog = """ P=V: P = Connection Property (below), V = Value to match. Multiple -f options are joined as logical AND. age Specifies the age, in seconds, of a connection cs-client-addr Specifies the clientside remote address of the active connections cs-client-port Specifies the clientside remote port of the active connections cs-server-addr Specifies the clientside local address of the active connections cs-server-port Specifies the clientside local port of the active connections protocol Specifies the protocol used for specified connections (for example: tcp, udp) ss-client-addr Specifies the serverside local address of the active connections ss-client-port Specifies the serverside local port of the active connections ss-server-addr Specifies the serverside remote address of the active connections ss-server-port Specifies the serverside remote port of the active connections type Specifies the connnection type used for specified connections (for example: any, mirror, self) """ args = parser.parse_args() (username, unused, host) = args.host.rpartition('@') if not username: username = raw_input('Username: ') password = getpass.getpass('Password: ') filter_prop = '+'.join(args.filter) filter_prop = filter_prop.replace('=', '+') (ctext, ver) = get_conn_list(host, username, password, filter_prop, args.detail) if not ctext: print('No connection list was returned from {}.'.format(host)) quit() if not args.xlout and not args.json: raw_output(ctext) else: if args.detail: conns, patterns = process_detailed_conns(ctext, ver) else: conns, patterns = process_short_conns(ctext, ver) if args.xlout: excel_output(conns, args.xlout, patterns) elif args.json: json_output(conns) # noinspection PyBroadException def get_conn_list(host, uname, pw, options, detail): conntext = [] version = '' b = requests.session() b.auth = (uname, pw) b.verify = False b.headers.update({'Content-Type': 'application/json'}) b_url = 'https://{}/mgmt/tm'.format(host) if detail: options += "+all-properties" try: resp = b.get(b_url + '/sys/connection/?options=' + options, timeout=6.05) if resp.status_code == requests.codes.ok: j = json.loads(resp.text) version = parse_qs(urlparse(j['selfLink']).query)['ver'][0] conntext = j['apiRawValues']['apiAnonymous'].splitlines() else: sys.stderr.write( 'Error: {} status returned from: {}\n {}\n'.format(resp.status_code, host, resp.reason)) except: sys.stderr.write('Error: Could not get data from {}: {}\n'.format(host, sys.exc_info()[0])) return conntext, version def process_detailed_conns(conntext, ver): connlist = [] patterns = {} detailed_record_length = 0 more_than_one_record = False for ndx, val in enumerate(conntext): if val.startswith(DETAILED_DEF[ver]['rec-sep']): if detailed_record_length == 0: detailed_record_length = ndx else: detailed_record_length = ndx - detailed_record_length more_than_one_record = True break else: for p in DETAILED_DEF[ver]['fields']: if val.lstrip().startswith(p): patterns[ndx - 1] = DETAILED_DEF[ver]['fields'][p] break if not more_than_one_record: # only one record returned detailed_record_length = ndx - 1 ndx = 0 while ndx < len(conntext) - 1: if ndx % detailed_record_length == 1: obj = {} for offset, data_def in iteritems(patterns): match = data_def['re'].search(conntext[ndx + offset]) for ndx2, col_heading in enumerate(data_def['id']): grouping = match.group(ndx2 + 1) obj[col_heading] = int(grouping) if grouping.isnumeric() else grouping connlist.append(obj) ndx += detailed_record_length else: ndx += 1 return connlist, patterns def process_short_conns(ctext, ver): connlist = [] patterns = {} if len(ctext) > 2: if SHORT_DEF[ver]['chs-indc'] in ctext[1]: patterns = SHORT_DEF[ver]['fields']['chassis'] else: patterns = SHORT_DEF[ver]['fields']['appliance'] for row in ctext: obj = {} match = patterns['re'].search(row) if match: for ndx, col_heading in enumerate(patterns['id']): grouping = match.group(ndx + 1) if grouping: obj[col_heading] = int(grouping) if grouping.isnumeric() else grouping connlist.append(obj) return connlist, {1: patterns} def excel_output(conns, fname, patterns): wb = Workbook() ws = wb.active header = [] for h in patterns.values(): header += h['id'] ws.append(header) for r in conns: row = [] for h in header: row.append(r[h]) ws.append(row) if not fname.endswith('.xlsx'): fname += '.xlsx' wb.save(fname) def json_output(conns): print(json.dumps(conns)) def raw_output(data): for l in data: print(l) if __name__ == '__main__': main() Tested this on version: 11.6Generate private key w/ CSR via iControl REST
Problem this snippet solves: Generate a private key w/ CSR How to use this snippet: To create a private key with a CSR via iControl REST: POST URL:https://10.1.1.165/mgmt/tm/sys/crypto/key Use the data below as your payload. For the name field, it must end in .key or you will get a false 404! Code : { "name":"www.testing.com.key", "commonName":"www.testing.com", "keySize":"4096", "keyType":"rsa-private", "options":[{"gen-csr":"www.testing.com"}], "organization":"Let It Snow Corp.", "ou":"Ice Engineering", "city":"Calhoun", "state":"AZ", "admin-email-address":"jerry@letit.snow", "email-address":"beth@letit.snow", "subject-alternative-name":"DNS:www.testing.com", "challenge-password":"myP4ssword" } Tested this on version: 13.0PowerShell module for the F5 LTM REST API
Problem this snippet solves: To report an issue with the F5-LTM or F5-BIGIP modules, please use the Issues sections of the GitHub repos (here and here) instead of commenting here. Thanks! This PowerShell module uses the iControlREST API to manipulate and query pools, pool members, virtual servers, and iRules. It aims to support version 11.5.1 and higher, and to conform to the schedule for technical support of versions, though this may eventually prove to become difficult. The module currently includes some functionality that, strictly speaking, is outside the scope of the LTM module. Hence, there is an active effort to wrap this LTM module into a larger BIG-IP module, and relocate that functionality elsewhere within that parent module, as well as expand the scope of functionality to include BIG-IP DNS (formerly GTM) and possibly other areas. Both the LTM module and the parent BIG-IP module are projects on github. Please use these projects to report any issues you discover. Thanks! The module contains the following functions. Add-iRuleToVirtualServer Add-iRuleToVirtualServer Add-PoolMember Add-PoolMonitor Disable-PoolMember Disable-VirtualServer Enable-PoolMember Enable-VirtualServer Get-CurrentConnectionCount (deprecated; use Get-PoolMemberStats | Select-Object -ExpandProperty 'serverside.curConns') Get-F5Session (will be deprecated in future versions. use New-F5Session) Get-F5Status Get-HealthMonitor Get-HealthMonitorType Get-iRule Get-iRuleCollection (deprecated; use Get-iRule) Get-Node Get-BIGIPPartition Get-Pool Get-PoolList (deprecated; use Get-Pool) Get-PoolMember Get-PoolMemberCollection (deprecated; use Get-PoolMember) Get-PoolMemberCollectionStatus Get-PoolMemberDescription (deprecated; use Get-PoolMember) Get-PoolMemberIP (deprecated; use Get-PoolMember) Get-PoolMembers (deprecated; use Get-PoolMember) Get-PoolMemberStats Get-PoolMemberStatus (deprecated; use Get-PoolMember) Get-PoolMonitor Get-PoolsForMember Get-StatusShape Get-VirtualServer Get-VirtualServeriRuleCollection (deprecated; use Get-VirtualServer | Where rules | Select -ExpandProperty rules) Get-VirtualServerList (deprecated; use Get-VirtualServer) Invoke-RestMethodOverride New-F5Session New-HealthMonitor New-Node New-Pool New-VirtualServer Remove-HealthMonitor Remove-iRule Remove-iRuleFromVirtualServer Remove-Pool Remove-PoolMember Remove-PoolMonitor Remove-ProfileRamCache Remove-Node Remove-VirtualServer Set-iRule Set-PoolLoadBalancingMode (deprecated; use Set-Pool) Set-PoolMemberDescription Set-Pool Set-VirtualServer Sync-DeviceToGroup Test-F5Session Test-Functionality Test-HealthMonitor Test-Node Test-Pool Test-VirtualServer How to use this snippet: To use the module, click 'Download Zip', extract the files, and place them in a folder named F5-LTM beneath your PowerShell modules folder. By default, this is %USERPROFILE%\Documents\WindowsPowerShell\Modules. The WindowsPowerShell and Modules folders may need to be created. You will most likely need to unblock the files after extracting them. Use the Unblock-File PS cmdlet to accomplish this. The Validation.cs class file (based on code posted by Brian Scholer) allows for using the REST API with LTM devices with self-signed SSL certificates. Nearly all of the functions require an F5 session object as a parameter, which contains the base URL for the F5 LTM and a credential object for a user with privileges to manipulate the F5 LTM via the REST API. Use the New-F5session function to create this object. This function expects the following parameters: The name or IP address of the F5 LTM device A credential object for a user with rights to use the REST API An optional TokenLifespan value for extending the life of the authentication token past the default 20 minutes You can create a credential object using Get-Credential and entering the username and password at the prompts, or programmatically like this: $secpasswd = ConvertTo-SecureString "PlainTextPassword" -AsPlainText -Force $mycreds = New-Object System.Management.Automation.PSCredential "username", $secpasswd Thanks to Kotesh Bandhamravuri and his blog entry for this snippet. There is a function called Test-Functionality that takes an F5Session object, a new pool name, a new virtual server, an IP address for the virtual server, and a computer name as a pool member, and validates nearly all the functions in the module. I've also contributed this code sample for how to gather some basic info about your LTM with this PS module. The module has been tested on: 11.5.1 Build 8.0.175 Hotfix 8 and later 11.6.0 Build 5.0.429 Hotfix 4 and later 12.0 / 12.1 13.0 Code : https://github.com/joel74/POSH-LTM-Rest Tested this on version: 11.5
iCR Python Module for iControl REST
Problem this snippet solves: This is a python module to simplify using iControl REST. Install using pip: pip install iCR or retrieve from https://pypi.python.org/pypi?:action=display&name=iCR&version=2.1 As simple as: #!/usr/bin/env python from iCR import iCR bigip = iCR("172.24.9.132","admin","admin") virtuals = bigip.get("ltm/virtual") for vs in virtuals['items']: print vs['name'] This prints out a list of Virtual Servers. Supported methods: init(hostname,username,password,[timeout,port,icontrol_version,folder,token,debug]) get(url,[select,top,skip,filter]) -> returns data or False getlarge(url,size,[select]) -> Used to retrieve large datasets in chunks. Returns data or False create(url,data) -> returns data or False modify(url,data,[patch=True]) -> returns data or False delete(url) -> returns True or False upload(file) -> file is a local file eg /var/tmp/test.txt, returns True or False download(file) -> files are located in /shared/images, returns True or False create_cert(files) -> files is an array containing paths to cert and key. Returns name of cert or False get_asm_id(name) -> name is the name of a policy. Returns an array of IDs or False create_hash(name) -> name is the name of the partition and policy. eg /Common/test_policy. This reduces the need to retrieve an array of hashes from the BIG-IP. Returns a string. get_token() -> this retrieves a BIG-IP token based on the username and password and sets it as the token in use. Returns the token ID or False delete_token() -> This deletes the object token from the BIG-IP and from the object create_transaction() -> creates a transaction and returns the transaction number ID as a string, or False. Subsequent requests will be added to thetransaction until commit_transaction is called. Transaction ID is stored in object.transaction commit_transaction() -> Commits the transaction stored in object.transaction. Returns True or False command(args,[cmd]) -> Runs a command using the arguments string args. Returns the returned output or True on success or False on failure. Note:Be sure to double-escape single quotes eg \\' and single escape double quotes eg \" cmd options are ping/save/load/restart/reboot Module Variables: icr_session - the link to the requests session raw - the raw returned JSON code - the returned HTTP Status Code eg 200 error - in the case of error, the exception error string headers - the response headers icontrol_version - set this to specify a specific version of iControl debug - boolean True or False to set debugging on or off port - set the port ( 443 by default ) folder - set this to create in a specific partition token - use this to set a specific token. If this is set, it will be used instead of basic auth select - use this with get to select the returned data top - use this with get to return a set number of records skip - use this to skip to a specific record number transaction - stores the Transaction ID How to use this snippet: Examples Setup a REST connection to a device #!/usr/bin/env python from iCR import iCR bigip = iCR("172.24.9.132","admin","admin",timeout=10) Create a Virtual Server vs_config = {'name':'test_vs'} createvs = bigip.create("ltm/virtual",vs_config,timeout=5) Retrieve the VS we just created virt = bigip.get("ltm/virtual/test_vs",select="name") print "Virtual Server created: " + virt['name'] Set the timeout bigip.timeout = 20 Now delete the VS we just created delvs = bigip.delete("ltm/virtual/test_vs") Retrieve ASM policy to ID mapping policies = bigip.get("asm/policies",select="name,id") Print a table of ASM policies with learning mode print print "Policy Name Learning Mode" print "------------------------------------------" for item in policies['items']: enabled = bigip.get("asm/policies/" + item['id'] + "/policy-builder",select="learningMode") print '{:32}'.format(item['name']) + enabled['learningMode'] File upload fp = "/home/pwhite/input.csv" if bigip.upload(fp): print "File " + fp + " uploaded" File download file="BIGIP-12.1.2.0.0.249.iso" download = bigip.download(file) if not download: print "File " + file + " download error" SSL Certificate creation In different folder bigip.folder = "TestFolder" files = ("TestCert.crt","TestCert.key") cert = bigip.create_cert(files) if cert: print "Certificate " + cert + " created" Turn on debugging bigip.debug = True Retrieve ASM policy IDs asm = bigip.get_asm_id("dummy_policy") print len(asm) + " IDs returned" print "ID: " + str(asm[0]) Convert an ASM policy name to hash hash = bigip.create_hash("/Common/test-policy") enabled = bigip.get("asm/policies/" + hash + "/policy-builder",select="learningMode") print '{:32}'.format(item['name']) + enabled['learningMode'] Retrieve and use a token bigip.get_token() Delete the token bigip.delete_token() Developed on Python 2.7 but works with v3. Works on TMOS 11.6 onwards though some features may not be implemented, such as tokens. If you use this and have found bugs, would like to discuss it or suggest features then please PM me on DevCentral. Tested this on version: 13.0
Overwriting or adding LTM SSL Traffic cert and key using iControlREST
Hi, I am trying to overwrite an existing cert and key within the LTM SSL Traffic cert and key using iControlREST. Here is the basic process, and result of each step. Upload key and cert PEM files to the uploads directory. I have tried this step both inside and outside of a transaction with the same result. This works fine. Create a transaction using the transaction REST endpoint. This works fine. Add a command to install the key over the desired SSL Traffic key referencing the local path from step 1 with the transaction id in the header. The command is set to install and from-local-file. Successfully added to the transaction commands. Add a command to install the key over the desired SSL Traffic cert referencing the local path from step 1 with the transaction id in the header. The command is set to install and from-local-file. Successfully added to the transaction commands. Get the transaction commands just to observe the contents. The commands are present, and the paths are correct per steps 3 & 4 above. Attempt to commit the transaction, and receive the failure with a message like the one below. message=transaction failed:01070712:3: file (/var/system/tmp/tmsh/GexeqO/IIS-F5v13.key) expected to exist. As you can see, F5 is looking in a different directory than specified in steps 3 & 4. I've closely examined all requests and responses using Fiddler, and there's no way to determine the randomly generated sub directory name ('GexeqO' in this particular case). It is different each transaction. Also note, this happens even when not overwriting existing entries. But I am using a transaction so that I don't get the 'key and certificate do not match' message. Any insights would be tremendously helpful. Best, Gary
How BIG-IP Token/Authentication works ?
I'm unable to find anywhere here/documentation/articles anyone that could explain a little bit better the authentication token when you get the response from the Rest. I'm sending the POST to the Rest, and the Rest is returning the Authentication. Here is an example: token : AD2GKZPXKVTE4WKJEQUZTIPOM3 name : AD2GKZPXKVTE4WKJEQUZTIPOM3 userName : admin authProviderName : tmos user : ... groupReferences : ... timeout : 1200 startTime : 2016-07-22T09:24:11.808-0500 address : 10.10.10.10 partition : [All] generation : 1 lastUpdateMicros : 1469197451807722 expirationMicros : 1469198651808000 kind : shared:authz:tokens:authtokenitemstate selfLink : https://localhost/mgmt/shared/authz/tokens/AD2GKZPXKVTE4WKJEQUZTIPOM3 Does anyone knows what is "lastUpdateMicros", "ExpirationMicros" and what is Timeout actually means ? I'm having several issues in my scripts when I call the Rest and the call just fail. If I try to get a new token the call works. I wonder if could be due the token is expired after is used once. Will the token expire only after 1200 seconds or that is not true ?Working with subsets of data-group records via iControl REST
The BIG-IP iControl REST interface method for data-groups does not define the records as a subcollection (like pool members.) This is problematic for many because the records are just a list attribute in the data-group object. This means that if you want to add, modify, or delete a single record in a data-group, you have replace the entire list. A short while back I was on a call with iRule extraordinaire John Alam, and he was showing me a management tool he was working on where he could change individual records in a data-group via REST. I was intrigued so we dug into the details and I was floored at how simple the solution to this problem is! UPDATE: Chris L mentioned in the comments below that working with subsets IS possible without the tmsh script, as helearned in this thread in the Q&A section. The normal endpoint (/mgmt/tm/ltm/data-group/internal/yourDGname) works just fine, but instead of trying to change a subset of the records attribute, which only results in the replace-all-with behavior, you can use theoptions query parameterand then pass the normal tmsh command records data as arguments. An example of this request would be to PATCHwith an empty json payload ({})to urlhttps://{{host}}/mgmt/tm/ltm/data-group/internal/mydg?options=records%20modify%20%7B%20k3%20%7Bdata%20v3%20%7D%20%7D(without the encoding, that query value format is “records modify { k3 { data v3 } }”). As this article is still a good learning exercise on how to use tmsh scripts with iControl REST, I’ll keep the article as is, but an updated script for the specific problem we’re solving can befound in this gist on Github. Enter the tmsh script! Even though the iControl REST doesn’t treat data-group records as individual objects, the tmsh cli does. So if you can create a tmsh script to manage the local manipulation of the records, pass your record data into that script, and execute it from REST, well, that’s where the gold is, people. Let’s start with the tmsh script, written by John Alam but modified very slightly by me. cli script dgmgmt { proc script::init {} { } proc script::run {} { set record_data [lindex $tmsh::argv 3] switch [lindex $tmsh::argv 1] { "add-record" { tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records add $record_data puts "Record [lindex $tmsh::argv 3] added." } "modify-record" { tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records modify $record_data puts "Record changed [lindex $tmsh::argv 3]." } "delete-record" { tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records delete $record_data puts "Record [lindex $tmsh::argv 3] deleted." } "save" { tmsh::save sys config puts "Config saved." } } } proc script::help {} { } proc script::tabc {} { } total-signing-status not-all-signed } This script is installed on the BIG-IP and is a regular object in the BIG-IP configuration, stored in the bigip_script.conf file. There are four arguments passed. The first (arg 0) is always the script name. The other args we pass to the script are: arg 1 - action. Are we adding, modifying, or deleting records? arg 2 - data-group name arg 3 - data-group records to be changed The commands are pretty straight forward. Notice, however, that the record data at the tail end of each of those commands is just the data passed to the script, so the required tmsh format is left to the remote side of this transaction. Since I’m writing that side of the solution, that’s ok, but if I were to put my best practices hat on, the record formatting work should really be done in the tmsh script, so that all I have to do on the remote side is pass the key/value data. Executing the script! Now that we have a shiny new tmsh script for the BIG-IP, we have two issues. We need to install that script on the BIG-IP in order to use it We need to be able to run that script remotely, and pass data to it This is where you grab your programming language of choice and go at it! For me, that would be python. And I’ll be using the BIGREST SDK to interact with BIG-IP. Let’s start with the program flow: if __name__ == "__main__": args = build_parser() b = instantiate_bigip(args.host, args.user) if not b.exist("/mgmt/tm/cli/script/dgmgmt"): print( "\n\tThe data-group management tmsh script is not yet on your system, installing..." ) deploy_tmsh_script(b) sleep(2) if not b.exist(f"/mgmt/tm/ltm/data-group/internal/{args.datagroup}"): print( f"\n\tThe {args.datagroup} data-group doesn't exist. Please specify an existing data-group.\n" ) sys.exit() cli_arguments = format_records(args.action, args.datagroup, args.dgvalues) dg_update(b, cli_arguments) dg_listing(b, args.datagroup) This is a cli script, so we need to create a parser to handle the arguments. After collecting the data, we instantiate BIG-IP. Next, we check for the existence of the tmsh script on BIG-IP and install it if it is not present. We then format the record data and proceed to supply that output as arguments when we make the REST call to run the tmsh script. Finally, we print the results. This last step is probably not something you'd want to do for large data sets, but it's included here for validation. Now, let's look at each step of the flow. The imports and tmsh script # Imports used in this script from bigrest.bigip import BIGIP from time import sleep import argparse import getpass import sys # The tmsh script DGMGMT_SCRIPT = 'proc script::init {} {\n}\n\nproc script::run {} {\nset record_data [lindex $tmsh::argv 3]\n\n' \ 'switch [lindex $tmsh::argv 1] {\n "add-record" {\n tmsh::modify ltm data-group internal ' \ '[lindex $tmsh::argv 2] type string records add $record_data\n ' \ 'puts "Record [lindex $tmsh::argv 3] added."\n }\n "modify-record" {\n ' \ 'tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records modify' \ ' $record_data\n puts "Record changed [lindex $tmsh::argv 3]."\n }\n "delete-record" {\n' \ ' tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records delete' \ ' $record_data\n puts "Record [lindex $tmsh::argv 3] deleted."\n }\n "save" {\n ' \ ' tmsh::save sys config\n puts "Config saved."\n }\n}\n}\n' \ 'proc script::help {} {\n}\n\nproc script::tabc {} {\n}\n' These are defined at the top of the script and are necessary to the appropriate functions defined in the below sections. You could move the script into a file and load it, but it's small enough that it doesn't clutter the script and makes it easier not to have to manage multiple files. The parser def build_parser(): parser = argparse.ArgumentParser() parser.add_argument("host", help="BIG-IP IP/FQDN") parser.add_argument("user", help="BIG-IP Username") parser.add_argument( "action", help="add | modify | delete", choices=["add", "modify", "delete"] ) parser.add_argument("datagroup", help="Data-Group name you wish to change") parser.add_argument( "dgvalues", help='Key or KV Pairs, in this format: "k1,k2,k3=v3,k4=v4,k5"' ) return parser.parse_args() This is probably the least interesting part, but I'm including it here to be thorough. The one thing to note is the cli format to supply the key/value pairs for the data-group records. I could have also added an alternate option to load a file instead, but I'll leave that as an exercise for future development. If you supply no arguments or the optional -h/--help, you'll get the help message. % python dgmgmt.py -h usage: dgmgmt.py [-h] host user {add,modify,delete} datagroup dgvalues positional arguments: host BIG-IP IP/FQDN user BIG-IP Username {add,modify,delete} add | modify | delete datagroup Data-Group name you wish to change dgvalues Key or KV Pairs, in this format: "k1,k2,k3=v3,k4=v4,k5" optional arguments: -h, --help show this help message and exit Instantiation def instantiate_bigip(host, user): pw = getpass.getpass(prompt=f"\n\tWell hello {user}, please enter your password: ") try: obj = BIGIP(host, user, pw) except Exception as e: print(f"Failed to connect to {args.host} due to {type(e).__name__}:\n") print(f"{e}") sys.exit() return obj I don't like typing out my passwords on the cli so I use getpass here to ask for it after I kick off the script. You'll likely want to add an argument for the password if you automate this script with any of your tooling. This function makes a request to BIG-IP and builds a local python object to be used for future requests. Uploading the tmsh script def deploy_tmsh_script(bigip): try: cli_script = {"name": "dgmgmt", "apiAnonymous": DGMGMT_SCRIPT} bigip.create("/mgmt/tm/cli/script", cli_script) except Exception as e: print(f"Failed to create the tmsh script due to {type(e).__name__}:\n") print(f"{e}") sys.exit() Because tmsh scripts are BIG-IP objects, we don't have to interact with the file system. It's just a simple object creation like creating a pool. I have taken the liberty to hardcode the script name to limit the number of arguments required to pass on the cli, but that can be updated if you so desire by either changing the name in the script, or adding arguments. Formatting the records def format_records(action, name, records): recs = "" for record in records.split(","): x = record.split("=") record_key = x[0] if len(x) == 1 and action != 'modify': recs += f"{record_key} " elif len(x) == 1 and action == 'modify': recs += f'{record_key} {{ data \\\"\\\" }} ' elif len(x) == 2: record_value = x[1] recs += f'{record_key} {{ data \\\"{record_value}\\\" }} ' else: raise ValueError("Max record items is 2: key or key/value pair.") return f"{action}-record {name} '{{ {recs} }}'" This is the function I spent the most time ironing out. As I pointed out earlier, it would be better handled in the tmsh script, but since that work was already completed by John, I focused on the python side of things. The few things that I fleshed out in testing that I didn't consider while making it work the first time: Escaping all the special characters that make iControl REST unhappy. Handling whitespace in the data value. This requires quotes around the data value. Modifying a key by removing an existing value. This requires you to provide an empty data reference. Executing the script def dg_update(bigip, cli_args): try: dg_mods = {"command": "run", "name": "/Common/dgmgmt", "utilCmdArgs": cli_args} bigip.command("/mgmt/tm/cli/script", dg_mods) except Exception as e: print(f"Failed to modify the data-group due to {type(e).__name__}:\n") print(f"{e}") sys.exit() With all the formatting out of the way, the update is actually anticlimactic. iControl REST requires a json payload for the command, which is running the cli script. The cli arguments for that script are passed in the utilCmdArgs attribute. Validating the results def dg_listing(bigip, dgname): dg = b.load(f'/mgmt/tm/ltm/data-group/internal/{dgname}') print(f'\n\t{args.datagroup}\'s updated record set: ') for i in dg.properties['records']: print(f'\t\tkey: {i["name"]}, value: {i["data"]}') print('\n\n') And finally, we bask in the validity of our updates! Like the update function, this one doesn't have much to do. It grabs the data-group contents from BIG-IP and prints out each of the key/value pairs. As I indicated earlier, this may not be desirable on large data sets. You could modify the function by passing the keys you changed and compare that to the full results returned from BIG-IP and only print the updates, but I'll leave that as another exercise for future development. This is a cool workaround to the non-subcollection problem with data-groups that I wish I'd thought of years ago! The full script is in the codeshare. I hope you got something out of this article, drop a comment below and let me know!Demystifying iControl REST Part 6: Token-Based Authentication
iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5. Several articles on basic usage have been written on iControl REST so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. This article will cover the details on how to retrieve and use an authentication token from the BIG-IP using iControl REST and the python programming language. This token is used in place of basic authentication on API calls, which is a requirement for external authentication. Note that for configuration changes, version 12.0 or higher is required as earlier versions will trigger an un-authorized error. The tacacs config in this article is dependent on a version that I am no longer able to get installed on a modern linux flavor. Instead, try this Dockerized tacacs+ server for your testing. The Fine Print The details of the token provider are here in the wiki. We’ll focus on a provider not listed there: tmos. This provider instructs the API interface to use the provider that is configured in tmos. For this article, I’ve configured a tacacs server and the BIG-IP with custom remote roles as shown below to show BIG-IP version 12’s iControl REST support for remote authentication and authorization. Details for how this configuration works can be found in the tacacs+ article I wrote a while back. BIG-IP tacacs+ configuration auth remote-role { role-info { adm { attribute F5-LTM-User-Info-1=adm console %F5-LTM-User-Console line-order 1 role %F5-LTM-User-Role user-partition %F5-LTM-User-Partition } mgr { attribute F5-LTM-User-Info-1=mgr console %F5-LTM-User-Console line-order 2 role %F5-LTM-User-Role user-partition %F5-LTM-User-Partition } } } auth remote-user { } auth source { type tacacs } auth tacacs system-auth { debug enabled protocol ip secret $M$Zq$T2SNeIqxi29CAfShLLqw8Q== servers { 172.16.44.20 } service ppp } Tacacs+ Server configuration id = tac_plus { debug = PACKET AUTHEN AUTHOR access log = /var/log/access.log accounting log = /var/log/acct.log host = world { address = ::/0 prompt = "\nAuthorized Access Only!\nTACACS+ Login\n" key = devcentral } group = adm { service = ppp { protocol = ip { set F5-LTM-User-Info-1 = adm set F5-LTM-User-Console = 1 set F5-LTM-User-Role = 0 set F5-LTM-User-Partition = all } } } group = mgr { service = ppp { protocol = ip { set F5-LTM-User-Info-1 = mgr set F5-LTM-User-Console = 1 set F5-LTM-User-Role = 100 set F5-LTM-User-Partition = all } } } user = user_admin { password = clear letmein00 member = adm } user = user_mgr { password = clear letmein00 member = mgr } } Basic Requirements Before we look at code, however, let’s take a look at the json payload requirements, followed by response data from a query using Chrome’s Advanced REST Client plugin. First, since we are sending json payload, we need to add the Content-Type: application/json header to the query. The payload we are sending with the post looks like this: { "username": "remote_auth_user", "password": "remote_auth_password", "loginProviderName": "tmos" } You submit the same remote authentication credentials in the initial basic authentication as well, no need to have access to the default admin account credentials. A successful query for a token returns data like this: { username: "user_admin" loginReference: { link: "https://localhost/mgmt/cm/system/authn/providers/tmos/1f44a60e-11a7-3c51-a49f-82983026b41b/login" }- token: { uuid: "4d1bd79f-dca7-406b-8627-3ad262628f31" name: "5C0F982A0BF37CBE5DE2CB8313102A494A4759E5704371B77D7E35ADBE4AAC33184EB3C5117D94FAFA054B7DB7F02539F6550F8D4FA25C4BFF1145287E93F70D" token: "5C0F982A0BF37CBE5DE2CB8313102A494A4759E5704371B77D7E35ADBE4AAC33184EB3C5117D94FAFA054B7DB7F02539F6550F8D4FA25C4BFF1145287E93F70D" userName: "user_admin" user: { link: "https://localhost/mgmt/cm/system/authn/providers/tmos/1f44a60e-11a7-3c51-a49f-82983026b41b/users/34ba3932-bfa3-4738-9d55-c81a1c783619" }- groupReferences: [1] 0: { link: "https://localhost/mgmt/cm/system/authn/providers/tmos/1f44a60e-11a7-3c51-a49f-82983026b41b/user-groups/21232f29-7a57-35a7-8389-4a0e4a801fc3" }- - timeout: 1200 startTime: "2015-11-17T19:38:50.415-0800" address: "172.16.44.1" partition: "[All]" generation: 1 lastUpdateMicros: 1447817930414518 expirationMicros: 1447819130415000 kind: "shared:authz:tokens:authtokenitemstate" selfLink: "https://localhost/mgmt/shared/authz/tokens/4d1bd79f-dca7-406b-8627-3ad262628f31" }- generation: 0 lastUpdateMicros: 0 } Among many other fields, you can see the token field with a very long hexadecimal token. That’s what we need to authenticate future API calls. Requesting the token programmatically In order to request the token, you first have to supply basic auth credentials like normal. This is currently required to access the /mgmt/shared/authn/login API location. The basic workflow is as follows (with line numbers from the code below in parentheses): Make a POST request to BIG-IP with basic authentication header and json payload with username, password, and the login provider (9-16, 41-47) Remove the basic authentication (49) Add the token from the post response to the X-F5-Auth-Token header (50) Continue further requests like normal. In this example, we’ll create a pool to verify read/write privileges. (1-6, 52-53) And here’s the code (in python) to make that happen: def create_pool(bigip, url, pool): payload = {} payload['name'] = pool pool_config = bigip.post(url, json.dumps(payload)).json() return pool_config def get_token(bigip, url, creds): payload = {} payload['username'] = creds[0] payload['password'] = creds[1] payload['loginProviderName'] = 'tmos' token = bigip.post(url, json.dumps(payload)).json()['token']['token'] return token if __name__ == "__main__": import os, requests, json, argparse, getpass requests.packages.urllib3.disable_warnings() parser = argparse.ArgumentParser(description='Remote Authentication Test - Create Pool') parser.add_argument("host", help='BIG-IP IP or Hostname', ) parser.add_argument("username", help='BIG-IP Username') parser.add_argument("poolname", help='Key/Cert file names (include the path.)') args = vars(parser.parse_args()) hostname = args['host'] username = args['username'] poolname = args['poolname'] print "%s, enter your password: " % args['username'], password = getpass.getpass() url_base = 'https://%s/mgmt' % hostname url_auth = '%s/shared/authn/login' % url_base url_pool = '%s/tm/ltm/pool' % url_base b = requests.session() b.headers.update({'Content-Type':'application/json'}) b.auth = (username, password) b.verify = False token = get_token(b, url_auth, (username, password)) print '\nToken: %s\n' % token b.auth = None b.headers.update({'X-F5-Auth-Token': token}) response = create_pool(b, url_pool, poolname) print '\nNew Pool: %s\n' % response Running this script from the command line, we get the following response: FLD-ML-RAHM:scripts rahm$ python remoteauth.py 172.16.44.15 user_admin myNewestPool1 Password: user_admin, enter your password: Token: 2C61FE257C7A8B6E49C74864240E8C3D3592FDE9DA3007618CE11915F1183BF9FBAF00D09F61DE15FCE9CAB2DC2ACC165CBA3721362014807A9BF4DEA90BB09F New Pool: {u'generation': 453, u'minActiveMembers': 0, u'ipTosToServer': u'pass-through', u'loadBalancingMode': u'round-robin', u'allowNat': u'yes', u'queueDepthLimit': 0, u'membersReference': {u'isSubcollection': True, u'link': u'https://localhost/mgmt/tm/ltm/pool/~Common~myNewestPool1/members?ver=12.0.0'}, u'minUpMembers': 0, u'slowRampTime': 10, u'minUpMembersAction': u'failover', u'minUpMembersChecking': u'disabled', u'queueTimeLimit': 0, u'linkQosToServer': u'pass-through', u'queueOnConnectionLimit': u'disabled', u'fullPath': u'myNewestPool1', u'kind': u'tm:ltm:pool:poolstate', u'name': u'myNewestPool1', u'allowSnat': u'yes', u'ipTosToClient': u'pass-through', u'reselectTries': 0, u'selfLink': u'https://localhost/mgmt/tm/ltm/pool/myNewestPool1?ver=12.0.0', u'serviceDownAction': u'none', u'ignorePersistedWeight': u'disabled', u'linkQosToClient': u'pass-through'} You can test this out in the Chrome Advanced Rest Client plugin, or from the command line with curl or any other language supporting REST clients as well, I just use python for the examples well, because I like it. I hope you all are digging into iControl REST! What questions do you have? What else would you like clarity on? Drop a comment below.
How to efficiently delete a node from LTM?
Hi, I am trying to delete a node from F5 LTM as part of our decommission process. I cannot delete a node until I remove all pool members that refer to it. The only way I know how to get pool members is to get all pools and then, for each pool, get members, and see if the IP address of a member matches. With thousands of pools we have this takes thousands of web requests. It is seriously slow even on local network but if I try to do it to our LB on another continent over 160ms round-trip link, it literally takes 15+minutes. So, is there more efficient way to delete a node? Maybe get a list of all back-references to a node? Or a list of all members of all pools in one shot? Or something else I did not think about? I am not even trying to tackle nodes used in iRules and other non-pool objects at this time (though it would be cool). I did try to delete node first and parse a 400 response but: - response only has one pool out of possibly many. - the name of the pool in response is within an arbitrary text "message" string. This string is already different between 11.4 and 11.5, and, since it is not part of API, can change at any hotfix.
