Forum Discussion

oguzy's avatar
oguzy
Icon for Cirrostratus rankCirrostratus
Aug 07, 2018

Upload programatically external data group file?

Hi everyone,

 

I use external data group file contains more than 350K records. And now I would like to programmatically update (overwrite) the data group file for the automation process. I did some research and found something bigsuds library but the python version we used is not supported.

 

Does someone have any idea about uploading external data group file using python?

 

F5 version is 12.1.3.3 Python version is 3.7.0.

 

Thanks in advance.

 

--

 

Oguz

 

  • Here's iControl REST curl examples. You should be able to find equivalent Python classes for these endpoints. 

    To create an LTM external group, you need to 1) Upload the file to BIG-IP and 2) Create the data-group object from the file.

    1) Uploading a data file and creating the file object

    To upload the remote file http://172.16.10.10/dg-testSw.txt to BIG-IP and create the file object named dg-testSw.txt based on the data file, run

    curl -sku admin: https:///mgmt/tm/sys/file/data-group \
      -X POST -H "Content-type: application/json" \
      -d '{"name":"dg-testSw.txt", "separator":":=", "type":"string", "sourcePath":"http://172.16.10.10/dg-testSw.txt"}'

    The above call is equivalent to the following tmsh command: 

    tmsh create sys file data-group dg-testSw.txt separator ":=" source-path http://172.16.10.10/dg-testSw.txt type string

    The file object dg-testSw.txt is created under the /Common partition by default. To store the object under a different partition, add the partition property to the JSON body: e.g., "partition":"testFolder".

    The separator property describes the key-value separator string. In this example, it is ":=". The type property describes the type of the data. In this example, it is "string". Possible values are: integer, ip or string. In this example, the following data is used.

    "darth" := "vader",
    "luke" := "skywalker",
    "han" := "solo",
    "leia" := "organa",
    "moff" := "tarkin",

    Please refer to v11: iRules Data Group Updates for the data format.

    To delete the data-group file object dg-testSw.txt, run

    curl -sku admin: https:///mgmt/tm/sys/file/data-group/dg-testSw.txt -X DELETE

     2) Creating the ltm data-group object

    To create the ltm data-group object dg-testSw from the above file object, run

    curl  -sku admin: https:///mgmt/tm/ltm/data-group/external \
      -X POST -H "Content-type: application/json" \
      -d '{"name":"dg-testSw", "externalFileName":"dg-testSw.txt"}'

    The above call is equivalent to the following tmsh command:  

    tmsh create ltm data-group external dg-testSw external-file-name dg-testSw.txt

    To delete the ltm data-group object dg-testSw, run

    curl -sku admin: https:///mgmt/tm/ltm/data-group/external/dg-testSw -X DELETE
  • Good to know that you found the solution.

     

    Here's an alternative solution using F5 Python SDK for iControl REST. While the example below does not give you much advantage over the Jason's pure Python code (uses 'request' module) in this use-case, you may find the SDK useful in some cases.

     

    Just like the curl examples above, the code relies on a http server for uploading the data-group file to BIG-IP, not via a separate scp or POST to the /mgmt/shared/file-transfer/uploads/file.txt endpoint.

     

    BTW, Jason Rahm's another post, "Demystifying iControl REST Part 5: Transferring Files" is a good read on the topic.

     

    Here's the code (Lazy me; I should have refactored it instead of copy-pasting the routines).

     

    from f5.bigip import ManagementRoot
     
     Get all the 'sys file data-group' files
    def getDGFiles(top):
        try:
            dgFiles = top.sys.file.data_groups.get_collection()
            print('Got {} file objects.'.format(len(dgFiles)))
            for idx, f in enumerate(dgFiles):
                print('{}: {}'.format(idx, f.raw))
        except Exception as ex:
            print('Error: {}'.format(ex))
     
     Upload the external data-group file
    def uploadDGFile(top, name, path, dataType):
        print('---- Uploading the file {} as {}'.format(path, name))
        try:
            dgFile = top.sys.file.data_groups.data_group.create(sourcePath=path, name=name, type=dataType)
            print(dgFile.raw)
        except Exception as ex:
            print('Error: {}'.format(ex))
     
     Delete the external data-group file
    def deleteDGFile(top, folder, name):
        print('---- Deleting file object /{}/{}'.format(folder, name))
        try:
            dgObject = top.sys.file.data_groups.data_group.load(name=name, partition=folder)
            dgObject.delete()
            print(dgObject.raw)
        except Exception as ex:
            print('Error: {}'.format(ex))
     
     Get all the 'ltm data-group external' objects
    def getDGObjects(top):
        try:
            dgObjects = top.ltm.data_group.externals.get_collection()
            print('Got {} file objects.'.format(len(dgObjects)))
            for idx, o in enumerate(dgObjects):
                print('{}: {}'.format(idx, o.raw))
        except Exception as ex:
            print('Error: {}'.format(ex))
     
     Add the external data-group object
    def createDGObject(top, name, file):
        print('---- Creating DG object {} from the file {}'.format(name, file))
        try:
            dgObject = top.ltm.data_group.externals.external.create(name=name, externalFileName=file)
            print(dgObject.raw)
        except Exception as ex:
            print('Error: {}'.format(ex))
     
     Delete the external data-group object
    def deleteDGObject(top, folder, name):
        print('---- Deleting DG object /{}/{}'.format(folder, name))
        try:
            dgObject = top.ltm.data_group.externals.external.load(name=name, partition=folder)
            dgObject.delete()
            print(dgObject.raw)
        except Exception as ex:
            print('Error: {}'.format(ex))
     
     
     ==== Main routines =====
    top = ManagementRoot('192.168.184.30', 'admin', 'admin').tm
    getDGFiles(top)
    uploadDGFile(top, 'dg-testSw.txt', 'http://172.16.10.10/dg-testSw.txt', 'string')
    getDGObjects(top)
    createDGObject(top, 'sat2', 'dg-testSw.txt')
    deleteDGObject(top, 'Common', 'sat2')
    deleteDGFile(top, 'Common', 'dg-testSw.txt')