Forum Discussion

smp_86112's avatar
smp_86112
Icon for Cirrostratus rankCirrostratus
Oct 21, 2011

Reverse Monitor and Pool Member Recovery

I have constructed a monitor to check a simple page and look for the string "up" in the response. Here's the sanitized definition:


ltm monitor http reverse {
    defaults-from http
    destination *:*
    interval 5
    recv "up"
    reverse enabled
    send "GET /monitor.html HTTP/1.1\r\nHost: app.mydomain.com\r\n\r\n"
    time-until-up 0
    timeout 6
}

In normal mode (i.e. "Reverse" is not enabled), I can change the text and the monitor goes up/down as I expect. Then when I enable the "Reverse" parameter while the page contains "up", the member gets marked down as I expect. However when I then change the HTML to "down" in Reverse mode, the member does not recover.

The doc doesn't really specifically describe what happens when a Reverse monitor recovers, but it seems logical to me that a monitor in Reverse mode should recover when the test fails. Am I not understanding now Reverse mode works? The only think I could find is SOL10443, but I am running 10.2.0.

config
design

10 Replies

Sort By
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 22, 2011
    mine seems to be ok. by the way, is timeout 6 correct? 

    [root@iris:Active] config  b version|grep -iA 1 version
BIG-IP Version 10.2.0 1707.0
Final Edition

[root@iris:Active] config  b pool foo list
pool foo {
   monitor all mymon
   members 10.10.70.110:http {}
}

[root@iris:Active] config  b monitor mymon list
monitor mymon {
   defaults from http
   reverse
   recv "up"
   send "GET /foo.html HTTP/1.1\r\nHost:\r\nConnection: Close\r\n\r\n"
}

[root@iris:Active] config  curl http://10.10.70.110/foo.html


WELCOME!!!!
This's 10.10.70.110.
up



[root@iris:Active] config  b pool foo|grep -i pool\ member
+-> POOL MEMBER foo/10.10.70.110:http   inactive,down

[root@iris:Active] config  curl http://10.10.70.110/foo.html


WELCOME!!!!
This's 10.10.70.110.
down



[root@iris:Active] config  b pool foo|grep -i pool\ member
+-> POOL MEMBER foo/10.10.70.110:http   active,up
  • smp_86112's avatar
    smp_86112
    Icon for Cirrostratus rankCirrostratus
    Oct 24, 2011
    I chose 6 seconds just because I was testing and I didn't want to wait.

     

     

    Thanks for posting your config. I copied your test exactly, and it behaved the way I expected. Then I started tweaking it a bit, and found that the Connection: close header is the key. When I remove it from the monitor, the member does not recover when the monitor is in Reverse mode.
  • smp_86112's avatar
    smp_86112
    Icon for Cirrostratus rankCirrostratus
    Oct 24, 2011

    I did some more testing by adding random characters to the html page (and validating that the receive string was not contained). I found that without the Connection: close header, a response containing less than 1705 bytes (including the headers) would not allow the member to recover. If I add one single caracter more to the page (a total of 1706 or more bytes), the member gets marked up/down as expected.

     

    Can someone else verify?

     

  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 25, 2011
    mine seems working fine. 

    [root@iris:Active] config  b monitor mymon list
monitor mymon {
   defaults from http
   reverse
   recv "up"
   send "GET /foo.html HTTP/1.1\r\nHost:\r\n\r\n"
}

[root@iris:Active] config  curl -i http://10.10.70.110/foo.html|grep -i up
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1418  100  1418    0     0   328k      0 --:--:-- --:--:-- --:--:--     0
up

[root@iris:Active] config  curl -I http://10.10.70.110/foo.html
HTTP/1.1 200 OK
Date: Tue, 25 Oct 2011 05:19:31 GMT
Server: Apache/2.0.59 (rPath)
Last-Modified: Tue, 25 Oct 2011 05:18:59 GMT
ETag: "66a9-58a-ad3cf6c0"
Accept-Ranges: bytes
Content-Length: 1418
Vary: Accept-Encoding
Set-Cookie: testcookie=helloworld
Content-Type: text/html; charset=UTF-8

[root@iris:Active] config  b pool foo|grep -i pool\ member
+-> POOL MEMBER foo/10.10.70.110:http   inactive,down

[root@iris:Active] config  curl -I http://10.10.70.110/foo.html
HTTP/1.1 200 OK
Date: Tue, 25 Oct 2011 05:21:00 GMT
Server: Apache/2.0.59 (rPath)
Last-Modified: Tue, 25 Oct 2011 05:20:45 GMT
ETag: "66ae-58c-b38e6540"
Accept-Ranges: bytes
Content-Length: 1420
Vary: Accept-Encoding
Set-Cookie: testcookie=helloworld
Content-Type: text/html; charset=UTF-8

[root@iris:Active] config  curl -I http://10.10.70.110/foo.html|grep -i up
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  1420    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

[root@iris:Active] config  b pool foo|grep -i pool\ member
+-> POOL MEMBER foo/10.10.70.110:http   active,up
  • smp_86112's avatar
    smp_86112
    Icon for Cirrostratus rankCirrostratus
    Oct 25, 2011
    Thanks again for posting your test. I calculate the length of your headers to be 290 bytes. So even when I add in the 1418 bytes of content, the total length is 1708 bytes. I would be curious to see the behavior if you removed 100 or so bytes from the content, just to be sure you are well under my theoretical threshold. But I understand if you don't want to do any more - thanks for responses. You were able to help me isolate this further than I would have alone.
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 25, 2011
    just a quick question. how come 290 bytes? is it a number of char? i count it 265 bytes. is there anything i missed?

     

     

    by the way, don't worry i can test it. just wanna make sure we are at the same number. ;-)

     

     

  • smp_86112's avatar
    smp_86112
    Icon for Cirrostratus rankCirrostratus
    Oct 25, 2011
    You have to think about this in terms of what's transmitted on the wire, so characters count, spaces count, and so do CRLF characters. I count 291 bytes in the header below, if you include a standard blank line which seperates the headers from the content. I pasted into a text file and ran the wc command. 

    
HTTP/1.1 200 OK
Date: Tue, 25 Oct 2011 05:21:00 GMT
Server: Apache/2.0.59 (rPath)
Last-Modified: Tue, 25 Oct 2011 05:20:45 GMT
ETag: "66ae-58c-b38e6540"
Accept-Ranges: bytes
Content-Length: 1420
Vary: Accept-Encoding
Set-Cookie: testcookie=helloworld
Content-Type: text/html; charset=UTF-8
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 25, 2011
    this is when it is not working. if i add one more char in the response, it starts working. 

    [root@iris:Active] config  b monitor mymon list
monitor mymon {
   defaults from http
   reverse
   recv "up"
   send "GET /foo.html HTTP/1.1\r\nHost:\r\n\r\n"
}

[root@iris:Active] config  curl -I http://10.10.70.110/foo.html
HTTP/1.1 200 OK
Date: Tue, 25 Oct 2011 06:42:24 GMT
Server: Apache/2.0.59 (rPath)
Last-Modified: Tue, 25 Oct 2011 06:41:35 GMT
ETag: "66a9-57a-d4a385c0"
Accept-Ranges: bytes
Content-Length: 1402
Vary: Accept-Encoding
Set-Cookie: testcookie=helloworld
Content-Type: text/html; charset=UTF-8

[root@iris:Active] config  curl -i http://10.10.70.110/foo.html|grep -i up
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1402  100  1402    0     0   292k      0 --:--:-- --:--:-- --:--:--  250k
up

[root@iris:Active] config  b pool foo|grep -i pool\ member
+-> POOL MEMBER foo/10.10.70.110:http   inactive,down

[root@iris:Active] config  curl -I http://10.10.70.110/foo.html
HTTP/1.1 200 OK
Date: Tue, 25 Oct 2011 06:43:03 GMT
Server: Apache/2.0.59 (rPath)
Last-Modified: Tue, 25 Oct 2011 06:42:57 GMT
ETag: "66ae-57c-d986be40"
Accept-Ranges: bytes
Content-Length: 1404
Vary: Accept-Encoding
Set-Cookie: testcookie=helloworld
Content-Type: text/html; charset=UTF-8

[root@iris:Active] config  curl -i http://10.10.70.110/foo.html|grep -i up
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1404  100  1404    0     0   173k      0 --:--:-- --:--:-- --:--:--     0

[root@iris:Active] config  b pool foo|grep -i pool\ member
+-> POOL MEMBER foo/10.10.70.110:http   inactive,down
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 25, 2011
    oop sorry. i forgot to say thanks for the byte's explanation. ;-)
  • smp_86112's avatar
    smp_86112
    Icon for Cirrostratus rankCirrostratus
    Oct 25, 2011

    No problem, thank you for all your independent test work. So it seems your threshold is 1695 bytes (1404 content + 291 headers), and mine is 1705 bytes. But in my mind, that's close enough to convince me this is a bug. It's not exactly the situation described in SOL10443, but it's evident there was some type of problem with that part of the TMM code.