Christoph_Fris1
Jul 10, 2017Nimbostratus
How to allow Google to crawl my Site, when DOS Profile is active?
Hello all,
we activated the DDos Protection on our F5 Cluster, but after that Google is no longer able to crawl our site. Although I already set all "Google" Signatures on the "Whitelist".
But always when I'm checking the crawl status the Google Bot get's this response:
...
Please enable JavaScript to view the page content.
And here is the config from our current Profile:
security dos profile Homepage {
app-service none
application {
Homepage {
bot-defense {
browser-legit-captcha disabled
browser-legit-enabled disabled
mode during-attacks
}
bot-signatures {
categories {
"DOS Tool" {
action block
}
"E-Mail Collector" {
action block
}
"Exploit Tool" {
action block
}
"Network Scanner" {
action block
}
"Search Engine" {
action report
}
"Spam Bot" {
action block
}
"Vulnerability Scanner" {
action block
}
"Web Spider" {
action block
}
"Webserver Stress Tool" {
action block
}
Spyware {
action block
}
}
check enabled
disabled-signatures {
"Facebook External Hit" { }
"Google AdsBot" { }
"Google Desktop" { }
"Google Feedfetcher" { }
"Google Page Speed Insights" { }
"Google Translate" { }
"Google favicon" { }
"Nokia-WAPToolkit.\* googlebot" { }
AppEngine-Google { }
Bing { }
Google { }
Google-Adwords-Instant { }
Google-Calendar-Importer { }
Google-Sitemaps { }
GoogleWebLight { }
Google_Analytics_Snippet_Validator { }
Java { }
Mediapartners-Google { }
YahooSeeker { }
}
}
captcha-response {
failure {
body "You have entered an invalid answer for the question. Please, try again.
%DOSL7.captcha.image% %DOSL7.captcha.change%
What code is in the image\?
%DOSL7.captcha.solution%
%DOSL7.captcha.submit%"
}
first {
body "This question is for testing whether you are a human visitor and to prevent automated spam submission.
%DOSL7.captcha.image% %DOSL7.captcha.change%
What code is in the image\?
%DOSL7.captcha.solution%
%DOSL7.captcha.submit%"
}
}
ip-whitelist {
xxx.xxx.xxx.xxx/xx { }
xxx.xxx.xxx.xxx/xx { }
xxx.xxx.xxx.xxx/xx { }
xxx.xxx.xxx.xxx/xx { }
xxx.xxx.xxx.xxx/xx { }
xxx.xxx.xxx.xxx/xx { }
xxx.xxx.xxx.xxx/xx { }
}
stress-based {
mode blocking
}
tcp-dump {
record-traffic enabled
}
tps-based {
device-client-side-defense enabled
device-rate-limiting enabled
ip-client-side-defense enabled
}
}
}
partition Common
whitelist none
}
Maybe you have a hint for me how to solve this. Current Big-IP version: 12.1.2 - ASM Signatures: v12.1.2/ASM-SignatureFile_20170403_145743
Thanks, Christoph