Crawlson search engine currently crawls from these IPs, if you are relevant service provider (ISP, WAF, Firewall, etc), please whitelist them, thank you.
188.8.131.52 184.108.40.206 220.127.116.11
IPs last updated 2022-01-22
Total urls crawled: 105 161 890.
Total domains: 14 526 654.
Total hosts: 16 013 739.
Total links indexed: 96 409 142.
Stats are updated once per day.
Crawlson search engine is currently under an experimental state and might perform slow for certain queries, we will add more servers if it proves reliable to do so.
Some search options
site:example.com - displays all pages we read from a site
domain:example.com - displays statistics about this domain, total backlinks, speed, etc
links:example.com/page.html - displays all links to a given url (coming soon)
Some webmaster tips
To avoid crawling the same content multiple times, make sure your site returns a Last-Modified header date for each page/file, our crawlers check modify dates before downloading the full url.
If you cannot find your URL, or you have a new website, you can submit it.