Crawlson search engine currently crawls from these IPs, if you are relevant service provider (ISP, WAF, Firewall, etc), please whitelist them, thank you.
18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206
A plain text list for automated processing is available here: https://www.crawlson.com/crawler-ips
IPs last updated 2022-07-04
Crawlson currently crawled 348 million urls in 47 million domains and indexed 226 million links.
See more statistics.
Crawlson search engine is currently under an experimental state and might perform slow for certain queries, we will add more servers if it proves reliable to do so.
site:example.com - displays all pages we read from a site
domain:example.com - displays statistics about this domain, total backlinks, speed, etc
links:example.com/page.html - displays all links to a given url (coming soon)
To avoid crawling the same content multiple times, make sure your site returns a Last-Modified header date for each page/file, our crawlers check modify dates before downloading the full url.
If you cannot find your URL, or you have a new website, you can submit it.
See the list of most popular domains, their total back links, etc
download a list of top 1 million domains sorted by link popularity.