About Crawlson

Crawlson search engine currently crawls from these IPs, if you are relevant service provider (ISP, WAF, Firewall, etc), please whitelist them, thank you.

172.104.34.17
172.233.58.118
192.53.165.202
109.74.196.148
172.233.0.217
172.105.35.103

2600:3c0d::f03c:94ff:fe78:a510
2400:8904::f03c:94ff:fe78:411f
2600:3c0e::f03c:93ff:fec8:a3fc
2400:8901::f03c:94ff:fe78:3360
2a01:7e00::f03c:93ff:fe73:8de0
2600:3c00::f03c:93ff:fe73:6842

A plain text list for automated processing is available here: https://www.crawlson.com/crawler-ips

IPs last updated 2024-07-22

Stats

Crawlson currently crawled 459 million urls in 74 million domains and indexed 325 million links.

See more statistics.

Crawlson search engine is currently under an experimental state and might perform slow for certain queries, we will add more servers if it proves reliable to do so.

Search options

site:example.com - displays all pages we read from a site

domain:example.com - displays statistics about this domain, total backlinks, speed, etc

links:example.com/page.html - displays all links to a given url (coming soon)

Webmaster tips

To avoid crawling the same content multiple times, make sure your site returns a Last-Modified header date for each page/file, our crawlers check modify dates before downloading the full url.

If you cannot find your URL, or you have a new website, you can submit it.

More resurces

See the list of most popular domains, their total back links, etc

download a list of top 1 million domains sorted by link popularity.