Another C# Web Crawler
Posted: 01/07/09 7:27 AM
A while back I wrote a little sample of a web crawler. It was not terribly sophisticated but has been the biggest draw to this blog through search engines for some time. This is likely because it ranks very high (why I am not sure) in Google’s rankings. For example, my crawler sample page ranks #1 in Google for “c# web crawl” and #3 for “c# web crawler”. Neat.
So the other day I dusted it off and started reworking it. It was such a simple sample and I thought some improvements could be made. Lo and behold, I just got an email about a different web crawler written in c#, which you can find here. I haven’t downloaded and looked at the source yet, but I figured I would point it out.
Comments
Ayush 2009-05-02 06:06:34
I was trying to devise a method to crawl over the website like Yahoo mail by filling in the authentication details, never been much of a ‘web’ developer. The code helped me a lot to understand the details and see the ‘Big Picture’.
Thanks!
Roman Hansen 2011-02-25 06:28:36
Great, just what I need!
Abhie 2011-11-04 12:48:30
Nice code.....thanks