User-460007017 posted
Hi suchanewbee,
This link provide a sample for the web crawler:
https://www.example-code.com/asp/spider_simpleCrawler.asp
In addition, I could provide the asp.net application sample:
string targetpath = @"d:\webcrawler\test.txt";
Console.WriteLine("Please enter the target URL:");
string URI = Console.ReadLine();
WebRequest myrequest = WebRequest.Create(URI);
WebResponse myresponse= myrequest.GetResponse();
Stream datastream = myresponse.GetResponseStream();
StreamReader reader = new StreamReader(datastream, Encoding.Default);
if(!File.Exists(targetpath))
{
File.Create(targetpath);
};
string htmlcontent = reader.ReadToEnd();
File.WriteAllText(targetpath,htmlcontent);
Best Regards,
Yuk Ding
