After working with different http components from my ASP/COM/VB days , I was pleased to find the .Net classes for web request and response in the System.Net namespace. There are lots of goodies in both that would have required lots of extra code in the past, especially for parsing out header responses.

HttpWebResponse Class
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemnethttpwebresponseclasstopic.asp

HttpWebRequest Class
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemnethttpwebresponseclasstopic.asp

Here is a few snippets of code I pulled out of the DNN FIND Spider to show some of the basics of both classes with comments and ideas for those who may try to use this:

'~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

'Create the web request object and pass in a URL to process
Dim webReq As HttpWebRequest = HttpWebRequest.Create(sURLToProcess)

'You can specify your own useragent using the UserAgent property
webReq.UserAgent = g_sUserAgent

'Create the web response object by retrieving the web request response
Dim webResponse As HttpWebResponse = webReq.GetResponse

'Check to make sure we get a 200 status response
If webResponse.StatusCode = HttpStatusCode.OK Then
 'We can grab the content type of the document
 sContentType = webResponse.ContentType.ToString()
 'We can check the size of the document
 lContentLength = webResponse.ContentLength
 'We now know what type of document this is, we could add code to
 'decide what we want to parse or save the data  

'We can stream the data (requires System.IO).
 Dim receiveStream As Stream = webResponse.GetResponseStream()
 'Using the stream instance, we could pipes the response stream to a
 'higher level stream reader with the required encoding format
 Dim encode As Encoding = System.Text.Encoding.GetEncoding("utf-8")
 Dim readStream As New StreamReader(receiveStream, encode)

'From here we can save or parse the data as needed...

Else
'If we did not get a 200 status, we could process the status response and
'return a friendly error message, or just ignore the document
End IF

' Release the resources of the response.
webResponse.Close()

'~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**PLEASE NOTE** - The above code is just a few snippets of the Spider class for DNN FIND Spider that I jotted down from memory. Make sure you declare your variables as the above is provided just to give you ideas for utilizing and understanding HTTP Web Request and Response classes.

I keep wondering why I waited so long to move to .Net, especially after reviewing some of the old code I had for a web crawler in VB and seeing how much less code that was required to do the same thing. Just thinking about that old code makes me nauseous.