HttpWebRequest Class
In this blog post, i will explain how we used the HttpWebRequest to retrieve a page which needs authentication.
My colleague had an issue in which he needed to retrieve a specific quotes from http://finance.yahoo.com. Of course to be able to retrieve his own created table, he needs to authenticate using his yahoo credentials. My idea was to use HttpWebRequest class by sending the proper authentication cookie in its header which will guarantee the page retrieval.
To retrieve the authenticated header cookie value, i used Fidler to intercept the request and response. After authentication was made, i looked at the filder and get the cookie value from the authenticated request header. In this way, if i added to the HttpWebRequest header collection the "Cookie" and its retrieved value, The web server will deal with my request as an authenticated one.
Below is the implementation (Import System.Net and System.IO)
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://finance.yahoo.com/p?k=pf_1&d=v4");
request.Proxy.Credentials = CredentialCache.DefaultCredentials;
request.Headers.Add("Cookie", "TheEncryptedValueRetrieved");
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
string strResult = sr.ReadToEnd();
StreamWriter sw = new StreamWriter(Server.MapPath("~/test.html"));
sw.WriteLine(strResult);
// Close StreamWriter and StreamReader
sr.Close();
sw.Close();
}
}
Response.Redirect("test.html");
catch(Exception ex)
{
Response.Write(ex.Message);
}
N.B: In my network, Internet access is under a proxy server which was displaying "The remote server returned an error: (407) Proxy Authentication Required.". To fix this issue, i had to add request.Proxy.Credentials = CredentialCache.DefaultCredentials; to the request object which will set the proxy credentials to the default one already configured.
The Encrypted value of the Cookie is replaced in this example by "TheEncryptedValueRetrieved" for security purpose.
Hope this helps