We Keep Coding

Identifying working links and not working links in selenium C#

I login to a website --> I read links from a text file(Which has working and broken links) --> I am using the HttpWebRequest and HttpWebResponse to detect the working and notWorking links ---Here is the main problem---> I am trying to separate the working and not working links and log into two different files. But everything goes into only one file. I am using the StatusCode property to distinguish between the two. But not successful. Please find the code below. I am very new to selenium C#.
public void Login()
{
WebDriver.Navigate().GoToUrl($"{ApplicationUrl}Login.aspx");
WebDriver.Manage().Window.Maximize();
WebDriver.FindElement(By.Id("UserName")).SendKeys(_user.Login);
WebDriver.FindElement(By.Id("tbPassword")).SendKeys(_user.Passwort);
IJavaScriptExecutor js = WebDriver as IJavaScriptExecutor;
js.ExecuteScript("arguments[0].click();",
WebDriver.FindElement(By.Id("LoginButton")));
Thread.Sleep(3000);
string fileName = #"TestFile.txt";
// Reading the text file using StreamReader
using (StreamReader sr = new StreamReader(fileName))
{
String line;
while ((line = sr.ReadLine()) != null)
{
WebDriver.Navigate().GoToUrl(line);
IsLinkWorking(line);
}
WorkingLinks.Close();
NotWorkingLinks.Close();
}
}
public void IsLinkWorking(string line)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(line);
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
WorkingLinks.WriteLine(WebDriver.Url);
// Releases the resources of the response.
response.Close();
}
else
{
NotWorkingLinks.WriteLine(WebDriver.Url);
response.Close();
}
}
catch
{ //TODO: Check for the right exception here
}
}

When HTTP status code outside of range 200-299 it will send to catch, set you code there
catch
{
NotWorkingLinks.WriteLine(WebDriver.Url);
}

HttpWebResponse returns an 'incomplete' stream

I´m making repeated requests to a web server using HttpWebRequest, but I randomly get a 'broken' response stream in return. e.g it doesn´t contain the tags that I KNOW is supposed to be there. If I request the same page multiple times in a row it turns up 'broken' ~3/5.
The request always returns a 200 response so I first thought there was a null value inserted in the response that made the StreamReader think it reached the end.
I´ve tried:
1) reading everything into a byte array and cleaning it
2) inserting a random Thread.Sleep after each request
Is there any potentially bad practice with my code below or can anyone tell me why I´m randomly getting an incomplete response stream? As far as I can see I´m closing all unmanaged resources so that shouldn´t be a problem, right?
public string ReturnHtmlResponse(string url)
{
string result;
var request = (HttpWebRequest)WebRequest.Create(url);
{
using(var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine((int)response.StatusCode);
var encoding = Encoding.GetEncoding(response.CharacterSet);
using(var stream = response.GetResponseStream())
{
using(var sr = new StreamReader(stream,encoding))
{
result = sr.ReadToEnd();
}
}
}
}
return result;
}

I do not see any direct flaws in you're code. What could be is that one of the 'Parent' using statements is done before the nested one. Try changing the using to a Dispose() and Close() method.
public string ReturnHtmlResponse(string url)
{
string result;
var request = (HttpWebRequest)WebRequest.Create(url);
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine((int)response.StatusCode);
var encoding = Encoding.GetEncoding(response.CharacterSet);
var stream = response.GetResponseStream();
var sr = new StreamReader(stream,encoding);
result = sr.ReadToEnd();
sr.Close();
stream.Close();
response.Close();
sr.Dispose();
stream.Dispose();
response.Dispose();
return result;
}

Correct usage of WebRequest

I need to use WebRequest to download a webpage content into a string.
I can't use WebClient instead because it doesn't support certain HTTP Headers which i need to use. I couldn't figure out what's the best practice for handling memory issues in this case (How to correctly dispose it). Is using a using statement enough or do i need to add some try catch stuff in here too?
This is my code so far:
var webRequest = (HttpWebRequest)WebRequest.Create("http://www.gooogle.com");
using (var webResponse = (HttpWebResponse)webRequest.GetResponse()) {
using (var responseStream = webResponse.GetResponseStream()) {
responseStream.ReadTimeout = 30;
using (var streamReader = new StreamReader(responseStream, Encoding.UTF8)) {
var page = streamReader.ReadToEnd();
}
}
Console.WriteLine("Done");
}

Your code is fine (except of course that some exception handling would be nice). You don't need to worry about disposing or closing streams when using using, the compiler generates the code for that.
The best thing would of course be to wrap the above code in a function that returns the page, and put a global try/catch in there, for instance:
public string GetHtmlPage(string urlToFetch)
{
string page = "";
try
{
... code ...
return page;
} catch (Exception exc)
{
throw new HtmlPageRetrievalException(exc);
}
}

Getting and parsing webpage from Internet on C# or C++

I'm making a WinForms project on C#/C++ (depending on the best way I could find to reach my goal, language could be changed). I need to get a page from website and parse it to get some information. I'm a very beginner in web programming with Visual C#/C++ and all the answers I found here are too complicated for me as a beginner. Could you help me to tell which standart classes should I use for getting page from Internet in some form and how to parse it then. I would be very pleased if you have any code examples, cause as I wrote above I have no experience in web coding and have no time to learn every term in detail. Thank you in advance.

You can use c# to download the specific webpage then do the analysis, an code example of downloading:
using System.Net;
using System.IO;
using System.Windows.Forms;
string result = null;
string url = "http://www.devtopics.com";
WebResponse response = null;
StreamReader reader = null;
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create( url );
request.Method = "GET";
response = request.GetResponse();
ContentType contentType = new ContentType(response.ContentType);
Encoding encoding = Encoding.GetEncoding(contentType.CharSet);
reader = new StreamReader( response.GetResponseStream(), encoding);
result = reader.ReadToEnd();
}
catch (Exception ex)
{
// handle error
MessageBox.Show( ex.Message );
}
finally
{
if (reader != null)
reader.Close();
if (response != null)
response.Close();
}

Check out this project 'here' and their code examples 'here'

How to check if a file exists on a server using c# and the WebClient class

In my application I use the WebClient class to download files from a Webserver by simply calling the DownloadFile method. Now I need to check whether a certain file exists prior to downloading it (or in case I just want to make sure that it exists). I've got two questions with that:
What is the best way to check whether a file exists on a server without transfering to much data across the wire? (It's quite a huge number of files I need to check)
Is there a way to get the size of a given remote file without downloading it?
Thanks in advance!

WebClient is fairly limited; if you switch to using WebRequest, then you gain the ability to send an HTTP HEAD request. When you issue the request, you should either get an error (if the file is missing), or a WebResponse with a valid ContentLength property.
Edit: Example code:
WebRequest request = WebRequest.Create(new Uri("http://www.example.com/"));
request.Method = "HEAD";
using(WebResponse response = request.GetResponse()) {
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}

When you request file using the WebClient Class, the 404 Error (File Not Found) will lead to an exception. Best way is to handle that exception and use a flag which can be set to see if the file exists or not.
The example code goes as follows:
System.Net.HttpWebRequest request = null;
System.Net.HttpWebResponse response = null;
request = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create("www.example.com/somepath");
request.Timeout = 30000;
try
{
response = (System.Net.HttpWebResponse)request.GetResponse();
flag = 1;
}
catch
{
flag = -1;
}
if (flag==1)
{
Console.WriteLine("File Found!!!");
}
else
{
Console.WriteLine("File Not Found!!!");
}
You can put your code in respective if blocks.
Hope it helps!

What is the best way to check whether a file exists on a server
without transfering to much data across the wire?
You can test with WebClient.OpenRead to open the file stream without reading all the file bytes:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
// ^ throws System.Net.WebException: 'Could not find file...' if file is not present
stream.Close();
}
This will indicate if the file exists at the remote location or not.
To fully read the file stream, you would do:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
StreamReader sr = new StreamReader(stream);
Console.WriteLine(sr.ReadToEnd());
stream.Close();
}

In case anyone stuck with ssl certificate issue
ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback
(
delegate { return true; }
);
WebRequest request = WebRequest.Create(new Uri("http://.com/flower.zip"));
request.Method = "HEAD";
using (WebResponse response = request.GetResponse())
{
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}