We Keep Coding

running a c# .net Console Application multiple time with different variables

I'm trying to automate a task. I have to repeat the task 14 times and want my application to do it for me. my application runs once without a problem. Can anyone explain how I do this without duplicating code?
My application scrapes html from a url and then saves it to a location on my computer. (the html is an email signature). I need the application to do this for 14 different people.
Example.
//person1
string urlAddress1 = "http://www.url.com/person1";
filestream 1 = #"C:\Users\ellio\Desktop\test\person1.htm
//person2
string urlAddress2 = "http://www.url.com/person2";
filestream 2 = #"C:\Users\ellio\Desktop\test\person2.htm
etc
using System;
using System.Net;
using System.Text;
using System.IO;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
namespace GetSignatureHtml
{
class Program
{
static void Main(string[] args)
{
string urlAddress = "http://www.url.com/person1";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (response.CharacterSet == null)
{
readStream = new StreamReader(receiveStream);
}
else
{
readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
}
string data = readStream.ReadToEnd();
using (FileStream fs = new FileStream(#"C:\Users\ellio\Desktop\test\person1.htm", FileMode.Create))
{
using (StreamWriter w = new StreamWriter(fs, Encoding.UTF8))
{
w.WriteLine(data);
}
}
Console.Write(data);
Console.ReadKey(true);
response.Close();
readStream.Close();
}
}
}
}

I'd suggest passing an argument into the Main method which is a file containing all the URLs to fetch. You can then read all those URLs into a list or array (e.g. with File.ReadAllLines, and infer the output file from the URL. Move most of your current code into a method that just accepts the URL, so the code will look something like this:
class Program
{
static void Main(string[] args)
{
if (args.Length != 1)
{
// Display some error message here
return;
}
string[] urls = File.ReadAllLines(args[0]);
foreach (var url in urls)
{
DownloadUrl(url);
}
}
private static void DownloadUrl(string url)
{
// Put most of your current code in here.
// You need to infer the name of the file to save -
// consider using new Uri(url), then Uri.LocalPath
}
}
Then you just need to put the URLs into a text file, and specify that when you run the code.
Note that this could all be done with just one URL per command line argument, but by the time you've got 14 URLs on a command line, it's going to be a bit painful to check. I tend to find that using a file for the data makes life easier.
As asides on the rest of the code, you should use a using statement for the response, and I'd encourage you to use File.WriteAllText as a simpler way of creating the file. You might want to look at using HttpClient as an alternative approach to the whole thing, although I'm not sure whether it applies the response encoding in the same way that you do.

Move your work into a method, then iterate over that method.
using System;
using System.Net;
using System.Text;
using System.IO;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
namespace GetSignatureHtml
{
class Program
{
static void Main(string[] args)
{
foreach(arg in args)
{
FindData(arg.urlAddress);
}
}
public void FindData(arg)
{
string urlAddress = "http://www.url.com/person1";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (response.CharacterSet == null)
{
readStream = new StreamReader(receiveStream);
}
else
{
readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
}
string data = readStream.ReadToEnd();
using (FileStream fs = new FileStream(#"C:\Users\ellio\Desktop\test\person1.htm", FileMode.Create))
{
using (StreamWriter w = new StreamWriter(fs, Encoding.UTF8))
{
w.WriteLine(data);
}
}
Console.Write(data);
Console.ReadKey(true);
response.Close();
readStream.Close();
}
}
}
}

Can't get ResponseStream from WebException

I have a desktop client, that communicates with serverside via Http.
When server has some issues with data processing it returns description of an error in JSON in Http response body with proper Http-code (mainly it is HTTP-400).
When i read HTTP-200 response everithing's fine and this code works:
using (var response = await httpRequest.GetResponseAsync(token))
{
using (var reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding("utf-8")))
{
return await reader.ReadToEndAsync();
}
}
But when an error occures and WebException is thrown and caught there is this code:
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
using (var response = (HttpWebResponse) ex.Response)
{
using (var stream = response.GetResponseStream())
{
using (var reader = new StreamReader(stream, Encoding.GetEncoding("utf-8")))
{
var json = reader.ReadToEnd();
}
}
}
}
}
I have already done something to it to maybe make it work, but the next happens:
response.ContentLength is valid (184)
but stream.Length is 0
and after that i can't read json (it's "")
I don't even know where to look, because everything looks like it should work.
What might be the problem?

After a month of almost everyday thinking I've found workaround.
The thing was that WebException.Response.GetResponseStream() returns not exactly the same stream that was obtained during request (can't find link to msdn right now) and by the time we get to catch the exception and read this stream the actual response stream is lost (or something like that, don't really know and was unable to find any info on the net, except looking into CLRCore which is now opensource).
To save the actual response until catching WebException you must set KeepAlive Property on your HttpRequest and voila, you get your response while catching exception.
So the working code looks like that:
try
{
var httpRequest = WebRequest.CreateHttp(Protocol + ServerUrl + ":" + ServerPort + ServerAppName + url);
if (HttpWebRequest.DefaultMaximumErrorResponseLength < int.MaxValue)
HttpWebRequest.DefaultMaximumErrorResponseLength = int.MaxValue;
httpRequest.ContentType = "application/json";
httpRequest.Method = method;
var encoding = Encoding.GetEncoding("utf-8");
if (httpRequest.ServicePoint != null)
{
httpRequest.ServicePoint.ConnectionLeaseTimeout = 5000;
httpRequest.ServicePoint.MaxIdleTime = 5000;
}
//----HERE--
httpRequest.KeepAlive = true;
//----------
using (var response = await httpRequest.GetResponseAsync(token))
{
using (var reader = new StreamReader(response.GetResponseStream(), encoding))
{
return await reader.ReadToEndAsync();
}
}
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
using (var response = (HttpWebResponse)ex.Response)
{
using (var stream = response.GetResponseStream())
{
using (var reader = new StreamReader(stream, Encoding.GetEncoding("utf-8")))
{
return reader.ReadToEnd();
//or handle it like you want
}
}
}
}
}
I don't know if it is good to keep all connection alive like that, but since it helped me to read actual responses from server, i think it might help someone, who faced the same problem.
EDIT: Also it is important not to mess with HttpWebRequest.DefaultMaximumErrorResponseLength.

I remember facing similar issue before and there was something related to setting the stream's position. Here is one of my solutions for reading webResponse that worked for me earlier. Please try if similar approach works for you:-
private ResourceResponse readWebResponse(HttpWebRequest webreq)
{
HttpWebRequest.DefaultMaximumErrorResponseLength = 1048576;
HttpWebResponse webresp = null;// = webreq.GetResponse() as HttpWebResponse;
var memStream = new MemoryStream();
Stream webStream;
try
{
webresp = (HttpWebResponse)webreq.GetResponse();
webStream = webresp.GetResponseStream();
byte[] readBuffer = new byte[4096];
int bytesRead;
while ((bytesRead = webStream.Read(readBuffer, 0, readBuffer.Length)) > 0)
memStream.Write(readBuffer, 0, bytesRead);
}
catch (WebException e)
{
var r = e.Response as HttpWebResponse;
webStream = r.GetResponseStream();
memStream = Read(webStream);
var wrongLength = memStream.Length;
}
memStream.Position = 0;
StreamReader sr = new StreamReader(memStream);
string webStreamContent = sr.ReadToEnd();
byte[] responseBuffer = Encoding.UTF8.GetBytes(webStreamContent);
//......
//.......
Hope this helps!

Streamed HTTP with GZIP being buffered by StreamReader?

Struggling to find anyone experiencing a similar issue or anything similar.
I'm currently consuming a stream over http (json) which has a GZip requirement, and I am experiencing a delay from when the data is sent, to when reader.ReadLine() reads it. It has been suggested to me that this could be related to the decoding keeping back data in a buffer?
This is what I have currently, it works fine apart from the delay.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";
request.PreAuthenticate = true;
request.Credentials = new NetworkCredential(username, password);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.ContentType = "application/json";
request.Accept = "application/json";
request.Timeout = 30;
request.BeginGetResponse(AsyncCallback, request);
Then inside the AsyncCallback method I have:
HttpWebRequest request = result.AsyncState as HttpWebRequest;
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
{
while (!reader.EndOfStream)
{
string line = reader.ReadLine();
if (string.IsNullOrWhiteSpace(line)) continue;
Console.WriteLine(line);
}
}
It just sits on reader.Readline() until more data is received, and then even holds back some of that. There are also keep-alive newlines received, these are often are read out all at once when it does decide to read something.
I have tested the stream running side by side with a curl command running, the curl command receives and decompresses the data perfectly fine.
Any insight would be terrific.
Thanks,
Dan
EDIT
Had no luck using the buffer size on streamreader.
new StreamReader(stream, Encoding.UTF8, true, 1)
EDIT
Also had no luck updating to .NET 4.5 and using
request.AllowReadStreamBuffering = false;

Update: This seems to have issues over long periods of time with higher rates of volume, and should only be used on small volume where the buffer is impacting the application's functionality. I have since switched back to a StreamReader.
So this is what I ended up coming up with. This works, without the delay. This does not get buffered by automated GZip decompression.
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
using (MemoryStream memory = new MemoryStream())
using (GZipStream gzip = new GZipStream(memory, CompressionMode.Decompress))
{
byte[] compressedBuffer = new byte[8192];
byte[] uncompressedBuffer = new byte[8192];
List<byte> output = new List<byte>();
while (stream.CanRead)
{
int readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);
memory.Write(compressedBuffer.Take(readCount).ToArray(), 0, readCount);
memory.Position = 0;
int uncompressedLength = gzip.Read(uncompressedBuffer, 0, uncompressedBuffer.Length);
output.AddRange(uncompressedBuffer.Take(uncompressedLength));
if (!output.Contains(0x0A)) continue;
byte[] bytesToDecode = output.Take(output.LastIndexOf(0x0A) + 1).ToArray();
string outputString = Encoding.UTF8.GetString(bytesToDecode);
output.RemoveRange(0, bytesToDecode.Length);
string[] lines = outputString.Split(new[] { Environment.NewLine }, new StringSplitOptions());
for (int i = 0; i < (lines.Length - 1); i++)
{
Console.WriteLine(lines[i]);
}
memory.SetLength(0);
}
}

There may be something to the Delayed ACK C.Evenhuis discusses, but I've got a weird gut feeling it's the StreamReader that's causing you headaches...you might try something like this:
public void AsyncCallback(IAsyncResult result)
{
HttpWebRequest request = result.AsyncState as HttpWebRequest;
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
{
var buffer = new byte[2048];
while(stream.CanRead)
{
var readCount = stream.Read(buffer, 0, buffer.Length);
var line = Encoding.UTF8.GetString(buffer.Take(readCount).ToArray());
Console.WriteLine(line);
}
}
}
EDIT: Here's the full harness I used to test this theory (maybe the difference from your situation will jump out at you)
(LINQPad-ready)
void Main()
{
Task.Factory.StartNew(() => Listener());
_blocker.WaitOne();
Request();
}
public bool _running;
public ManualResetEvent _blocker = new ManualResetEvent(false);
public void Listener()
{
var listener = new HttpListener();
listener.Prefixes.Add("http://localhost:8080/");
listener.Start();
"Listener is listening...".Dump();;
_running = true;
_blocker.Set();
var ctx = listener.GetContext();
"Listener got context".Dump();
ctx.Response.KeepAlive = true;
ctx.Response.ContentType = "application/json";
var outputStream = ctx.Response.OutputStream;
using(var zipStream = new GZipStream(outputStream, CompressionMode.Compress))
using(var writer = new StreamWriter(outputStream))
{
var lineCount = 0;
while(_running && lineCount++ < 10)
{
writer.WriteLine("{ \"foo\": \"bar\"}");
"Listener wrote line, taking a nap...".Dump();
writer.Flush();
Thread.Sleep(1000);
}
}
listener.Stop();
}
public void Request()
{
var endPoint = "http://localhost:8080";
var username = "";
var password = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";
request.PreAuthenticate = true;
request.Credentials = new NetworkCredential(username, password);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.ContentType = "application/json";
request.Accept = "application/json";
request.Timeout = 30;
request.BeginGetResponse(AsyncCallback, request);
}
public void AsyncCallback(IAsyncResult result)
{
Console.WriteLine("In AsyncCallback");
HttpWebRequest request = result.AsyncState as HttpWebRequest;
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
{
while(stream.CanRead)
{
var buffer = new byte[2048];
var readCount = stream.Read(buffer, 0, buffer.Length);
var line = Encoding.UTF8.GetString(buffer.Take(readCount).ToArray());
Console.WriteLine("Reader got:" + line);
}
}
}
Output:
Listener is listening...
Listener got context
Listener wrote line, taking a nap...
In AsyncCallback
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}
Listener wrote line, taking a nap...
Reader got:{ "foo": "bar"}

This may have to do with Delayed ACK in combination with Nagle's algorithm. It occurs when the server sends multiple small responses in a row.
On the server side, the first response is sent, but subsequent response data chunks are only sent when the server has received an ACK from the client, or until there is enough data for a big packet to send (Nagle's algorithm).
On the client side, the first bit of response is received, but the ACK is not sent immediately - since traditional applications have a request-response-request-response behavior, it assumes it can send the ACK along with the next request - which in your case does not happen.
After a fixed amount of time (500ms?) it decides to send the ACK anyway, causing the server to send the next packages it has accumulated sofar.
The problem (if this is indeed the problem you're experiencing) can be fixed on the server side at the socket level by setting the NoDelay property, disabling Nagle's algorithm. I think you can also disable it operating system wide.
You could also temporarily disable Delayed ACK (I know windows has a registry entry for it) on the client side to see if this is indeed the problem, without having to change anything on your server. Delayed ACK prevents DDOS attacks, so make sure you restore the setting afterwards.
Sending keepalives less frequently may also help, but you'll still have a chance for the problem to occur.

Is there a way to read from a website, one line at a time?

I have this code:
string downloadedString;
System.Net.WebClient client;
client = new System.Net.WebClient();
downloadedString = client.DownloadString(
"http://thebnet.x10.mx/HWID/BaseHWID/AlloweHwids.txt");
It's a HWID-type security (it will check your HWID to see if you are allowed to use the program)
Anyway, I want to be able to put multiple lines on it at a time, example:
xjh94jsl <-- Not a real HWID
t92jfgds <-- Also not real
And be able to read each line, one by one, and update it to downloadedString.

Don't download the url as a string, read it as a stream.
using System.IO;
using System.Net;
var url ="http://thebnet.x10.mx/HWID/BaseHWID/AlloweHwids.txt";
var client = new WebClient();
using (var stream = client.OpenRead(url))
using (var reader = new StreamReader(stream))
{
string line;
while ((line = reader.ReadLine()) != null)
{
// do stuff
}
}

Never ending Stream in HttpWebResponse

how can i read some bytes and disconnect? i use such code
using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
{
using (Stream sm = resp.GetResponseStream())
{
using (StreamReader sr = new StreamReader(sm, Encoding.Default))
{
sr.Read();
sr.Close();
}
}
}
but it wait for end of stream

You probably don't want to use a StreamReader to read a WebResonse stream unless you know for sure that the stream contains newlines. StreamReader likes to think in terms of lines, and if there aren't any newlines in the stream, it's going to hang.
Your best bet is to read as many bytes as you want into a byte[] buffer, and then convert that to text. For example:
int BYTES_TO_READ = 1000;
var buffer = new byte[BYTES_TO_READ];
using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
{
using (Stream sm = resp.GetResponseStream())
{
int totalBytesRead = 0;
int bytesRead;
do
{
// You have to do this in a loop because there's no guarantee that
// all the bytes you need will be ready when you call.
bytesRead = sm.Read(buffer, totalBytesRead, BYTES_TO_READ-totalBytesRead);
totalBytesRead += bytesRead;
} while (totalBytesRead < BYTES_TO_READ);
// Sometimes WebResponse will hang if you try to close before
// you've read the entire stream. So you can abort the request.
request.Abort();
}
}
At this point, the buffer has the first BYTES_TO_READ bytes from the buffer. You can then convert that to a string, like this:
string s = Encoding.Default.GetString(buffer);
Or you can open a MemoryStream on the buffer if you want to use StreamReader.
I have run into WebResponse hanging sometimes if you don't read everything. I don't know why it does that, and I can't reliably reproduce it, but I've found that if I do request.Abort() before closing the stream, everything works. See
On a side note, the word you want is "unresponsive" rather than "unresponsible."

Could you do something like this?
string GetWebPageContent(string url)
{
string result = string.Empty;
HttpWebRequest request;
const int bytesToGet = 1000;
request = WebRequest.Create(url) as HttpWebRequest;
//get first 1000 bytes
request.AddRange(0, bytesToGet - 1);
using (WebResponse response = request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
}
}
return result;
}
The key is using AddRange in your request.

It is beacuse http 1.1 connection is persistent connection default and the tcp connection has not been closed,so the stream dose not receive the end.
you can use myHttpWebRequest1.KeepAlive=false;
so the tcp connection will close after the http reponse.
https://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.connection(v=vs.110).aspx#

If you talking winforms or webforms I would put the request into a threadpool (or Task if you are using .net 4). Streams, even with a good handling, are too easy to put GUI in a wait-state that dislikes by most users.

I had similar never-ending request parsing with examples listed here. The following is what I came up with to eliminate those issues:
// url is a string where the request is going.
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
// Do what you have to do with the Request.
StringBuilder builder = new StringBuilder();
int toRead = 1000;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
HttpStatusCode status = response.StatusCode;
using (Stream receiveStream = response.GetResponseStream())
{
using (StreamReader readStream = new StreamReader(receiveStream, Encoding.GetEncoding("utf-8")))
{
Char[] read = new Char[toRead];
int count = readStream.Read(read, 0, toRead);
while (count > 0)
{
string str = new String(read, 0, count);
builder.Append(str);
count = readStream.Read(read, 0, toRead);
}
readStream.Close();
}
}
response.Close();
}
return builder.ToString();