I have a problem in my application, written in c# using WatiN.
The application creates few threads,and each thread open browser and the same page.
The page consist of HTML select element: and a submit button.
The browsers should select a specific option and click on the submit button at the same time but instead they do it "one by one".
Here is the main code lines:
[STAThread]
static void Main(string[] args)
{
for (int i = 0; i < numOfThreads;i++ )
{
var t = new Thread(() => RealStart(urls[i]));
t.SetApartmentState(ApartmentState.STA);
t.IsBackground = true;
t.Start();
}
}
private static void RealStart(string url)
{
using (var firstBrowser = new IE())
{
firstBrowser.GoTo(url);
firstBrowser.BringToFront();
OptionCollection options = firstBrowser.SelectList("Select").Options;
options[1].Select();
firstBrowser.Button(Find.ByName("Button")).Click();
firstBrowser.Close();
}
}
What is the cause of the "one by one" selection instead of simultaneously selection?
Solution:
After a long research I gave up using WatiN for this porpuse.
Instead, I have created HttpWebRequest and post it to the specific URL.
Works Like a charm:
HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create("http://domain.com/page.aspx");
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "username=user";
postData += "&password=pass";
byte[] data = encoding.GetBytes(postData);
httpWReq.Method = "POST";
httpWReq.ContentType = "application/x-www-form-urlencoded";
httpWReq.ContentLength = data.Length;
using (Stream stream = httpWReq.GetRequestStream())
{
stream.Write(data,0,data.Length);
}
HttpWebResponse response = (HttpWebResponse)httpWReq.GetResponse();
string responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
I send those requests simultaneously, by creating a Thread for each request.
Related
I'm trying to login to www.autoscout24.de and retrieve adds and messages. Login form has a random generated hidden input/token. Being new to C#, I've read different tuts about using C# to login to websites and all I found was simple codes that work only in simple login forms (user:pass). I've imagined a 2-step approach: first make a GET request to retrieve needed data and a POST request with login credentials and other needed imputes. Using HtmlAgilityPack I'm passed first step but the second request just returns the login page again instead of "My account" page.
My code:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace WebRequest__custom
{
class Program
{
static void Main(string[] args)
{
CookieContainer _cookies;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://angebot.autoscout24.de/login?fromNavi=myAS24");
WebResponse _response = request.GetResponse();
Stream stream = _response.GetResponseStream();
StreamReader strReader = new StreamReader(stream);
string _cookiesHeader = _response.Headers["Set-cookie"];
_cookies = request.CookieContainer;
string _content = strReader.ReadToEnd();
//Console.WriteLine(_content.Substring(0,500));
var _dom = new HtmlAgilityPack.HtmlDocument();
_dom.LoadHtml(_content);
// Get POST link
var _postLinkNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form");
var postLink = _postLinkNode.Attributes["action"].Value;
//Console.WriteLine(postLink);
//get Token
var _tokenNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form/input");
var token = _tokenNode.Attributes["value"].Value;
//Console.WriteLine(token);
// Start login request
HttpWebRequest requestLogin = (HttpWebRequest)WebRequest.Create("https://accounts.autoscout24.com"+ postLink);
requestLogin.ContentType = "application/x-www-form-urlencoded";
requestLogin.Method = "POST";
requestLogin.KeepAlive = true;
requestLogin.AllowAutoRedirect = true;
string postData = "&__RequestVerificationToken=" + token;
postData += "&Username=web-cppxt#mail-tester.com";
postData += "&Password=Qwert123!";
postData += "&RememberMeCheckBox=on&RememberMe=true";
byte[] _bytes = Encoding.UTF8.GetBytes(postData);
requestLogin.ContentLength = _bytes.Length;
requestLogin.CookieContainer = _cookies;
using(Stream sr = requestLogin.GetRequestStream())
{
sr.Write(_bytes, 0, _bytes.Length);
}
WebResponse loginResponse = requestLogin.GetResponse();
StreamReader loginStreamReader = new StreamReader(loginResponse.GetResponseStream());
string secondPage = loginStreamReader.ReadToEnd();
Console.WriteLine(secondPage.Substring(0,500));
Console.ReadKey();
}
}
}
I have been assigned to take over someones position, however I do not really know C#. There is a server (192.268. something ect) that will post data to a site (unknown site, lets say bleh.com)
This is what the posting code snippet looks like:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(
"https://server-ip-here/postlistener?filename=filename.zlib");
req.UseNagleAlgorithm = true;
req.AllowWriteStreamBuffering = true;
req.Method = "POST";
req.Accept = "application/xml";
req.ServicePoint.Expect100Continue = false;
System.Net.ServicePointManager.CheckCertificateRevocationList = false;
req.Proxy = new WebProxy();
filename = "filename.dat";
byte[] postData = File.ReadAllBytes(filename);
Stream stream = req.GetRequestStream();
stream.Write(postData, 0, postData.Length);
stream.Flush();
stream.Close();
req.BeginGetResponse(new AsyncCallback(responseHandler), this);
Which I beleive I get the post request in this form
www.blah.com/upload?filename=file_1234_12389126495129847980.zlib
I am unsure how to listen for post requests and then get the data from them and save them as a file.
Currently I have tried this:
private void Form1_Load(object sender, EventArgs e)
{
listener = new HttpListener();
// listener.Prefixes.Add("http://localhost:8000/");
listener.Prefixes.Add("http://127.0.0.1:8000/");
listener.AuthenticationSchemes = AuthenticationSchemes.Anonymous;
listener.Start();
listenThread1 = new Thread(new ParameterizedThreadStart(startlistener));
listenThread1.Start();
}
private void startlistener(object s)
{
while (true)
{
// blocks until a client has connected to the server
ProcessRequest();
}
}
private void ProcessRequest()
{
var result = listener.BeginGetContext(ListenerCallback, listener);
result.AsyncWaitHandle.WaitOne();
}
private void ListenerCallback(IAsyncResult result)
{
var context = listener.EndGetContext(result);
Thread.Sleep(1000);
var data_text = new StreamReader(
context.Request.InputStream,
context.Request.ContentEncoding).ReadToEnd();
var cleaned_data = System.Web.HttpUtility.UrlDecode(data_text);
context.Response.StatusCode = 200;
context.Response.StatusDescription = "OK";
MessageBox.Show(cleaned_data);
context.Response.Close();
}
Which listens on the local host (Would sub local host for the website once we establish what it will be).
Not sure how to grab the post though, right now I can only listen for it. Ideally I would only like to accept posts from a specific IP address also.
Any ideas how I can go about grabbing the post data (which will be binary) and saving it as a file?
http://www.conquerclub.com/game.php?game=13025037
Please does anyone know how I can programatically log onto this address and then get all the html of the page back as a string Using something like HttpWebRequest/HttpWebResponse.
username - "testuser1"
password - "testing"
(I've created an account on the site with these credentials so it will actually logon)
So far I've got this
private void toolStripButton1_Click(object sender, EventArgs e)
{
var request = WebRequest.Create("http://www.conquerclub.com/game.php?game=13025037");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
richTextBox1.Text = reader.ReadToEnd();
}
This automatically re-directs and returns the html of the login page. How do I pass the username and password to login automatically so it retrieves the game page instead?
I know there are quite a few similar questions on this and I've spent several hours trying different things but can't get anything to work.
Edit:-
Looked into this further and this site uses cookies for logging in.
Tried this code but is still just returning the login page.
private void toolStripButton1_Click(object sender, EventArgs e)
{
string loginUri = "http://www.conquerclub.com/login.php";
string username = "testuser1";
string password = "testing";
string reqString = "username=" + username + "&password=" + password;
byte[] requestData = Encoding.UTF8.GetBytes(reqString);
CookieContainer cc = new CookieContainer();
var request = (HttpWebRequest)WebRequest.Create(loginUri);
request.Proxy = null;
request.AllowAutoRedirect = false;
request.CookieContainer = cc;
request.Method = "post";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = requestData.Length;
using (Stream s = request.GetRequestStream())
s.Write(requestData, 0, requestData.Length);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.Cookies != null)
{
foreach (Cookie c in response.Cookies)
Console.WriteLine(c.Name + " = " + c.Value);
}
}
string newloginUri = "http://www.conquerclub.com/game.php?game=13025037";
HttpWebRequest newrequest = (HttpWebRequest)WebRequest.Create(newloginUri);
newrequest.Proxy = null;
newrequest.CookieContainer = cc;
using (HttpWebResponse newresponse = (HttpWebResponse)newrequest.GetResponse())
using (Stream resSteam = newresponse.GetResponseStream())
using (StreamReader sr = new StreamReader(resSteam))
richTextBox1.Text = sr.ReadToEnd();
}
Have found that the code at the bottom to log into the game page can be made to work by first logging in manually while using fiddler on firefox then copy and pasting the cookies and hard coding them into newrequest like this.
string newloginUri = "http://www.conquerclub.com/game.php?game=13025037";
HttpWebRequest newrequest = (HttpWebRequest)WebRequest.Create(newloginUri);
newrequest.Proxy = null;
newrequest.CookieContainer = new CookieContainer();
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("PHPSESSID","86bte1ipainiq760vm2flv4h13"));
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("phpbb3_jer7c_u", "648113"));
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("phpbb3_jer7c_k", ""));
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("phpbb3_jer7c_sid", "3eebb0771a68c4a58581e495c34b2c93"));
using (HttpWebResponse newresponse = (HttpWebResponse)newrequest.GetResponse())
using (Stream resSteam = newresponse.GetResponseStream())
using (StreamReader sr = new StreamReader(resSteam))
richTextBox1.Text = sr.ReadToEnd();
This returns the game page as I want it to but can't figure out how to get the login working so that it will return the right cookies. On debugging the code the cookies that come back are completely different to the ones I'm seeing in fiddler so looks like it's just not logging in.
The code that you've written does a GET of the URL given; however, to retrieve the content of the page when you log in you need to pretend that your WebRequest is actually filling in the form, by passing in all the form variables and issuing a POST request instead.
http://msdn.microsoft.com/en-us/library/debx8sh9.aspx gives a good walkthrough of the steps you require. The following isn't 100% complete, but should give you a step in the right direction:
var request = WebRequest.Create("http://www.conquerclub.com/login.php");
request.Method = "POST";
var parameters = new Dictionary<string, string> {
{ "username", "testuser1" },
{ "password", "testing" },
{ "redirect", "game.php?game=13025037" },
{ "direct", "63###www.conquerclub.com###" },
{ "protocol", "HTTP" },
{ "submit", "Login" } };
var content = string.Join( "&", ( from p in parameters select p.Key + "=" + HttpUtility.UrlEncode( p.Value) ).ToArray() ); ;
byte[] bytes = new byte[content.Length * sizeof(char)];
System.Buffer.BlockCopy(content.ToCharArray(), 0, bytes, 0, bytes.Length);
request.ContentLength = bytes.Length;
request.ContentType = "application/x-www-form-urlencoded";
Stream dataStream = request.GetRequestStream();
dataStream.Write( bytes, 0, bytes.Length );
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string result = reader.ReadToEnd();
Note that content includes various values that aren't visible on-screen; these can be seen by viewing the form's source, or (easier) by using the 'Network' tab on your browser debug toolbar of choice and monitoring the data being sent when the form is submitted.
You need to mimic the process of logging in. Look at the HTML code for the login page, and find out three things:
The <form> that sends the request, specifically the target (found in the attribute action) of the form.
Find out the method of sending the data (found in the attribute method)
Find the ID of the fields that represent the user name and password.
Having those you can easily construct a WebRequest that mimics a login.
Note, that this assumes a straight forward login screen (no AJAX, no CAPTCHAs).
protected void UploadFile(object sender, EventArgs e)
{
if (fileUpload.HasFile)
{
if (fileUpload.PostedFile.ContentType == "text/xml")
{
Stream inputstream = fileUpload.PostedFile.InputStream;
byte[] streamAsBytes = (ConvertStreamToByteArray(inputstream));
string stringToSend = BitConverter.ToString(streamAsBytes);
xmlstream.Value = stringToSend;
sendXML.Visible = true;
infoLabel.Text = string.Empty;
/*
string path = Server.MapPath("GenericHandler.ashx");
WebClient wc = new WebClient();
wc.UploadFile(path,"POST", fileUpload.PostedFile);
Something like this maybe? But is there any way to do it without saving the file? */
}
else
{
infoLabel.Text = "Please select an XML file";
sendXML.Visible = false;
}
}
}
This is my current code. The xml gets saved in a hidden field as a hex string and sent via jquery ajax. But it would be much better to send the file itself and process it in the handler. Is that possible?
Try the following code, i haven't tested it but it should work, instead of string pass the byte[] to the method
private string PostData(string url, byte[] postData)
{
HttpWebRequest request = null;
Uri uri = new Uri(url);
request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postData.Length;
using (Stream writeStream = request.GetRequestStream())
{
writeStream.Write(postData, 0, postData.Length);
}
string result = string.Empty;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (StreamReader readStream = new StreamReader(responseStream, Encoding.UTF8))
{
result = readStream.ReadToEnd();
}
}
}
return result;
}
Found the code here at Http Post in C#
Yes, you can create a HttpWebRequest, set it's Method to POST (if that's what you need) and then create a form field in the request with your file data. You will need to understand a little bit about how HTTP requests work and how to properly create that form field in the request, but it's doable (and not overly difficult).
Hi guys just wondering if somebody could help me try and correctly thread my application, I am constantly hitting an hurdle after another, I have never been to clued up on threading in applications. I have tryed following this http://www.developerfusion.com/code/4654/asynchronous-httpwebrequest/ tutorial.
basically I'm just trying to stop my request from hanging my application
public class Twitter
{
private const string _username = "****",
_password = "****";
private WebResponse webResp;
public string getTimeLine()
{
Thread thread = new Thread(new ThreadStart(TwitterRequestTimeLine));
thread.IsBackground = true;
thread.Start();
using (Stream responseStream = webResp.GetResponseStream())
{
//
using (StreamReader reader = new StreamReader(responseStream))
{
return reader.ReadToEnd();
}
}
}
private void TwitterRequestTimeLine()
{
string aUrl = "http://168.143.162.116/statuses/home_timeline.xml";
HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create(aUrl);
SetRequestParams(request);
request.Credentials = new NetworkCredential(_username, _password);
//WebResponse tempResp = request.GetResponse();
ThreadState state = new ThreadState();
IAsyncResult result = request.BeginGetResponse(new AsyncCallback(???), ???);
}
private static void SetRequestParams( HttpWebRequest request )
{
request.Timeout = 500000;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "AdverTwitment";
}
}
}
anyone help would be greatly appricated
You really don't need to thread HttpWebRequest.
When you use BeginGetResponse() and EndGetResponse() with HttpWebRequest, it already uses a background thread for you in order to work asynchronously. There is no reason to push this into a background thread.
As for usage: The help for HttpWebRequest.BeginGetResponse demonstrates a complete, asynchronous request.
If this is a WinForms app, the easiest way to keep the GUI responsive while executing the WebRequest is to use a BackgroundWorker component. Drop a BackgroundWorker on your form and call its RunWorkAsync() method. Put the code to execute the WebRequest and read the Response in the DoWork event handler.
Try using an AsyncCallback like Rubens suggested but have the callback call into a separate method to load the data to its destination. If the getTimeline method doesn't return immediately it will cause the application to hang, because the UI Thread is what is running the request itself.
If you use a separate AsyncCallback to be called after the request is done and have it load the data then the method will return immediately and your UI thread can do other things while it waits.
What about this:
private string getTimeLine()
{
string responseValue = "";
string aUrl = "http://168.143.162.116/statuses/home_timeline.xml";
AutoResetEvent syncRequest = new AutoResetEvent(false);
WebRequest request = WebRequest.Create(aUrl);
request.Method = "POST";
request.BeginGetResponse(getResponseResult =>
{
HttpWebResponse response =
(HttpWebResponse)request.EndGetResponse(getResponseResult);
using (StreamReader reader =
new StreamReader(response.GetResponseStream()))
{
responseValue = reader.ReadToEnd();
}
syncRequest.Set();
}, null);
syncRequest.WaitOne();
return responseValue;
}
EDIT: Ok, I tried to keep a method returning a string, that's why I used AutoResetEvent; If you use a BackgroundWorker, you'll get notified when your data is available:
BackgroundWorker worker = new BackgroundWorker();
string responseValue = "";
worker.RunWorkerCompleted += (sender, e) =>
{
// update interface using responseValue variable
};
worker.DoWork += (sender, e) =>
{
string aUrl = "http://168.143.162.116/statuses/home_timeline.xml";
WebRequest request = WebRequest.Create(aUrl);
// .. setup
using(StreamReader reader =
new StreamReader(request.GetResponse().GetResponseStream()))
responseValue = reader.ReadToEnd();
};
worker.RunWorkerAsync();