I'd like to download some data from a forum. The page containing the data is visible only to registered users. Here's an example webpage containing user data;
http://www.bikeforums.net/member.php/227664-StackOverflow
I'd like to get the data using wget or C#. I tried logging in via Firefox, then passing the cookies file (hopefully containing the login information) to wget. That was more of a makeshift hack and not a real solution, but it still failed. How do I do this properly?
I set up an account for testing if that's helpful.
User: StackOverflow
Pass: so123
Using firebug you can easily get the POST data for the login page and use it to create a WebRequest and Login to the Forum.
The Server create the cookies for authentication and we can use this cookies in the next request on the forum page so the server can authenticate the request and return all the data.
Here I've tested a simple console application that achieves this mechanism.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Globalization;
using System.Web;
using System.Net;
using System.Xml;
using System.Xml.XPath;
using System.Xml.Linq;
using System.IO;
namespace ConsoleApplication2
{
class Program
{
static void Main(string[] args)
{
CookieContainer cookieContainer = new CookieContainer();
HttpWebRequest wpost = (HttpWebRequest) HttpWebRequest.Create("http://www.bikeforums.net/login.php?do=login");
wpost.CookieContainer = cookieContainer;
wpost.Method = "POST";
string postData = "do=login&vb_login_md5password=d93bd4ce1af6a9deccaf0ea844d6c05d&vb_login_md5password_utf=d93bd4ce1af6a9deccaf0ea844d6c05d&s=&securitytoken=guest&url=%2Fmember.php%2F227664-StackOverflow&vb_login_username=StackOverflow&vb_login_password=";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Set the ContentType property of the WebRequest.
wpost.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
wpost.ContentLength = byteArray.Length;
// Get the request stream.
System.IO.Stream dataStream = wpost.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
HttpWebResponse response = (HttpWebResponse) wpost.GetResponse();
// Request
wpost = (HttpWebRequest)WebRequest.Create("http://www.bikeforums.net/member.php/227664-StackOverflow");
//Assing the cookies created on the server to the new request
wpost.CookieContainer = cookieContainer;
wpost.Method = "GET";
response = (HttpWebResponse)wpost.GetResponse();
Stream receiveStream = response.GetResponseStream();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
//Display the result to console...
Console.WriteLine(readStream.ReadToEnd());
response.Close();
readStream.Close();
Console.Read();
}
}
}
Related
I am new to the C#, I need to transfer .xlx file from specified location to FTP path (\ServerHostName\ExtractedFile). using C# code, Could you please help me
The MSDN page for FtpWebRequest contains a few examples dealing with FTP in C# & .NET. One of the examples is exactly what you would like to do, uploading a file. This example is asynchronous.
https://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest(v=vs.110).aspx
On another page, there is a simpler example:
https://msdn.microsoft.com/en-us/library/ms229715(v=vs.110).aspx
using System;
using System.IO;
using System.Net;
using System.Text;
namespace Examples.System.Net
{
public class WebRequestGetExample
{
public static void Main ()
{
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://www.contoso.com/test.htm");
request.Method = WebRequestMethods.Ftp.UploadFile;
// This example assumes the FTP site uses anonymous logon.
request.Credentials = new NetworkCredential ("anonymous","janeDoe#contoso.com");
// Copy the contents of the file to the request stream.
StreamReader sourceStream = new StreamReader("testfile.txt");
byte [] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
sourceStream.Close();
request.ContentLength = fileContents.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Console.WriteLine("Upload File Complete, status {0}", response.StatusDescription);
response.Close();
}
}
}
}
I have a web service in my QA box that accepts and process an XML string, send an email and then returns another XML string. To test it, I want to overload it by making a console app to call it by sending it an XML, making make a loop in the console app so it sends it 1,000 times in one shoot. I seen a lot of sample in the internet but I can't get them to work.
This is what I have:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Linq;
namespace WSStressTest
{
class Program
{
static void Main()
{
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "<?xml version='1.0'?><ROOT><SOMETHING></SOMETHING></ROOT>";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://url:1111/WebService.asmx/ProcessWebService");
request.Method = "POST";
byte[] data = encoding.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(data, 0, data.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
Console.WriteLine(responseFromServer);
reader.Close();
dataStream.Close();
response.Close();
//Console.ReadLine();
}
}
}
I always get an 500 error in HttpWebResponse. Now, looking at the code the only thing I can think that is missing is the name of the parameter for POST. What I mean, if I make a form
<form method="post" action="http://url:1111/WebService.asmx/ProcessWebService">
<input type="text" name="sXML" /> <input type="submit" value="submit" />
</form>
placing the xml in the input and submit will work. But I'm naming the data post to the web service sXML. How do I name the xml post in the console app sXML? Is that what I'm missing or is there another error in my code?
Additional question: What would be the best way to get this to run 1,000 times. I was thinking of putting the call inside a while <1000. Is there a better way for it?
Thanks!
I am looking for a sample code to send request to a web service( SOAP) using C# .net
Get the response back and then validate the response automatically
Ex:http://www.w3schools.com/webservices/tempconvert.asmx?op=CelsiusToFahrenheit
Send celcius data to this web service and get some response back and then validate this response with expected values.
I am looking to do this programmatically using C# nunit framework
Does anyone have idea how to achieve this?
There should be no logic worth testing in the web service itself.
You cannot easily test the web service call, and nor should you. This is the responsibility of the Framework.
Your web service should be a wrapper around the class that does the real work. You would then unit test the class with the logic in, not the web service.
In your example, you should have a class who's responsibility it is to convert Fahrenheit to Celsius. The public method that does this would be covered by your unit tests.
The end point you call to actually perform the conversion (for example, the methods in your .asmx file) would simply call the code above.
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Xml;
using NUnit.Framework;
namespace Examples.System.Net
{
public class WebRequestPostExample
{
public static void Main()
{
String b = "OK";
// Create a request using a URL that can receive a post.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("your end point here");
request.Headers.Add ("SOAPAction", "some soap action here");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = #"<s:Envelope xmlns:s="Som request that you want to send";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentLength = byteArray.Length;
Stream stream = request.GetRequestStream();
stream.Write(byteArray, 0, byteArray.Length);
int a = 0;
stream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Assert.AreEqual(response.StatusCode, b, "Pass");
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
Please assist with the code below.
When I use it on a Pc that doesn't require a proxy and the proxy settings are commented out it works fine but now I have to run the program on a proxy enabled network but it continuously gives me the error:
An unhandled exception of type 'System.Net.WebException' occurred in
System.dll Additional information: The remote server returned an
error: (407) Proxy Authentication Required.
Please note if you try to run the program it wont be able to fetch the data as it needs to be run from a PC whose IP is on the white lists.
Code
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Xml;
using System.Xml.Linq;
using System.Web;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
//System.Net.WebRequest.GetSystemWebProxy();
string urlDemo = "http://www.api.myoptions.com/api";
// ReadCountries();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlDemo);
request.Credentials = new NetworkCredential("matt.luck", "Mambo88e","JOC");
WebProxy aProxy = new WebProxy();
aProxy.Address = new Uri("Http://182.155.1.205:8080",true)
request.Proxy = aProxy;
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = "api_username=usernameapi_password=password";
postData += "&MODULE=Customer&COMMAND=view&FILTER[id]=1";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length; request.Timeout = 60000;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
// Get the response.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// Display the status.
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
// Display the content.
Console.WriteLine(responseFromServer);
Console.WriteLine("\nClick On Enter To Close Window");
Console.ReadLine();
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
}
}
}
Make sure your proxy details are correct and provide the credentials to the WebProxy object
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlDemo);
request.Credentials = new NetworkCredential("matt.luck", "Mambo88e","JOC");
WebProxy aProxy = new WebProxy();
aProxy.Address = new Uri("Http://182.155.1.205:8080",true)
aProxy.Credentials = new NetworkCredential("matt.luck", "Mambo88e","JOC");
Related: how-do-i-use-webrequest-to-access-an-ssl-encrypted-site-using-https
How to send an HTTPS GET Request in C#?
Add ?var1=data1&var2=data2 to the end of url to submit values to the page via GET:
using System.Net;
using System.IO;
string url = "https://www.example.com/scriptname.php?var1=hello";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
Simple Get Request using HttpClient Class
using System.Net.Http;
class Program
{
static void Main(string[] args)
{
HttpClient httpClient = new HttpClient();
var result = httpClient.GetAsync("https://www.google.com").Result;
}
}
I prefer to use WebClient, it seems to handle SSL transparently:
http://msdn.microsoft.com/en-us/library/system.net.webclient.aspx
Some troubleshooting help here:
https://clipperhouse.com/webclient-fiddler-and-ssl/