private string DownloadHtml(string address)
{
string result = "https://potnhub.com";
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12;
//ServicePointManager.SecurityProtocol = (SecurityProtocolType)768 | (SecurityProtocolType)3072;
WebRequest wrq = WebRequest.Create(address);
Stream Mystream = wrq.GetResponse().GetResponseStream();
byte[] buffer = new byte[256];
int Readed = 0;
do
{
Readed = Mystream.Read(buffer, 0, buffer.Length);
result += Encoding.ASCII.GetString(buffer, 0, Readed);
} while (Readed!=0);
Mystream.Close();
return result;
}
I want to download HTML code from sites like pornhub or tiktok or youtube
this code is working for normal sites, and it works for youtube too with vpn but for pornhub and tiktok i get the error
The underlying connection was closed: The connection was closed unexpectedly
Screenshot enter image description here
I also use Psiphon VPN and that is because the country that I live in blocks sites like youtube etc.
at first i thought it was because of the vpn so i used other vpn but i got the same result
You can do this very easily using HttpClient
public static async Task<string> DownloadHtml()
{
using var client = new HttpClient();
HttpResponseMessage response = await client.GetAsync("https://www.google.com");
return await response.Content.ReadAsStringAsync();
}
As for why your code isnt working, Id assume its likely due to an invalid URL being used
Related
I use the code below to get an oAuth code using a browser login, it works just fine ie. auth code is returned BUT there is some timing issue between responseOutput.WriteAsync() and http.stop().
When in debug (breakpoint on **** below), then the response is returned to the browser as expected.
If I the comment out the line http.stop() (**** below), then the response is returned to the browser as expected.
BUT If I run the code as usual then the browser shows "page cannot be found", so, it looks as though responseOutput.WriteAsync() is not actually completing (not begin 'awaited').
Do I need to do anything else to ensure that the response is completely sent before stopping the listener?
await GetAuthorizationCode();
public async Task GetAuthorizationCode() {
// Creates an Listener
string redirectUri = "http://127.0.0.1:12345";
HttpListener http = new HttpListener();
http.Prefixes.Add(redirectUri);
http.Start();
// Open auth page in browser
string authUri = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize?........";
var authorizationRequest = authUri;
Process.Start(authorizationRequest);
// Wait for auth response.
HttpListenerContext context = await http.GetContextAsync();
var sCode = context.Request.QueryString.Get("code");
//Send response to the browser.
HttpListenerResponse response = context.Response;
string responseString = string.Format("<html><head></head><body>Auth OK.</body></html>");
var buffer = Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.Length;
using (Stream responseOutput = response.OutputStream)
{
await responseOutput.WriteAsync(buffer, 0, buffer.Length);
responseOutput.Close();
}
//****** this line causes problems
http.Stop();
AuthorizationCode = sCode;
}
It seems that I have to set KeepAlive prior to http.stop()
response.KeepAlive = False
Somehow even with calling response.close and/or with a 'using' block around the response it still needed this setting.
From an old http handler I made some time ago, I have this code:
protected static void WriteString(HttpListenerResponse response, HttpContentType contentType, Encoding encoding, string content)
{
byte[] outputBuffer = encoding.GetBytes(content);
response.ContentType = contentType.Value + "; charset=" + encoding.BodyName;
response.ContentLength64 = outputBuffer.Length;
response.Close(outputBuffer, true);
}
This code have been active on a program that usually stayed days if not weeks serving few thousand requests each day and never had any kind of issues regarding memory leaks.
From docs:
You can customize the response by setting various properties, such as
StatusCode, StatusDescription, and Cookies. Use the
HttpListenerResponse.OutputStream property to obtain a Stream instance
to which response data can be written. Finally, send the response data
to the client by calling the Close method.
https://learn.microsoft.com/en-us/dotnet/api/system.net.httplistenerresponse?view=net-6.0
Oposed to my though, the HttpListener docs states to close the output stream:
https://learn.microsoft.com/en-us/dotnet/api/system.net.httplistener?view=net-6.0
Which is quite confusing:
System.IO.Stream output = response.OutputStream;
output.Write(buffer,0,buffer.Length);
// You must close the output stream.
output.Close();
listener.Stop();
I would suggest to try the follwing code:
using (Stream responseOutput = response.OutputStream)
{
await responseOutput.WriteAsync(buffer, 0, buffer.Length);
//The following two lines are possibly unnecesary in an unsing statement
//responseOutput.Flush(); //Ensure content placed on the right place
//responseOutput.Close();
}
response.Close();
http.Stop();
--- Update ---
I could run the following code under .Net6.0 without any issue and got the response content on the browser:
class Program
{
static async Task Main(string[] args)
{
await RunUntilServeOneRequest();
}
private static async Task RunUntilServeOneRequest()
{
Console.WriteLine("Starting...");
// Creates an Listener
string listenUri = "http://127.0.0.1:12346/";
HttpListener http = new HttpListener();
http.Prefixes.Add(listenUri);
http.Start();
// Wait for request.
Console.WriteLine("Awaiting request to " + listenUri);
HttpListenerContext context = await http.GetContextAsync();
//Send response to the browser.
HttpListenerResponse response = context.Response;
string responseString = string.Format($"<html><head></head><body>Hello world: {DateTime.Now.ToString("yyyy/MM/dd HH:mm:ss")}</body></html>");
byte[] buffer = Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.Length;
using (Stream responseOutput = response.OutputStream)
{
await responseOutput.WriteAsync(buffer, 0, buffer.Length);
responseOutput.Close();
}
response.Close();
Console.WriteLine("Request answered");
http.Stop();
}
}
I try do download a web page using the WebClient, but it hangs until the timeout in WebClient is reached, and then fails with an Exception.
The following code will not work
WebClient client = new WebClient();
string url = "https://www.nasdaq.com/de/symbol/aapl/dividend-history";
string page = client.DownloadString(url);
Using a different URL, the transfer works fine. For example
WebClient client = new WebClient();
string url = "https://www.ariva.de/apple-aktie";
string page = client.DownloadString(url);
completes very quick and has the whole html in the page variable.
Using a HttpClient or WebRequest/WebResponse gives the same result on the first URL: block until timeout exception.
Both URLs load fine in a browser, in roughly 2-5 seconds.
Any idea what the problem is, and what solution is available?
I noticed that when using a WebBrowser control on a Windows Forms dialog, the first URL loads with 20+ javascript errors that need to be confirm-clicked. Same can be observed when developer tools are open in a browser when accessing the first URL.
However, WebClient does NOT act on the return it gets. It does not run the javascript, and does not load referenced pictures, css or other scripts, so this should not be a problem.
Thanks!
Ralf
The first site, "https://www.nasdaq.com/de/symbol/aapl/dividend-history";, requires:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
ServicePointManager.ServerCertificateValidationCallback
A set User-Agent Header
A CookieContainer is, apparently, not required. It should be set anyway.
The User-agent here is important. If a recent User-agent is specified in the WebRequest.UserAgent, the WebSite may activate the Http 2.0 protocol and HSTS (HTTP Strict Transport Security). These are supported/understood only by recent Browsers (as a reference, FireFox 56 or newer).
Using a less recent Browser as User-agent is necessary, otherwise the WebSite will expect (and wait for) a dynamic response. Using an older User-agent, the WebSite will activate the Http 1.1 protocol and never HSTS.
The second site, "https://www.ariva.de/apple-aktie";, requires:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
No Server Certificate validation is required
No specific User-agent is required
I suggest to setup a WebRequest (or a corresponding HttpClient setup) this way:
(WebClient could work, but it'd probably require a derived Custom Control)
private async void button1_Click(object sender, EventArgs e)
{
button1.Enabled = false;
Uri uri = new Uri("https://www.nasdaq.com/de/symbol/aapl/dividend-history");
string destinationFile = "[Some Local File]";
await HTTPDownload(uri, destinationFile);
button1.Enabled = true;
}
CookieContainer httpCookieJar = new CookieContainer();
//The 32bit IE11 header is the User-agent used here
public async Task HTTPDownload(Uri resourceURI, string filePath)
{
// Windows 7 may require to explicitly set the Protocol
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
// Only blindly accept the Server certificates if you know and trust the source
ServicePointManager.ServerCertificateValidationCallback += (s, cert, ch, sec) => { return true; };
ServicePointManager.DefaultConnectionLimit = 50;
var httpRequest = WebRequest.CreateHttp(resourceURI);
try
{
httpRequest.CookieContainer = httpCookieJar;
httpRequest.Timeout = (int)TimeSpan.FromSeconds(15).TotalMilliseconds;
httpRequest.AllowAutoRedirect = true;
httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
httpRequest.ServicePoint.Expect100Continue = false;
httpRequest.UserAgent = "Mozilla / 5.0(Windows NT 6.1; WOW32; Trident / 7.0; rv: 11.0) like Gecko";
httpRequest.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
httpRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8");
httpRequest.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
using (var httpResponse = (HttpWebResponse)await httpRequest.GetResponseAsync())
using (var responseStream = httpResponse.GetResponseStream())
{
if (httpResponse.StatusCode == HttpStatusCode.OK) {
try {
int buffersize = 132072;
using (var fileStream = File.Create(filePath, buffersize, FileOptions.Asynchronous)) {
int read;
byte[] buffer = new byte[buffersize];
while ((read = await responseStream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
await fileStream.WriteAsync(buffer, 0, read);
}
};
}
catch (DirectoryNotFoundException) { /* Log or throw */}
catch (PathTooLongException) { /* Log or throw */}
catch (IOException) { /* Log or throw */}
}
};
}
catch (WebException) { /* Log and message */}
catch (Exception) { /* Log and message */}
}
The first WebSite (nasdaq.com) returned payload length is 101.562 bytes
The second WebSite (www.ariva.de) returned payload length is 56.919 bytes
Obviously there is a problem with downloading that link (incorrect url, unothorized access, ...), however you may use Async Method to solve the socking part:
WebClient client = new WebClient();
client.DownloadStringCompleted += (s, e) =>
{
//here deal with downloaded file
};
client.DownloadStringAsync(url);
When I try to make POST requests in Xamarin the time the function GetRequestStream() takes is 10-20 seconds. The answer time on my server is below 1 second and so is the POST request I made from a website.
I already tried:
Setting proxy server to null so it won't look it up
Using blocks around the requests so it get's flushed
Increased the maximum connections
Did everything with async multithreading
Even tried another class called RestSharp - same result.
Nothing I did was actually helping to reduce the runtime even by 100 milliseconds. I just cannot imagine that this is Xamarins fault because I can't be the only one who decided to do some HTTP requests in his cross platform app. I already lost UWP since the ServiceManager, which I use to connect to TLS sites isn't supported in UWP - thank you Xamarin.
I really need solutions so please help :)
this is the code i used and optimized:
byte[] byteArray = Encoding.UTF8.GetBytes(query);
HttpWebRequest webRequest = new HttpWebRequest(uri);
webRequest.Method = "POST";
webRequest.Proxy = null;
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Timeout = 1000;
webRequest.ReadWriteTimeout = 1000;
Stream dataStream = await webRequest.GetRequestStreamAsync();
await dataStream.WriteAsync(byteArray, 0, byteArray.Length);
dataStream.Close();
WebResponse webResponse = await webRequest.GetResponseAsync();
using (StreamReader reader = new StreamReader(webResponse.GetResponseStream(), Encoding.UTF8))
{
string ret = await reader.ReadToEndAsync();
return ret;
}
and this is the code i tried with the ModernHttpClient
System.Net.Http.HttpClient client = new System.Net.Http.HttpClient(new NativeMessageHandler());
Dictionary<string, string> values = new Dictionary<string, string>();
string[] q = query.Split('&');
for (int i = 0; i < q.Length; i++)
values.Add(q[i].Split('=')[0], q[i].Split('=')[1]);
FormUrlEncodedContent content = new FormUrlEncodedContent(values);
HttpResponseMessage response;
try
{
response = await client.PostAsync(uri, content);
string answer = await response.Content.ReadAsStringAsync();
return answer;
}
catch (Exception e)
{
return "";
}
and of course i added these lines before calling the whole network stuff
ServicePointManager.ServerCertificateValidationCallback += (a, b, c, d) => { return true; };
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 100;
It's definitely a Xamarin or an Android Problem. I don't have other devices to test it,but... what can i do now? I tested it on a .Net console application, so it's definitely not .Net related.
I just found the answer!
it speeds up when i set the Protocol Version. It's mentione basically nowhere but i googled for android post slowdowns and there i found it, finally.
setting the protocol version to HTTP 1.1 speeds things up very fast!
now i just have to fix the boot delay with xamarin >.<
Thank you so much for you help!
Try implement ModernHttpClientPlugin into your code.
Just add this line when initializing new HttpClient, and everything should run smoother.
var httpClient = new HttpClient(new NativeMessageHandler());
Another idea is to try your implementation in new blank console aplication. Then you will see if this is problem with your server/client or Xamarin itself. It should look something like this.
class Program
{
static void Main(string[]args)
{
MainAsync(null);
Console.ReadKey();
}
static async System.Threading.Tasks.Task MainAsync(string[] args)
{
System.Net.Http.HttpClient client = new System.Net.Http.HttpClient();
Dictionary<string, string> values = new Dictionary<string, string>();
var query = "user=mickey&passwd=mini";
string[] q = query.Split('&');
for (int i = 0; i < q.Length; i++)
values.Add(q[i].Split('=')[0], q[i].Split('=')[1]);
FormUrlEncodedContent content = new FormUrlEncodedContent(values);
HttpResponseMessage response;
try
{
response = await client.PostAsync(new Uri("https://www.example.com/login.php"), content);
string answer = await response.Content.ReadAsStringAsync();
}
catch (Exception e)
{
}
}
}
Please, give us feedback how fast that goes.
In my case 'WebClient' and 'HttpWebReuest' cause first-time delays about 20 secs (possible, during resolve of new hosts) on some specific Android 7 devices, but works good on others.
I tried different ways to solve this problem but for me only replacement to 'HttpClient' solves it.
I have a WPF application and that calling 4 web services (Written with Java) on the same base URL and it was working perfectly until I install google chrome. I installed chrome and I've got this error:
Could not establish secure channel for SSL/TLS C# Web service
I didn't write another code. That happened for just I installed chrome then I remove chrome but didn't work and I tried system restore, uninstall eset smart security and I cleaned all windows(8.1 single languages) certificate. So how can I figure it out? Here is my web service caller
public string call(string url, string json)
{
try
{
var webrequest = (HttpWebRequest)WebRequest.Create(url);
var key = JsonConvert.SerializeObject(LoginService.SessionData.SessionKey);
UTF8Encoding uTF8Encoding = new UTF8Encoding();
byte[] requestBytes = uTF8Encoding.GetBytes(json);
WebClient client = new WebClient();
webrequest.Method = "POST";
webrequest.Headers.Add("SESSION_KEY", LoginService.SessionData.SessionKey);
webrequest.ContentType = "application/json";
webrequest.ContentLength = requestBytes.LongLength;
Stream requestStream = webrequest.GetRequestStream();//here the exception
requestStream.Write(requestBytes, 0, requestBytes.Length);
using (var response = webrequest.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
var responseBuf = reader.ReadToEnd();
String responseJson = Convert.ToString(responseBuf);
return responseJson;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
return null;
}
By my observation there are a large number of services over the last several months that are turning off SSL and/or older TLS versions to mitigate security problems inherent in them.
Anywhere within the AppDomain you can just force the connection to use TLS 1.2 like this:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
You can also OR together multiple versions if you need to support older ones as well:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12
I have a piece of code written in C#.Net which reads from a database, collects records, builds a SOAP envelope around them and sends them to the following URL:
https://nodeD1.production.webservices.amadeus.com/SECRETNUMBER
I am able to copy my SOAP envelope string outputted from my code and paste it into SOAPUI, and it works flawlessly. But when I try to send it to the URL through my code, I always get:
The remote server returned an error: (500) Internal Server Error.
My entire SOAP xml request (including the soap header and everything else) is a single string by the time my code is done building it, and I have tried sending to the URL using the following code:
ServicePointManager.ServerCertificateValidationCallback = ValidateServerCertficate;
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
string httpResponseString = "";
webRequest.Proxy = null; //This will result in a quicker post.
byte[] requestBytes = null;
byte[] responseBytes = null;
requestBytes = System.Text.Encoding.UTF8.GetBytes(xmlRequestString);
//Set HttpWebRequest properties
webRequest.Method = "POST";
webRequest.ContentLength = requestBytes.Length;
webRequest.ContentType = "application/soap+xml;charset=UTF-8;";
//webRequest.Headers.Add("SOAPAction: http://webservices.amadeus.com/Profile_PartialUpdateNGProfile_11.1");
//webRequest.ContentType = "text/xml";
using (Stream requestStream = webRequest.GetRequestStream())
{
requestStream.Write(requestBytes, 0, requestBytes.Length);
}
try
{
using (var response = (HttpWebResponse)webRequest.GetResponse())
{
Stream stream = response.GetResponseStream();
using (MemoryStream ms = new MemoryStream())
{
int count = 0;
do
{
byte[] buf = new byte[1024];
count = stream.Read(buf, 0, 1024);
ms.Write(buf, 0, count);
}
while (stream.CanRead && count > 0);
responseBytes = ms.ToArray();
}
}
httpResponseString = System.Text.Encoding.Default.GetString(responseBytes);
}
catch (Exception e)
{
return e.Message;
}
return httpResponseString;
Am I forgetting something obvious? I have been stuck on this problem for a couple days now and I'm not sure what it could be. I have tried commenting and uncommenting the line:
webRequest.Headers.Add("SOAPAction: http://webservices.amadeus.com/Profile_PartialUpdateNGProfile_11.1");
But no dice.
NOTE: My piece of code is an asp.net web service (so I am a web service trying to communicate with another web service), could it be something in my Web.config file that's causing this?
Looks like all I had to do in my case was change my soap action in my soap envelope from:
https://nodeD1.production.webservices.amadeus.com/SECRETNUMBER
to:
https://noded1.production.webservices.amadeus.com/SERCRETNUMBER
Thanks to Fiddler I was able to see the following message:
<faultstring>A header representing a Message Addressing Property is not valid and the message cannot be processed</faultstring>
I Googled the error message and I found a StackOverflow post about it:
Error in Amadeus Service: A header representing a Message Addressing Property is not valid and the message cannot be processed
Apparently .net doesn't like capital letters in URLs in SOAP envelopes. The thing is I thought this might be the case, but I tried with lowercase and it still failed because my SOAP envelope wasn't properly built.