I'm trying to read the response i get from nyc.gov. I used Fiddler to construct the WebRequest and it keeps timing out.
Important: this works if the url is https://www.google.com so it's got to be something from the nyc.gov server. But how can it know the difference between my code and Chrome?
I tried settings the KeepAlive to true/false/none.
I tried using Http1.0
I tried setting request.ServicePoint.Expect100Continue to false
I tried setting request.ContentLength = 0;
I tried enclosing in "using"
I added to app.config
<system.net>
<connectionManagement>
<add address="*" maxconnection="1000" />
</connectionManagement>
</system.net>
Here is my code:
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://a810-bisweb.nyc.gov/bisweb/bispi00.jsp");
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", #"1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9");
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException e)
{
if (e.Status == WebExceptionStatus.ProtocolError) response = (HttpWebResponse)e.Response;
else return false;
}
catch (Exception)
{
if (response != null) response.Close();
return false;
}
Here is the RAW request (provided by Fiddler) from Chrome - WORKS:
GET http://a810-bisweb.nyc.gov/bisweb/bispi00.jsp HTTP/1.1
Host: a810-bisweb.nyc.gov
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
And this is the RAW request from my code - HANGS (and eventually times out)
GET http://a810-bisweb.nyc.gov/bisweb/bispi00.jsp HTTP/1.1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Host: a810-bisweb.nyc.gov
Connection: Keep-Alive
Related
I'm trying to connect to an api, but I get the following result :
StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://example.url
Date: Sat, 30 Jan 2021 22:56:45 GMT
Set-Cookie: TS0182ab0d=0180bb6f22515cbe2cddec42f2bdc8cb4b394bf2447928c095c41f950fab6ce3b59180574be0cf84ba91749969bb6cfafcaf801f7d; Path=/; Domain=.api2.mofidonline.com
Content-Length: 0
}
and this is my headers
request.Headers.TryAddWithoutValidation("authority", "api2.example.url");
request.Headers.TryAddWithoutValidation("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36");
request.Headers.TryAddWithoutValidation("x-requested-with", "XMLHttpRequest");
string str = "BasicAuthentication" + mCookie["api-token"];
request.Headers.TryAddWithoutValidation("authorization", str);
request.Headers.TryAddWithoutValidation("accept", "*/*");
request.Headers.TryAddWithoutValidation("origin", "https://example.url");
request.Headers.TryAddWithoutValidation("sec-fetch-site", "same-site");
request.Headers.TryAddWithoutValidation("sec-fetch-mode", "cors");
request.Headers.TryAddWithoutValidation("sec-fetch-dest", "empty");
request.Headers.TryAddWithoutValidation("referer", "https://example.url/Home/Default/page-1");
request.Headers.TryAddWithoutValidation("accept-language", "en-US,en;q=0.9");
This is not correct:
string str = "BasicAuthentication" + mCookie["api-token"];
It's "Basic" and then you need a space followed by the username and password joined by a colon, as a Base64 encoded string for the credentials.
For example:
string base64EncodedCredentials = Convert.ToBase64String(Encoding.ASCII.GetBytes("username:password"));
string authorizationHeader = "Basic " + base64EncodedCredentials;
I am currently working on a console app to import data into Joplin for Windows 10, using C# and Flurl.
Joplin's API description can be found here.
I am trying to create a new resource in Joplin for a file on my system, so it can be attached to a Joplin note.
With CURL I can create the resource using command:
curl -F "data=#c:\\temp\\Test.pptx" -F "props={\"title\":\"my resource title\"}" http://localhost:41184/resources?token=MyToken
(note: it only works with "data=#c:\temp\Test.pptx", NOT with "data=c:\temp\Test.pptx")
When I try this with Flurl in c# I get a 400 response from Joplin, in the log I find:
Error: Resource cannot be created without a file
at Api.action_resources (C:\Program Files\Joplin\resources\app.asar\lib\services\rest\Api.js:351:37)
at Api.route (C:\Program Files\Joplin\resources\app.asar\lib\services\rest\Api.js:140:42)
at execRequest (C:\Program Files\Joplin\resources\app.asar\lib\ClipperServer.js:157:39)
at C:\Program Files\Joplin\resources\app.asar\lib\ClipperServer.js:185:8
at C:\Program Files\Joplin\resources\app.asar\node_modules\multiparty\index.js:136:9
at C:\Program Files\Joplin\resources\app.asar\node_modules\multiparty\index.js:115:9
at processTicksAndRejections (internal/process/task_queues.js:75:11)"
I have tried this so far:
try
{
var url = BaseUrl
.WithHeader("User_Agent", browserUserAgent)
.AppendPathSegment("resources")
.SetQueryParam("token", Token);
using (var fs = new FileStream("c:\\temp\\Test.pptx", FileMode.Open, FileAccess.Read))
{
var resource = url.PostMultipartAsync(mp => mp
.AddJson("props", new { title = "test title" })
.AddFile("data", fs, "Test.pptx", "application/octet-stream")
)
.ReceiveJson<JoplinResource>()
.Result;
}
}
and:
try
{
var url = BaseUrl
.WithHeader("User_Agent", browserUserAgent)
.AppendPathSegment("resources")
.SetQueryParam("token", Token);
var resource = url.PostMultipartAsync(mp => mp
.AddJson("props", new { title = "test title" })
.AddFile("data", "c:\\temp\\Test.pptx")
)
.ReceiveJson<JoplinResource>()
.Result;
}
I hooked up fiddler to see what is the difference between my application and CURL.
Curl:
POST http://127.0.0.1:41184/resources?token=MyToken HTTP/1.1
Host: 127.0.0.1:41184
User-Agent: curl/7.70.0
Accept: */*
Connection: Keep-Alive
Content-Length: 33648
Content-Type: multipart/form-data; boundary=------------------------91ab181cbb0247ba
--------------------------91ab181cbb0247ba
Content-Disposition: form-data; name="props"
{"title":"my resource title"}
--------------------------91ab181cbb0247ba
Content-Disposition: form-data; name="data"; filename="Test.pptx"
Content-Type: application/octet-stream
...
My Console app:
POST http://localhost:41184/resources?token=MyToken HTTP/1.1
User_Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36
Content-Type: multipart/form-data; boundary="f603841b-5c32-4e77-985a-69c2ffb6eed0"
Host: localhost:41184
Content-Length: 33612
Expect: 100-continue
Accept-Encoding: gzip, deflate
--f603841b-5c32-4e77-985a-69c2ffb6eed0
Content-Disposition: form-data; name=props
{"title":"My Resource"}
--f603841b-5c32-4e77-985a-69c2ffb6eed0
Content-Disposition: form-data; name=data; filename=Test.pptx; filename*=utf-8''Test.pptx
...
NOTE the differences:
props and data are in quotes when using CURL, not with FLURL
FLURL sends a second file name: filename*=utf-8''Test.pptx
How do I get this to work properly?
The issue was in the missing quotes for the "data" and "props":
try
{
var url = BaseUrl
.WithHeader("User_Agent", browserUserAgent)
.AppendPathSegment("resources")
.SetQueryParam("token", Token);
var resource = url.PostMultipartAsync(mp => mp
.AddJson("\"props\"", new { title = "My Resource" })
.AddFile("\"data\"", "c:\\temp\\Test.pptx")
)
.ReceiveJson<JoplinResource>()
.Result;
}
Raw request header is now:
POST http://localhost:41184/resources?token=MyToken HTTP/1.1
User_Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36
Content-Type: multipart/form-data; boundary="c6b2377a-1240-4ae3-872f-fa24b643d3e0"
Host: localhost:41184
Content-Length: 33616
Expect: 100-continue
Accept-Encoding: gzip, deflate
--c6b2377a-1240-4ae3-872f-fa24b643d3e0
Content-Disposition: form-data; name="props"
{"title":"My Resource"}
--c6b2377a-1240-4ae3-872f-fa24b643d3e0
Content-Disposition: form-data; name="data"; filename=Test.pptx; filename*=utf-8''Test.pptx
...
And the Joplin REST service creates a new resource...
I´m trying to develop a web crawler to extract some information from my company's web site, but I'm getting the error as below:
An exception of type System.Net.WebException occurred in System.dll but was not handled in user code
Additional information: The remote server returned an error: (500) Internal Server Error.
Here is the requestHeaders of the website:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:WASReqURL=https://:9446/ProcessPortal/jsp/index.jsp; com_ibm_bpm_process_portal_hash=null
Host:ca8webp.itau:9446
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.60 Safari/537.17
Here is my code:
public bool Acessar(CredencialWAO credencial)
{
bool logado = true;
#region REQUISIÇÃO PAGINA INICIAL
ParametrosCrawler parametrosRequest = new ParametrosCrawler("https://ca8webp.itau:9446/ProcessPortal/login.jsp");
parametrosRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
parametrosRequest.AcceptEncoding = "gzip, deflate, sdch";
parametrosRequest.AcceptLanguage = "en-US,en;q=0.8";
parametrosRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.60 Safari/537.17";
parametrosRequest.CacheControl = "max-age=0";
}
protected override WebResponse GetWebResponse(WebRequest request)
{
var response = base.GetWebResponse(request);
AtualizarCookies(((HttpWebResponse)response).Cookies);
return response;
}
The error occurs when calling "base.GetWebResponse(request)"
I'm currently writing an API that gets data from a Point of Sale System's web interface. So far, I haven't had any problems logging in and generating reports to get data from until this situation.
In general, I can use the follow method to return an HttpWebRequest object that does the trick for most request to the web server.
private HttpWebRequest DefaultRequestObject(string path)
{
var request = (HttpWebRequest)WebRequest.Create(_baseUrl + path);
request.Method = "GET";
request.Host = _baseUrl.Substring(8); // Cut off the https://
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Accept-Language", "en-US,en;q=0.5");
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.SendChunked = false;
request.AllowAutoRedirect = true;
request.ProtocolVersion = HttpVersion.Version11;
var sp = request.ServicePoint;
sp.Expect100Continue = false;
var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
prop.SetValue(sp, (byte)0, null);
request.CookieContainer = _cookieJar;
if (!String.IsNullOrEmpty(_cookieString))
request.Headers.Add(HttpRequestHeader.Cookie, _cookieString);
return request;
}
When I send a GET request, I use the following method:
public MgrngResponse GetContent(string serverPath, Dictionary<string, string> args)
{
if (!serverPath.StartsWith("/"))
serverPath = "/~pos/mgrng/" + serverPath;
var requestString = serverPath + "?" + HttpDataFormat(args);
var request = DefaultRequestObject(requestString);
try
{
var response = (HttpWebResponse)request.GetResponse();
var mgrngResponse = new MgrngResponse(response);
if (!String.IsNullOrEmpty(mgrngResponse.HttpResponse.GetResponseHeader("Set-Cookie")))
SaveMgrngResponseCookies(mgrngResponse);
_sessionExpiration = DateTime.Now.AddMinutes(15);
UpdateStatus(mgrngResponse.Content);
return mgrngResponse;
}
catch (WebException webException)
{
using (WebResponse response = webException.Response)
{
var httpResponse = (HttpWebResponse)response;
Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
using (Stream data = response.GetResponseStream())
using (var reader = new StreamReader(data))
{
string text = reader.ReadToEnd();
Console.WriteLine(text);
}
}
var eventArgs = new SessionUpdateEventArgs(SessionStatus.ConnectionError, "Unable to GET data");
RaiseStatusChangeEvent(eventArgs);
return null;
}
}
This works well for all of the pages I've attempted so far, but now I'm running into a new problem where when I try to get the response for a particular page, the method throw a WebException with a 500 Internal Server Error.
I used Fiddler to match a browser's request exactly with mine (with the exception of the order of the headers and the cookie values obviously) but I'm still getting the Internal Server Error.
Below are the Raw Requests from Fiddler. The first one is from Firefox and the second one is from my program.
GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
Host: location.posprovider.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: USER=af182fda473354eb3199522726ca61c9d5516c95f165cffd63b9522726ca61c9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9fc516c950a6607ae63b9522726ca61c9; PHPSESSID=9d7f54f9a1769a3e0572745fe0db3d97
Connection: keep-alive
GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Host: location.posprovider.com
Cookie: PHPSESSID=16ad21c9d69fe85b3d185ab284f8620b; USER=30dca66f355e0ba89b6eda3c3e822ea24a95e5209e0f90bec94eda3c3e822ea243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e826395e520780b58edc94eda3c3e822ea2
Connection: Keep-Alive
I even tried logging into the web interface and then copy and pasting my generated request string into the browser and I got the desired data.
Any ideas?
Pull your programs request into Fiddlers composer and gradually eliminate the remaining differences. What do you find doing that?
Great advice! Apparently there's something wrong with my cookie line.
I copy and pasted the cookie line from the browser's version into mine
and it worked. Now I just need to resolve why my cookies aren't
correct...
I agree with that diagnosis. Who knows what's going on inside of the server. Probably some fragile code that expect a very exact cookie string format.
I have simple code for getting response from a vietnamese website: http://vnexpress.net , but there is a small problem. For the first time, it downloads ok, but after that, the content contains unknown symbols like this:�\b\0\0\0\0\0\0�\a`I�%&/m.... What is the problem?
string address = "http://vnexpress.net";
WebClient webClient = new WebClient();
webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 AlexaToolbar/alxg-3.1");
webClient.Encoding = System.Text.Encoding.UTF8;
return webClient.DownloadString(address);
You'll find that the response is GZipped. There doesn't appear to be a way to download that with WebClient, unless you create a derived class and modify the underlying HttpWebRequest to allow automatic decompression.
Here's how you'd do that:
public class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var req = base.GetWebRequest(address) as HttpWebRequest;
req.AutomaticDecompression = DecompressionMethods.GZip;
return req;
}
}
And to use it:
string address = "http://vnexpress.net";
MyWebClient webClient = new MyWebClient();
webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 AlexaToolbar/alxg-3.1");
webClient.Encoding = System.Text.Encoding.UTF8;
return webClient.DownloadString(address);
try with code and you'll be fine:
string address = "http://vnexpress.net";
WebClient webClient = new WebClient();
webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 AlexaToolbar/alxg-3.1");
return Encoding.UTF8.GetString(Encoding.Default.GetBytes(webClient.DownloadString(address)));
DownloadString requires that the server correctly indicate the charset in the Content-Type response header. If you watch in Fiddler, you'll see that the server instead sends the charset inside a META Tag in the HTML response body:
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
If you need to handle responses like this, you need to either parse the HTML yourself or use a library like FiddlerCore to do this for you.