We Keep Coding

I get HttpWebResponse with unclear characters when use a class in web application

I created a class in Windows Form application that uses HttpWebResponse.
This class gets the response from some page before login and after login and I analyze this pages.
Everything is good working at all time.
BUT
I used this class in a web application and uploaded project files in server.
I get good response from all page EXCEPT login page. Login page response contains unclear characters like this:
� �Z�N#G�v��CM�'��X؎2�LvV�e�H{��݅����t�
(ڋa���z.��2�͒H��膇
�+�9U�v�n��$pW��S�|��s�)=z����ߗ����0*T��~JuF5y�Ǟ���D��
and it happens often, that means I get sometimes normal response.
For example when extract my zip file again in server, it may work fine.
I am confused, I can't tell regular rule.
URL = "http://www.nonsense-website.com/login";
var request2 = (HttpWebRequest)WebRequest.Create(URL);
string username1 = jandoe;
string password1 = pass1234;
string postData = "hidlogin=1&username=" + username1 + "&password=" + password1;
var data = Encoding.UTF8.GetBytes(postData);
request2.Method = "POST";
request2.ContentType = "application/x-www-form-urlencoded";
request2.ContentLength = data.Length;
request2.UserAgent = "Mozilla/5.0 (Windows NT 10.0; rv:55.0) Gecko/20100101 Firefox/55.0";
request2.Headers.Add("Cookie:" + cookie);
request2.Headers.Add("Accept-Language: en-US,en;q=0.5");
request2.Headers.Add("Accept-Encoding: gzip, deflate");
request2.KeepAlive = true;
request2.Headers.Add("Upgrade-Insecure-Requests: 1");
using (var stream = request2.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
try
{
using (var response2 = (HttpWebResponse)request2.GetResponse())
{
responseString = new StreamReader(response2.GetResponseStream(),Encoding.UTF8).ReadToEnd();
AnalyzeResponse(responseString); // other function that analyze response data
}
}
catch (Exception ex)
{
WriteData("Exception in Login" + Environment.NewLine + ex.Message); // write in file
}
and my exception:
Exception in Login
Error in Parse the HTML
Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index

You're accepting gzip and deflate compressed responses (via Accept-Encoding), but you aren't checking whether the response is compressed. Turn on automatic decompression to have this done for you:
request2.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

I'm trying to write WebApi code to send an image file to Node.js using a POST request

I'm writing a WebApi program in C# that sends REST requests to a Node.js program written by a third party. I've got the source code for the Node program running on my laptop, but my knowledge of Node is limited. I'm using Fiddler to test sending requests. (Fiddler is similar to Postman and Curl.)
One of the things the WebApi program needs to do is to send an image file using a POST request. (The image file is associated with an order, so the order id is sent in the url.) When I send the request using Fiddler, Node returns OK and the image data is tranmitted successfully. When I send the request using the WebApi program, Node also returns OK, but the image data is not transmitted.
I would like to include some screenshots, but, apparently, I don't have enough credits to do that yet. In the successful Fiddler post, the Node.js console returns a json string containing UploadedFileMetaData data, followed by a Json string containing order_id and filename. In the WebApi case, UploadedFileMetaData is not returned. Instead, a file field containing byte data is added to the returned Json string.
I've listed below the steps that the WebApi program executes to send the request. Can anyone tell me what Fiddler is doing differently that causes the file data to be transmitted correctly? Thanks.
Variables are set with the following values:
string formDataBoundary = string.Format("----------{0:N}", Guid.NewGuid());
string contentType = "multipart/form-data; boundary=" + formDataBoundary;
string header = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"; filename=\"{2}\";\r\nContent-Type: {3}\r\n\r\n",
boundary,
param.Key,
fileToUpload.FileName ?? param.Key,
fileToUpload.ContentType ?? "application/octet-stream");
string footer = "\r\n--" + boundary + "--\r\n";
(The header string resolves to the following:
------------d19a2dfc4c734778b0849649617dff78
Content-Disposition: form-data; name="file"; filename="Ducky.jpg";
Content-Type: image/jpeg)
A memory stream is created:
Stream formDataStream = new System.IO.MemoryStream();
The header, file contents, and footer are written to the memory stream:
formDataStream.Write(encoding.GetBytes(header), 0, encoding.GetByteCount(header));
formDataStream.Write(fileToUpload.File, 0, fileToUpload.File.Length);
formDataStream.Write(encoding.GetBytes(footer), 0, encoding.GetByteCount(footer));
The stream is dumped into a byte array:
formDataStream.Position = 0;
byte[] formData = new byte[formDataStream.Length];
formDataStream.Read(formData, 0, formData.Length);
formDataStream.Close();
An http request is created containing the following info:
request.Method = "POST";
request.ContentType = contentType;
request.UserAgent = userAgent; // Value doesn't really matter
request.CookieContainer = new CookieContainer(); // Will not be used
request.ContentLength = formData.Length;
The request is posted:
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(formData, 0, formData.Length);
}
I get the response using request.GetResponse(). It returns OK. The content type returned is "application/json; charset=utf-8". GetResponseStream returns 131,072 bytes.

I think here's another simple solution for you.
//// Upload api written in expressjs.
var apiUpload = "http://localhost:3000/upload";
var filePath =Server.MapPath("~/App_Data/1023.jpg");
using (var client = new WebClient())
{
var responseBytes = client.UploadFile(apiUpload, "POST", filePath);
var result = System.Text.Encoding.UTF8.GetString(responseBytes);
}
I created a simple upload API with expressjs and above sample worked properly with it.

Sending xml to SOAP web service returning error 500

I have a piece of code written in C#.Net which reads from a database, collects records, builds a SOAP envelope around them and sends them to the following URL:
https://nodeD1.production.webservices.amadeus.com/SECRETNUMBER
I am able to copy my SOAP envelope string outputted from my code and paste it into SOAPUI, and it works flawlessly. But when I try to send it to the URL through my code, I always get:
The remote server returned an error: (500) Internal Server Error.
My entire SOAP xml request (including the soap header and everything else) is a single string by the time my code is done building it, and I have tried sending to the URL using the following code:
ServicePointManager.ServerCertificateValidationCallback = ValidateServerCertficate;
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
string httpResponseString = "";
webRequest.Proxy = null; //This will result in a quicker post.
byte[] requestBytes = null;
byte[] responseBytes = null;
requestBytes = System.Text.Encoding.UTF8.GetBytes(xmlRequestString);
//Set HttpWebRequest properties
webRequest.Method = "POST";
webRequest.ContentLength = requestBytes.Length;
webRequest.ContentType = "application/soap+xml;charset=UTF-8;";
//webRequest.Headers.Add("SOAPAction: http://webservices.amadeus.com/Profile_PartialUpdateNGProfile_11.1");
//webRequest.ContentType = "text/xml";
using (Stream requestStream = webRequest.GetRequestStream())
{
requestStream.Write(requestBytes, 0, requestBytes.Length);
}
try
{
using (var response = (HttpWebResponse)webRequest.GetResponse())
{
Stream stream = response.GetResponseStream();
using (MemoryStream ms = new MemoryStream())
{
int count = 0;
do
{
byte[] buf = new byte[1024];
count = stream.Read(buf, 0, 1024);
ms.Write(buf, 0, count);
}
while (stream.CanRead && count > 0);
responseBytes = ms.ToArray();
}
}
httpResponseString = System.Text.Encoding.Default.GetString(responseBytes);
}
catch (Exception e)
{
return e.Message;
}
return httpResponseString;
Am I forgetting something obvious? I have been stuck on this problem for a couple days now and I'm not sure what it could be. I have tried commenting and uncommenting the line:
webRequest.Headers.Add("SOAPAction: http://webservices.amadeus.com/Profile_PartialUpdateNGProfile_11.1");
But no dice.
NOTE: My piece of code is an asp.net web service (so I am a web service trying to communicate with another web service), could it be something in my Web.config file that's causing this?

Looks like all I had to do in my case was change my soap action in my soap envelope from:
https://nodeD1.production.webservices.amadeus.com/SECRETNUMBER
to:
https://noded1.production.webservices.amadeus.com/SERCRETNUMBER
Thanks to Fiddler I was able to see the following message:
<faultstring>A header representing a Message Addressing Property is not valid and the message cannot be processed</faultstring>
I Googled the error message and I found a StackOverflow post about it:
Error in Amadeus Service: A header representing a Message Addressing Property is not valid and the message cannot be processed
Apparently .net doesn't like capital letters in URLs in SOAP envelopes. The thing is I thought this might be the case, but I tried with lowercase and it still failed because my SOAP envelope wasn't properly built.

Cannot download iTunes Connect Weekly Opt-In Report from autoingest using c# (The magic number in GZip header is not correct error)

We are successfully getting many of the reports from the iTunes autoingest endpoint:
https://reportingitc.apple.com/autoingestion.tft
However some of the reports, rather than returning the data directly in the body return a ZIP file and whenever we try to read the response we get:
The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
The response headers look something like the following, indicating the zip file:
{
Content-Encoding:
Content-Disposition: attachment;filename=O_S_W_XXXXXXXX_20120805.zip
filename: O_S_W_XXXXXXXX_20120805.zip
Transfer-Encoding: chunked
Content-Type: application/a-gzip
Date: Wed, 29 Aug 2012 08:54:35 GMT
Set-Cookie: JSESSIONID=XXXXXXXXXXXXXXXX; Path=/
Server: Apache-Coyote/1.1
}
But I do not really see how to access this attachment and I think the error might be something to do with the fact that I am trying to ready both the attachment and other parts of the response in one operation whereas only part of the response is actually zipped.
The code looks something like this:
HttpWebRequest w = (HttpWebRequest)WebRequest.Create(url);
w.Method = "POST";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
w.ContentLength = byteArray.Length;
using (Stream dataStream = w.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
using (WebResponse resp = w.GetResponse())
{
using (System.IO.Compression.GZipStream s = new System.IO.Compression.GZipStream(resp.GetResponseStream(), System.IO.Compression.CompressionMode.Decompress))
{
// Just trying to read one byte causes the error.
s.ReadByte();
}
}
Any ideas greatly appreciated.

Take a look at this question and its accepted answer.
Use Fiddler to intercept the bytes on the wire (see the Inspectors tab).
As it turns out from the API: the stream is not GZipped but Zipped.
Have a look here for a library to unzip a stream in c#
At the start of your app add to overcome http 100 responses:
ServicePointManager.Expect100Continue = true;
Using that you'll implement the unzipping as follows, just after you obtaind the WebResponse:
using (ZipInputStream s = new ZipInputStream(resp.GetResponseStream()) {
ZipEntry theEntry;
while ((theEntry = s.GetNextEntry()) != null) {
Console.WriteLine(theEntry.Name);
string directoryName = Path.GetDirectoryName(theEntry.Name);
string fileName = Path.GetFileName(theEntry.Name);
// create directory
if ( directoryName.Length > 0 ) {
Directory.CreateDirectory(directoryName);
}
if (fileName != String.Empty) {
using (FileStream streamWriter = File.Create(theEntry.Name)) {
int size = 2048;
byte[] data = new byte[2048];
while (true) {
size = s.Read(data, 0, data.Length);
if (size > 0) {
streamWriter.Write(data, 0, size);
} else {
break;
}
}
}
}
}
}

Finally figured this out!
I needed to set:
webRequest.AutomaticDecompression = DecompressionMethods.None;
Previously I had it set to try to:
w.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
And .NET was trying and failing to unzip the response before I even had a chance to read it.
I can now read the byte array and decrypt the file after it has been saved to disk.

Screen scrape an ASP.NET Page not working

I am trying to bring back the calendar events on the page at the following site: http://www.wphospital.org/News-Events/Calendar-of-Events.aspx
Notice that this site has a link called "Month" - I need to be able POST data requesting calendar events for a particular month. I cannot get this to work. Here is the code:
private static void GetData(ref string buf)
{
try
{
//First, request the search form to get the viewstate value
HttpWebRequest webRequest = default(HttpWebRequest);
webRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://www.wphospital.org/News-Events/Calendar-of-Events.aspx");
StreamReader responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
string responseData = responseReader.ReadToEnd();
responseReader.Close();
//Extract the viewstate value and build out POST data
string viewState = ExtractViewState(responseData);
string eventValidation = ExtractEventValidation(responseData);
string postData = null;
postData = String.Format("ctl00$manScript={0}&__EVENTTARGET=&__EVENTARGUMENT&__LASTFOCUS=&__VIEWSTATE={1}&lng={2}&__EVENTVALIDATION={3}&ctl00$searchbox1$txtWord={4}&textfield2={5}&ctl00$plcMain$lstbxCategory={6}&ctl00$plcMain$lstbxSubCategory={7}", "ctl00$plcMain$updMonthNav|ctl00$plcMain$btnNextMonth", viewState, "en-US", eventValidation, "Search", "your search here", 0, 0);
var encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
//Now post to the search form
webRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://www.wphospital.org/News-Events/Calendar-of-Events.aspx");
webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)";
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = data.Length;
var newStream = webRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
//And read the response
responseData = responseReader.ReadToEnd();
responseReader.Close();
buf = responseData;
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
Console.Write("The server returned protocol error ");
// Get HttpWebResponse so that you can check the HTTP status code.
HttpWebResponse httpResponse = (HttpWebResponse)ex.Response;
int sc = (int)httpResponse.StatusCode;
string strsc = httpResponse.StatusCode.ToString();
}
}
}
private static string ExtractViewState(string s)
{
string viewStateNameDelimiter = "__VIEWSTATE";
string valueDelimiter = "value=\"";
int viewStateNamePosition = s.IndexOf(viewStateNameDelimiter);
int viewStateValuePosition = s.IndexOf(valueDelimiter, viewStateNamePosition);
int viewStateStartPosition = viewStateValuePosition + valueDelimiter.Length;
int viewStateEndPosition = s.IndexOf("\"", viewStateStartPosition);
return HttpUtility.UrlEncodeUnicode(s.Substring(viewStateStartPosition, viewStateEndPosition - viewStateStartPosition));
}
Can anyone point me in the right direction?

This may or may not solve your problem because I don't know exactly what the problem is when you say it's not working. But as "Al W" noted - the response from an async postback is not going to look like a straight HTML stream. So if your problem is parsing it afterwards then this might help.
I had the "opportunity" to discover this recently because I needed to rewrite that output. I'm working on a C# jQuery port and found that I was breaking WebForms pages when I tried to re-render the output stream during an async postback. I went through the client script that parses the response and figured out the format of the response.
Each panel that is updated will return a block of data that is formatted like:
"Length|Type|ID|Content"
There could be any number of these strung together. Type is "updatePanel" for UpdatePanels. ID is the UniqueID of the control, and Content is the actual HTML data. Length is equal to the number of bytes in Content, and you need to use that to parse each block, because the separator character may appear inside Content itself. So if you decided you wanted to rewrite this data before sending it back to an ASP.NET page (like I did) you need to update Length to reflect the final length of your content.
The code I used to parse and rewrite it is in Server/CsQueryHttpContext.

For POST operations, you want it to be UTF-8 encoded, so just re-do the one line
//var encoding = new ASCIIEncoding();
//byte[] data = encoding.GetBytes(postData);
//do this instead.....
byte[] data = Encoding.UTF8.GetBytes(postData);
and see if this helps you out

Below is the network trace I get in chrome when clicking the monthly button. Notice the __EVENTTARGET:ctl00$plcMain$monthBtn asp.net has a javascript framework in there that is calling a javascript:postback() method when that link is clicked, which sets the event target. That's basically how ASP.NET webforms knows which event to fire on a postback. One tricky thing here is that the web page is using an update panel so you might not get a true html response. If you can get your request to look something like this, then you should get a successful response. Hope this helps.
Request URL:http://www.wphospital.org/News-Events/Calendar-of-Events.aspx
Request Method:POST
Status Code:200 OK
Request Headers
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:no-cache
Content-Length:9718
Content-Type:application/x-www-form-urlencoded
Cookie:CMSPreferredCulture=en-US; ASP.NET_SessionId=h2nval45vq0q5yb0cp233huc; __utma=101137351.234148951.1312486481.1312486481.1312486481.1; __utmb=101137351.1.10.1312486481; __utmc=101137351; __utmz=101137351.1312486481.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __unam=ef169fe-131964a5f2a-24ec879b-1
Host:www.wphospital.org
Origin:http://www.wphospital.org
Proxy-Connection:keep-alive
Referer:http://www.wphospital.org/News-Events/Calendar-of-Events.aspx
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.124 Safari/534.30
X-MicrosoftAjax:Delta=true
Form Dataview URL encoded
ctl00$manScript:ctl00$plcMain$updTab|ctl00$plcMain$monthBtn
__EVENTTARGET:ctl00$plcMain$monthBtn
__EVENTARGUMENT:
__LASTFOCUS:
__VIEWSTATE:<removed for brevity>
lng:en-US
__EVENTVALIDATION:/wEWEgLbj/nSDgKt983zDgKWlOLbAQKr3LqFAwKL3uqpBwK9kfRnArDHltMCAuTk0eAHAsfniK0DAteIosMPAsiIosMPAsmIosMPAsuIosMPAoD0ookDApCbiOcPAo biOcPAombiOcPAoubiOcPyfqRx8FdqYzlnnkXcJEJZzzopJY=
ctl00$searchbox1$txtWord:Search
textfield2:Enter your search here
ctl00$plcMain$lstbxCategory:0
ctl00$plcMain$lstbxSubCategory:0
ctl00$plcMain$hdnEventCount:2