We Keep Coding

Asp.net core FromBody always getting NULL

Hello friends i have created Asp.net core web service and calling this web service form web application. in this web api user upload pdf and document type and File number.
this is my asp.net core controller
[HttpPost ]
[Route("v1/ReceiveNoteData")]
public string ReceiveNoteData([FromBody] ReceiveNoteDataReq request)
{
return _IReceiveNoteData.ReceiveNoteDataResponse(request);
}
ReceiveNoteDataReq class
public class ReceiveNoteDataReq
{
public byte[] DocumentBody { get; set; }
public string DocType { get; set; }
public string FileNumber { get; set; }
}
when i am trying to call this web service using webrequest on service side frombody always getting NULL
if i am sending only file number and doc type i am getting data in from body but when i am trying to send pdf data from body getting NULL.
this is how i am calling the web service in my web application.
ReceiveNoteDataReq req = new ReceiveNoteDataReq()
req.DocumentBody = bytes;// pdf byte data
req.FileNumber = txtid.Text;
req.DocType = doctype;
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;
string url = "http://localhost:905/v1/ReceiveNoteData";
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "application/json";
webRequest.MediaType = "application/json";
webRequest.Accept = "application/json";
webRequest.CookieContainer = cookieJar;
webRequest.KeepAlive = true;
webRequest.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36";
webRequest.Credentials = CredentialCache.DefaultCredentials;
StreamWriter writer = new StreamWriter(webRequest.GetRequestStream());
JavaScriptSerializer jss = new JavaScriptSerializer();
string yourdata = jss.Serialize(req); // here i am converting class to json string.
writer.Write(yourdata);
writer.Close();
var myWebResponse = webRequest.GetResponse();
var responseStream = myWebResponse.GetResponseStream();
if (responseStream != null)
{
var myStreamReader = new StreamReader(responseStream, Encoding.Default);
result = myStreamReader.ReadToEnd();
}
Thanks in advance.

when i am trying to call this web service using webrequest on service side frombody always getting NULL if i am sending only file number and doc type i am getting data in from body but when i am trying to send pdf data from body getting NULL.
If you'd like to post pdf data with other information from request body to you backend service, you can refer to the following example.
//...
string base64String = Convert.ToBase64String(bytes, 0, bytes.Length);
string yourdata = System.Text.Json.JsonSerializer.Serialize(
new
{
DocType = doctype,
FileNumber = txtid.Text,
DocumentBody = base64String
});
writer.Write(yourdata);
writer.Close();
//...
The ByteArrayModelBinder can help bind data to byte[] DocumentBody property well.
public class ReceiveNoteDataReq
{
[ModelBinder(BinderType = typeof(ByteArrayModelBinder))]
public byte[] DocumentBody { get; set; }
public string DocType { get; set; }
public string FileNumber { get; set; }
}
Test Result
Besides, as #SabirHossain mentioned, you can try to modify your code to upload file through FormData, and this SO thread discussed similar requirement, you can refer to it.

Not able to download zip file using httpwebrequest through c# code. It gets downloaded through browser

I want to download the zip file from one of the website https://eqrreportviewer.ferc.gov/. The way in which the zip file gets downloaded is that you click on the filing inquiries tab first. In the reportType dropdown select SubmissionsBydate and in export dropdown select CSV. Now click on submit button and the zip file gets downloaded. I want to automate this process. I have written a code in C# by capturing the request along with its headers and passing that details to the site, but I am not able to download the file through code.
This is the code that I have written:
public static string PageSourceCode { get; set; }
//The ASP.NET SessionID to add validation to posts
public static string SessionID { get; set; }
//The value we are posting to the page on subsequent calls
public static string PostBackValue { get; set; }
public static string AcquisitionURL = "https://eqrreportviewer.ferc.gov";
static void Main(string[] args)
{
Acquire();
}
private static void Acquire()
{
GetLandingPage();
PopulatePostBackValueForSubmitBtn();
PostToPageForSubmitBtn();
}
private static void GetLandingPage()
{
string mainPageOutput = string.Empty;
HttpWebRequest objRequestLandingPage = (HttpWebRequest)WebRequest.Create(AcquisitionURL);
objRequestLandingPage.Method = WebRequestMethods.Http.Get;
objRequestLandingPage.Headers.Add("Cache-Control", "max-age=0");
objRequestLandingPage.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
objRequestLandingPage.Headers.Add("Accept-Encoding", "gzip, deflate, br");
objRequestLandingPage.Headers.Add("Accept-Language", "en-US,en;q=0.9");
objRequestLandingPage.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36";
objRequestLandingPage.Headers.Add("Sec-Fetch-Dest", "document");
objRequestLandingPage.Headers.Add("Sec-Fetch-Mode", "navigate");
objRequestLandingPage.Headers.Add("Sec-Fetch-Site", "none");
objRequestLandingPage.Headers.Add("Sec-Fetch-User", "?1");
objRequestLandingPage.Headers.Add("Upgrade-Insecure-Requests", "1");
//objRequestLandingPage.Headers.Add("Connection", "keep-alive");
objRequestLandingPage.KeepAlive = true;
objRequestLandingPage.Host = "eqrreportviewer.ferc.gov";
using (WebResponse objResponseLandingPage = objRequestLandingPage.GetResponse())
{
WebHeaderCollection headers = objResponseLandingPage.Headers;
using (Stream streamLandingPage = objResponseLandingPage.GetResponseStream())
using (StreamReader streamReaderLandingPage = new StreamReader(streamLandingPage))
{
mainPageOutput = streamReaderLandingPage.ReadToEnd();
}
SessionID = headers["Set-Cookie"];
}
SessionID = StripCookie(SessionID);
//Set the source code of the page
PageSourceCode = mainPageOutput;
}
private static void PopulatePostBackValueForSubmitBtn()
{
if (!String.IsNullOrEmpty(PageSourceCode))
{
// get fields from landing page
Dictionary<string, string> formFields = GetFormFields(PageSourceCode);
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelSummaryReports$ddlReportTypeSum"] = "0";
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelSummaryReports$ddlReportPeriodSum"] = "650";
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelSummaryReports$ListSearchExtender1_ClientState"] = String.Empty;
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelFilingInquiries$ddlReportType"] = "4";
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelFilingInquiries$txtFromSubmissionDate"] = System.DateTime.Now.Date.AddDays(-30).ToShortDateString();
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelFilingInquiries$txtToSubmissionDate"] = System.DateTime.Now.Date.ToShortDateString();
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelFilingInquiries$ddlExport"] = "2";
formFields["TabContainerReportViewer$TabPanelReporting$TabContainerReports$TabPanelFilingInquiries$btnSubmitOptional"] = "Submit";
formFields["TabContainerReportViewer$TabPanelDownloads$TabContainerDownloads$TabPanelSelectiveFilings$txtCID"] = String.Empty;
formFields["TabContainerReportViewer$TabPanelDownloads$TabContainerDownloads$TabPanelSelectiveFilings$txtFilingOrg"] = String.Empty;
formFields["TabContainerReportViewer$TabPanelDownloads$TabContainerDownloads$TabPanelSelectiveFilings$ddlQuarter"] = "Pick";
formFields["TabContainerReportViewer$TabPanelDownloads$TabContainerDownloads$TabPanelSelectiveFilings$ddlDownloadType"] = "CSV";
formFields["TabContainerReportViewer$TabPanelDownloads$TabContainerDownloads$TabPanelSelectiveFilings$txtName"] = String.Empty;
formFields["TabContainerReportViewer$TabPanelDownloads$TabContainerDownloads$TabPanelSelectiveFilings$txtEmail"] = String.Empty;
formFields["__EVENTTARGET"] = String.Empty;
formFields["__EVENTARGUMENT"] = String.Empty;
formFields["__LASTFOCUS"] = String.Empty;
formFields["__AjaxControlToolkitCalendarCssLoaded"] = String.Empty;
formFields["TabContainerReportViewer_ClientState"] = "{\"ActiveTabIndex\" : 0,\"TabState\": [true,true]}";
formFields["TabContainerReportViewer_TabPanelReporting_TabContainerReports_ClientState"] = "{\"ActiveTabIndex\" : 1,\"TabState\": [true,true]}";
formFields["TabContainerReportViewer_TabPanelDownloads_TabContainerDownloads_ClientState"] = "{\"ActiveTabIndex\" : 0,\"TabState\": [true,true]}";
formFields["__VIEWSTATE"] = ViewState;
formFields["__VIEWSTATEGENERATOR"] = ViewStateGenerator;
formFields["__VIEWSTATEENCRYPTED"] = ViewStateEncrypted;
string postString = FormatPostString(formFields);
PostBackValue = postString;
}
}
private static void PostToPageForSubmitBtn()
{
HttpWebRequest objRequestPostPage = (HttpWebRequest)WebRequest.Create(AcquisitionURL);
objRequestPostPage.Method = WebRequestMethods.Http.Post;
objRequestPostPage.ContentLength = PostBackValue.Length;
objRequestPostPage.ContentType = "application/x-www-form-urlencoded";
objRequestPostPage.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
objRequestPostPage.KeepAlive = true;
objRequestPostPage.Host = "eqrreportviewer.ferc.gov";
objRequestPostPage.Headers.Add("Cache-Control", "max-age=0");
objRequestPostPage.Headers.Add("Sec-Fetch-Dest", "document");
objRequestPostPage.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36";
objRequestPostPage.Headers.Add("Origin", "https://eqrreportviewer.ferc.gov");
objRequestPostPage.Headers.Add("Sec-Fetch-Site", "same-origin");
objRequestPostPage.Headers.Add("Sec-Fetch-Mode", "navigate");
objRequestPostPage.Referer = "https://eqrreportviewer.ferc.gov/";
objRequestPostPage.Headers.Add("Accept-Encoding", "gzip, deflate,br");
objRequestPostPage.Headers.Add("Accept-Language", "en-US,en;q=0.9");
//Pass in the ASP.NET Session ID
objRequestPostPage.Headers.Add("Cookie", SessionID);
objRequestPostPage.Headers.Add("Upgrade-Insecure-Requests", "1");
objRequestPostPage.Headers.Add("Sec-Fetch-User", "?1");
objRequestPostPage.ServicePoint.Expect100Continue = false;
StreamWriter streamWriterPostPage = new StreamWriter(objRequestPostPage.GetRequestStream());
//Post the arguments
streamWriterPostPage.Write(PostBackValue);
streamWriterPostPage.Close();
//Get response
HttpWebResponse responsePostPage = (HttpWebResponse)objRequestPostPage.GetResponse();
WebHeaderCollection responseHeaders = responsePostPage.Headers;
Stream responseStream = responsePostPage.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
PageSourceCode = reader.ReadToEnd();
using (FileStream file = new FileStream(#"C:\Test\test.csv", FileMode.Create, FileAccess.Write))
{
WriteFile(responseStream, file);
}
}
Can anyone let me know if there is something wrong that I am doing. Right now all the values are hard coded but if it works I can organize that properly.
Also I don't get the Content Disposition response header in the response that I am getting but I get this header when its gets runned from Chrome browser.
What is the code that I can do differently or if I am missing something?
Any help/suggestion would be great help moving forward with this issue.

I was not able to do this using C#.
Finally I used python in combination with selenium and chrome web driver to get the task done.
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-extensions")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")
options.add_experimental_option("prefs", {"download.default_directory":"/databricks/driver"})
driver = webdriver.Chrome(chrome_options=options)
driver.implicitly_wait(5)
url = "https://eqrreportviewer.ferc.gov/"
driver.get(url)
driver.implicitly_wait(5)
#Filing Inquiries
driver.find_element_by_xpath('//*[#id="__tab_TabContainerReportViewer_TabPanelReporting_TabContainerReports_TabPanelFilingInquiries"]').click()
driver.implicitly_wait(5)
#Submission by Date
driver.find_element_by_xpath('//*[#id="TabContainerReportViewer_TabPanelReporting_TabContainerReports_TabPanelFilingInquiries_ddlReportType"]/option[5]').click()
driver.implicitly_wait(5)
#CSV
driver.find_element_by_xpath('//*[#id="TabContainerReportViewer_TabPanelReporting_TabContainerReports_TabPanelFilingInquiries_ddlExport"]/option[2]').click()
driver.implicitly_wait(15)
#Submit
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//*[#id="TabContainerReportViewer_TabPanelReporting_TabContainerReports_TabPanelFilingInquiries_btnSubmitOptional"]')))
element.click()
driver.implicitly_wait(15) #putting wait here to make sure file gets downloaded before driver is stopped.
driver.quit()

C# Error on multithreading webrequests

i am trying to make webrequests with multiple threads but if i try with more than 2 i get error
Index was outside the bonds of the array
on this line:
string username = ScrapeBox1.Lines[NamesCounter].ToString();
Here's the code:
while (working)
{
while (usernamescount > NamesCounter)
{
string username = ScrapeBox1.Lines[NamesCounter].ToString();
string url = "http://www.someforum.com/members/" + username + ".html";
var request = (HttpWebRequest)(WebRequest.Create(url));
var response = request.GetResponse();
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0";
using (var responseStream = response.GetResponseStream())
{
using (var responseStreamReader = new StreamReader(responseStream))
{
var serverResponse = responseStreamReader.ReadToEnd();
int startpoint = serverResponse.IndexOf("Contact Info</span>");
try
{
string strippedResponse = serverResponse.Remove(0, startpoint);
ExtractEmails(strippedResponse);
}
catch { }
}
}
NamesCounter++;
textBox1.Text = NamesCounter.ToString();
}
}

This code is not thread safe.
You need the code for performing an HttpWebRequest to be atomic and outside the context of looping through the collection.
For example
public void MakeHttpWebRequest(string userName)
{
string url = "http://www.someforum.com/members/" + userName + ".html";
var request = (HttpWebRequest)(WebRequest.Create(url));
var response = request.GetResponse();
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0";
using (var responseStream = response.GetResponseStream())
{
using (var responseStreamReader = new StreamReader(responseStream))
{
var serverResponse = responseStreamReader.ReadToEnd();
int startpoint = serverResponse.IndexOf("Contact Info</span>");
try
{
string strippedResponse = serverResponse.Remove(0, startpoint);
ExtractEmails(strippedResponse);
}
catch { }
}
}
}
Assuming ScrapeBox.Lines implements IEnumerable, I would reccomend using Parallel.ForEach and passing the ScrapeBox.Lines as the IEnumerable over which to iterate.
Now, there is one additional problem, the code for reading the response from the HttpWebRequest still needs to write its output to a shared location. To accomplish that in a thread safe manner. A common way to do this is with a semaphore. You need a object accessible to each thread instance. A class level private variable private object sharedMutex = new object(); would work. Then the code ExtractEmails(strippedResponse); should be changed to
lock(sharedMutex)
{
ExtractEmails(strippedResponse);
}
Without having the code for the ExtractEmails(<string>) method, I can't provide a thread safe implementation for that, so that part of the solution may still cause a problem.

Dotnet webclient timesout but browser works file for json webservice

I am trying to get the result of the following json webservice https://mtgox.com/code/data/getDepth.php into a string using the following code.
using (WebClient client = new WebClient())
{
string data = client.DownloadString("https://mtgox.com/code/data/getDepth.php");
}
but it always returns a timeout exception and no data. I plan to use fastjson to turn the response into objects and expected that to be that hard part not the returning of the content of the page.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://mtgox.com/code/data/getDepth.php");
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
string data = sr.ReadToEnd();
}
}
Also resulted in the same error. Can anyone point out what i am doing wrong?

Hmm, strage, this works great for me:
class Program
{
static void Main()
{
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";
var result = client.DownloadString("https://mtgox.com/code/data/getDepth.php");
Console.WriteLine(result);
}
}
}
Notice that I am specifying a User Agent HTTP header as it seems that the site is expecting it.

I had similar issue before. request.KeepAlive = false solved my problem. Try this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://mtgox.com/code/data/getDepth.php");
request.KeepAlive = false;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
string data = sr.ReadToEnd();
}
}

stackoverflow search api

I would like to use the search method of stackoverflow API to return the json structure of results based on a search keyword and then display those results (title, description and the url) in the SearchResults div.
I am new to C# and my first attempt went something like this:
protected void searchStockOverflow(string y)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://api.stackoverflow.com/1.1/search?intitle="+y);
httpWebRequest.ContentType = "text/json";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = "{ \"intitle\": \"" + y + "\"}";
streamWriter.Write(json);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var responseText = streamReader.ReadToEnd();
SearchResults.InnerHtml += "<div style='border:1px solid blue;margin:5px;'>";
SearchResults.InnerHtml += responseText + "<br />";
SearchResults.InnerHtml += "</div><br style='clear:both;' />";
}
}
The issue is that what is returned looks like dingbats rubbish - i guess because it is serialized and need to be deserialized?

I would definitely say consider using the REST client; however, to look at the issues... generally you want to deserialize the data as JSON manually, then run that data through your UI code. For example:
static void SearchStackOverflow(string y)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://api.stackoverflow.com/1.1/search?intitle=" + Uri.EscapeDataString(y));
httpWebRequest.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
httpWebRequest.Method = "GET";
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
string responseText;
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
var result = (SearchResult)new JavaScriptSerializer().Deserialize(responseText, typeof(SearchResult));
.... do something with result ...
}
class SearchResult
{
public List<Question> questions { get; set; }
}
class Question
{
public string title { get; set; }
public int answer_count { get; set; }
}
Which uses the JavaScriptSerializer from System.Web.Extensions.dll

Also Take a look at Stacky StackApps .Net Client Library which is REST-based API that provides access to stackoverflow family of websites.

Unfortunately I'm on my Mac and can't run a test on your code. You might want to check the character encoding of both your page and the response stream coming back. If they don't match; it could cause the characters coming from the response stream to be rendered incorrectly, hence the rubbish you're seeing.