Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
Hello i'm trying to collect all data from website.
On start i collecting all links on that website. After that accessing them.
The problem is when i accessing multiple pages of website the "Invalid URI: The URI is empty" error is showing in some pages. These are some of those failed urls.
http://syayoyu.com/category/zensinyokudiet
http://syayoyu.com/fruitgranola-6759.html
http://syayoyu.com/category/diet/fruitgranola
Accessing from browser and accessing them one by one is working ok. But when i try to access in loop the error is occuring. Would you please tell me what is wrong with it.
This is my accessing code
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.UserAgent = "A .NET Web Crawler";
WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string htmlText = reader.ReadToEnd();
List<string> list = new List<string>(){
#"http://syayoyu.com/category/zensinyokudiet",
#"http://syayoyu.com/fruitgranola-6759.html",
#"http://syayoyu.com/category/diet/fruitgranola"
};
foreach (var url in list)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.UserAgent = "A .NET Web Crawler";
WebResponse response = request.GetResponse();
//Stream stream = response.GetResponseStream();
//StreamReader reader = new StreamReader(stream);
//string htmlText = reader.ReadToEnd();
//Console.WriteLine(htmlText);
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
string htmlText = reader.ReadToEnd();
Console.WriteLine(htmlText);
}
}
}
In my solution, it works fine,so is there any questions you don't know?
Related
I want to create PDF file by URL using ASPOSE.PDF and try the following code:
string dataDir =#"C:\Users\UbaidUllah\Documents\Visual Studio 2015\Projects\aspose\Data\AsposePDF\DocumentConversion\";
WebRequest request = WebRequest.Create("https://www.cricbuzz.com/cricket-stats/icc-rankings/men/batting");
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
reader.Close();
dataStream.Close();
response.Close();
MemoryStream stream = new MemoryStream(System.Text.Encoding.UTF8.GetBytes(responseFromServer));
HtmlLoadOptions options = new HtmlLoadOptions("https://www.cricbuzz.com/");
Document pdfDocument = new Document(stream, options);//----not execute this line nor give any error-----
pdfDocument.Save(dataDir + "WebPageToPDF_out.pdf");
When i debug this code, it does not go forward from the second last line(as described in comment) nor give any error, i wait much time but do not get any response.
I don't know where the mistake is. please review the code and help me to solve the issue.
Thank you very much!
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
I used webclient in c# to fetch html result but it is incorrect.
This is my code snippet.
WebClient client = new WebClient();
string htmlCode = client.DownloadString("https://kat.cr/");
MessageBox.Show(htmlCode);
And the result is not what is seen on the HTML page.
‹ í}ksÛ8²èçɯÀjÏnì›#âS¢òÚ’_I6vâµ<ÉÌÉM¥ ’8¦H–•=ûáVÝ_zÉíI=lÓ&dFÊNmjÆ’H
4ÝFãÅŸ>ìŸÿzzHÆÉÄ{õè~«‰çÇ/ã$ ŸµZÓé´9Õ›A4j©Ýn·u…eÄq£—
/‰X‹3çÕ#ÿ^LxÂÖ¤ü÷Ô½|ÙØü„û =Ÿ…¼Aìì×ËF¯’BzNì1‹bž¼L“!µû õ“™ÇogÇñjuŸMøËƥ˧a%Kŧ®“Œ_:üÒµ9?ž×w—y4¶™Ç_ªM¥AnæðØŽÜ0q ^Ÿ³Èæ;Ð/±Æ1ñùTü²Ë rNÎ?’xLcòWñ•G.Ÿ’Ip™}¦±k‹
À7ç„$nâñWïzç„’w®}Á öyE€7~ÑÊÞf%=׿ ÷^6bì¢xÌ9Ðœ#G-õG|ø²Ñj „%®Ý´ƒI‹yí¨ŠcèNS”*Ƥ‘
What should I do so that I can retrieve the actual page?
the html code is compressed.. set the AutomaticDecompression to GZip.
Just try this code and it will work (tested it with youe webpage)
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
request.AutomaticDecompression = DecompressionMethods.GZip;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (response.CharacterSet == null)
readStream = new StreamReader(receiveStream);
else
readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
string data = readStream.ReadToEnd();
response.Close();
readStream.Close();
}
Since you asked for a "webClient" Solution (see comment):
Because the webClient implementation is without decompression you will have to make your own DecompressedWebClient. Pretty easy if you inherit from the WebClient class:
public class AutomaticDecompressionWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address) as HttpWebRequest;
if (request == null) throw new InvalidOperationException("You can not use this WebClient implementation with an address that is not an http uri.");
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
I am trying to decode JSON returned by the website http://wthrcdn.etouch.cn/weather_mini?city=北京. When I look at this URL with Firefox using Unicode encoding, I see the following valid JSON:
{"desc":"OK","status":1000,"data":{"wendu":"20","ganmao":"各项气象条件适宜,发生感冒机率较低。但请避免长期处于空调房间中,以防感冒。","forecast":[{"fengxiang":"无持续风向","fengli":"微风级","high":"高温 27℃","type":"晴","low":"低温 15℃","date":"30日星期四"},{"fengxiang":"南风","fengli":"3-4级","high":"高温 30℃","type":"多云","low":"低温 16℃","date":"1日星期五"},{"fengxiang":"北风","fengli":"3-4级","high":"高温 25℃","type":"小到中雨","low":"低温 14℃","date":"2日星期六"},{"fengxiang":"无持续风向","fengli":"微风级","high":"高温 25℃","type":"多云","low":"低温 14℃","date":"3日星期天"},{"fengxiang":"无持续风向","fengli":"微风级","high":"高温 26℃","type":"多云","low":"低温 13℃","date":"4日星期一"}],"yesterday":{"fl":"微风","fx":"无持续风向","high":"高温 27℃","type":"晴","low":"低温 15℃","date":"29日星期三"},"aqi":"149","city":"北京"}}
But if I try to get and decode this JSON with HttpWebRequest the result is always gibberish:
string http = "http://wthrcdn.etouch.cn/weather_mini?city=北京";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(http); //创建一个请求示例
HttpWebResponse response = (HttpWebResponse)request.GetResponse(); //获取响应,即发送请求
Stream responseStream = response.GetResponseStream();
StreamReader streamReader = new StreamReader(responseStream, Encoding.UTF8);
string jsonStr = streamReader.ReadToEnd();
Console.WriteLine(jsonStr);
Produces
"�\b\0\0\0\0\0\0\0���JA�_%��`��\"_�� �t5���,�\v̏0�4LRJʲ����]jfֹ�:��-v!�������o��ƑG��ȅ�\ri:�#r�%I�!�1r�QT\rz\"�%��pp�`E�i^b����N��������Y��t�n��afw�)�}On�����#��ӭ}~�g�m���5[Ơ�b�}^�'�64!�&/�\r������ªk:r-ƑW\r���8�V:e�\r���gY�/�(�\r����5M���~�2��e�u[S��[:U=���G��\b���\f�\"aw0\a�P$Vj�rM8W*(a�J������i�O`E��i��1���m�B�8��L���Vt'Gw;0n^�pX���c�;\0�$s�j(#�F�4f&�P,��qt�(�dC1U�հ\a���\r|����j����7�=�kb�v���~=6�ĸB��'��L�\0\0"
Decoding with Encoding.Unicode produces:
"謟\b\0\0\0鎭䫝䄂윔╟⋕ꅟẋ먠琘섵�ጕⰑ௭迌됰䰴䩒닊늈橝홦㨕邃瘭픡컝�칯워䞑�藈එ㩩⏖爚◙뉉༡㇖앲呑稍ဢ░灰蜅䕠槳罞鵢캶鏀諸鳱릡如놬赴Ὦꪰ晡ٷ⦯鋲佽鉮췁�폨維�핧洚覤㖑왛鎠뭢幽➺㛝ℴ⚽�鬍蟀슆᮪㩫釆ൗ훺㣽Ϩ嘕攺ힳ柦驙蠯밨㗫䴔᳣쮊�鐲旸疙卛㩛唅똽說䜞냷萈ꨌ愢〓鄇橖犲㡍⩗愨䪳递뎾榕俇䕠育ᢘㆅ賶涅䋿ꌓ踸䳎隈嚝❴睇〻幮ᢹ瀓ј퇤럞揚뼂;ⓧ끳⡪责㒟♦僑ᨬ톎瑱⢓撱ㅃ퍕냕윇࿄ᅼ쎅迺罪趼�蜷ᠽ숅扫皣᳇�㵾밶쐘䊸쒉G퐃"
How can I decode the JSON from this web site using HttpWebRequest?
The response is compressed so you need to enable automatic decompression. Also, you should wrap your disposables in using statements:
string http = "http://wthrcdn.etouch.cn/weather_mini?city=北京";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(http); //创建一个请求示例
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
using (var response = (HttpWebResponse)request.GetResponse()) //获取响应,即发送请求
using (var responseStream = response.GetResponseStream())
using (var streamReader = new StreamReader(responseStream, Encoding.UTF8))
{
string jsonStr = streamReader.ReadToEnd();
// Use the JSON string.
// Starts out with {"desc":"OK","status":1000,"data":{"wendu":"20","ganmao":"各项气象条件适宜,...
Console.WriteLine(jsonStr);
}
Note that my console doesn't display Chinese characters. If yours doesn't either, the result of writing to the console will contain many "?" characters.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I'm in the process of creating a C# console application which reads text from a text file, turns it into a JSON formatted string (held in a string variable), and needs to POST the JSON request to a web api. I'm using .NET Framework 4.
My struggle is with creating the request and getting the response, using C#. What is the basic code that is necessary? Comments in the code would be helpful. What I've got so far is the below, but I'm not sure if I'm on the right track.
//POST JSON REQUEST TO API
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("POST URL GOES HERE?");
request.Method = "POST";
request.ContentType = "application/json";
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
byte[] bytes = encoding.GetBytes(jsonPOSTString);
request.ContentLength = bytes.Length;
using (Stream requestStream = request.GetRequestStream())
{
// Send the data.
requestStream.Write(bytes, 0, bytes.Length);
}
//RESPONSE HERE
Have you tried using the WebClient class?
you should be able to use
string result = "";
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.ContentType] = "application/json";
result = client.UploadString(url, "POST", json);
}
Console.WriteLine(result);
Documentation at
http://msdn.microsoft.com/en-us/library/system.net.webclient%28v=vs.110%29.aspx
http://msdn.microsoft.com/en-us/library/d0d3595k%28v=vs.110%29.aspx
Try using Web API HttpClient
static async Task RunAsync()
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://domain.com/");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// HTTP POST
var obj = new MyObject() { Str = "MyString"};
response = await client.PostAsJsonAsync("POST URL GOES HERE?", obj );
if (response.IsSuccessStatusCode)
{
response.//.. Contains the returned content.
}
}
}
You can find more details here Web API Clients
This may be a pathetically simple problem, but I cannot seem to format the post webrequest/response to get data from the Wikipedia API. I have posted my code below if anyone can help me see my problem.
string pgTitle = txtPageTitle.Text;
Uri address = new Uri("http://en.wikipedia.org/w/api.php");
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string action = "query";
string query = pgTitle;
StringBuilder data = new StringBuilder();
data.Append("action=" + HttpUtility.UrlEncode(action));
data.Append("&query=" + HttpUtility.UrlEncode(query));
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data.ToString());
request.ContentLength = byteData.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(byteData, 0, byteData.Length);
}
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream.
StreamReader reader = new StreamReader(response.GetResponseStream());
divWikiData.InnerText = reader.ReadToEnd();
}
You might want to try a GET request first because it's a little simpler (you will only need to POST for wikipedia login). For example, try to simulate this request:
http://en.wikipedia.org/w/api.php?action=query&prop=images&titles=Main%20Page
Here's the code:
HttpWebRequest myRequest =
(HttpWebRequest)WebRequest.Create("http://en.wikipedia.org/w/api.php?action=query&prop=images&titles=Main%20Page");
using (HttpWebResponse response = (HttpWebResponse)myRequest.GetResponse())
{
string ResponseText;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
ResponseText = reader.ReadToEnd();
}
}
Edit: The other problem he was experiencing on the POST request was, The exception is : The remote server returned an error: (417) Expectation failed. It can be solved by setting:
System.Net.ServicePointManager.Expect100Continue = false;
(This is from: HTTP POST Returns Error: 417 "Expectation Failed.")
I'm currently in the final stages of implementing an C# MediaWiki API which allows the easy scripting of most MediaWiki viewing and editing actions.
The main API is here: http://o2platform.googlecode.com/svn/trunk/O2%20-%20All%20Active%20Projects/O2_XRules_Database/_Rules/APIs/OwaspAPI.cs and here is an example of the API in use:
var wiki = new O2MediaWikiAPI("http://www.o2platform.com/api.php");
wiki.login(userName, password);
var page = "Test"; // "Main_Page";
wiki.editPage(page,"Test content2");
var rawWikiText = wiki.raw(page);
var htmlText = wiki.html(page);
return rawWikiText.line().line() + htmlText;
You seem to be pushing the input data on HTTP POST, but it seems you should use HTTP GET.
From the MediaWiki API docs:
The API takes its input through
parameters in the query string. Every
module (and every action=query
submodule) has its own set of
parameters, which is listed in the
documentation and in action=help, and
can be retrieved through
action=paraminfo.
http://www.mediawiki.org/wiki/API:Data_formats