We Keep Coding

Cannot get rendered html via WebBrowser

I want to get html code from website. In Browser I usually can just click on ‘View Page Source’ in context menu or something similar. But how can I automatized it? I’ve tried it with WebBrowser class but sometimes it doesn’t work. I am not web developer so I don’t really know if my approach at least make sense. I think main problem is that I sometimes get html where not all code was executed. Hence it is uncompleted. I have problem with e.g. this site: http://www.sreality.cz/en/search/for-sale/praha
My code (I’ve tried to make it small but runnable on its own):
using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using System.Windows.Forms;
namespace WebBrowserForm
{
internal static class Program
{
[STAThread]
private static void Main()
{
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
for (int i = 0; i < 10; i++)
{
Form1 f = new Form1();
f.ShowDialog();
}
// Now I can check Form1.List and see that some html is final and some is not
}
}
public class Form1 : Form
{
public static List<string> List = new List<string>();
private const string Url = "http://www.sreality.cz/en/search/for-sale/praha";
private System.Windows.Forms.WebBrowser webBrowser1;
public Form1()
{
this.webBrowser1 = new System.Windows.Forms.WebBrowser();
this.SuspendLayout();
this.webBrowser1.Dock = System.Windows.Forms.DockStyle.Fill;
this.webBrowser1.Name = "webBrowser1";
this.webBrowser1.TabIndex = 0;
this.ResumeLayout(false);
Load += new EventHandler(Form1_Load);
this.webBrowser1.ObjectForScripting = new MyScript();
}
private void Form1_Load(object sender, EventArgs e)
{
webBrowser1.Navigate(Url);
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webBrowser1_DocumentCompleted);
}
private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
if (webBrowser1.ReadyState == WebBrowserReadyState.Complete)
{
// Final html for 99% of web pages, but unfortunately not for all
string tst = webBrowser1.Document.GetElementsByTagName("HTML")[0].OuterHtml;
webBrowser1.DocumentCompleted -= new WebBrowserDocumentCompletedEventHandler(webBrowser1_DocumentCompleted);
Application.DoEvents();
webBrowser1.Navigate("javascript: window.external.CallServerSideCode();");
Application.DoEvents();
}
}
[ComVisible(true)]
public class MyScript
{
public void CallServerSideCode()
{
HtmlDocument doc = ((Form1)Application.OpenForms[0]).webBrowser1.Document;
string renderedHtml = doc.GetElementsByTagName("HTML")[0].OuterHtml;
// here I sometimes get full html but sometimes the same as in webBrowser1_DocumentCompleted method
List.Add(renderedHtml);
((Form1)Application.OpenForms[0]).Close();
}
}
}
}
I would expect that in ‘webBrowser1_DocumentCompleted’ method I could get final html. It usually works, but with this site it doesn’t. So I’ve tried get html in my own code which should be executed in web site -> method ‘CallServerSideCode’. What is strange that sometimes I get final html (basically the same as if I do it manually via Browser) but sometimes not. I think the problem is caused because my script start before whole web site is rendered instead after. But I am not really sure since this kind of things are far from my comfort zone and I don’t really understand what I am doing. I’m just trying to apply something what I found on the internet.
So, does anyone knows what is wrong with the code? Or even more importantly how to easily get final html from the site?
Any help appreciated.

You should use WebClient class to download HTML page. No display control necessary.
You want method DownloadString

May be it will be helpful if you add calling of your external function to the end of the body and wrap it by Jquery "ondomready" function. I mean something like this:
private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
if (webBrowser1.ReadyState == WebBrowserReadyState.Complete)
{
// Final html for 99% of web pages, but unfortunately not for all
string tst = webBrowser1.Document.GetElementsByTagName("HTML")[0].OuterHtml;
webBrowser1.DocumentCompleted -= new WebBrowserDocumentCompletedEventHandler(webBrowser1_DocumentCompleted);
HtmlElement body = webBrowser1.Document.GetElementsByTagName("body")[0];
HtmlElement scriptEl = webBrowser1.Document.CreateElement("script");
IHTMLScriptElement element = (IHTMLScriptElement)scriptEl.DomElement;
element.text = "$(function() { window.external.CallServerSideCode(); });";
body.AppendChild(scriptEl);
}
}
[ComVisible(true)]
public class MyScript
{
public void CallServerSideCode()
{
HtmlDocument doc = ((Form1)Application.OpenForms[0]).webBrowser1.Document;
string renderedHtml = doc.GetElementsByTagName("HTML")[0].OuterHtml;
// here I sometimes get full html but sometimes the same as in webBrowser1_DocumentCompleted method
List.Add(renderedHtml);
((Form1)Application.OpenForms[0]).Close();
}
}

Wait for complete statement or function in winform of c#

I developing winform (c#) to read html form website.
When i click button, Textbox1 don't set text after 1 seconds. It wait unit the end forech.
Now i want, function will set text for textbox in 1 seconds.
how do it?
this is the code:
when kick button1:
private void button1_Click(object sender, EventArgs e)
{
string url = "http://truyentranh8.com/danh_sach_truyen/";
var web = new HtmlWeb();
var doc = web.Load(url);
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//tbody/tr/td[#class='tit']/a[#class='tipsy']"))
{
textBox1.Text += node.InnerText + "\n";
Thread.Sleep(1000);
}
}

Thread.Sleep in your case puts the main thread in in sleep mode. It can't update the UI till it gets released and the button1_Click method is over. So you don't see text changes per second. All you'll see is Text being updated all at once.
So make it asynchronous. If you're using .Net 4.5, you can use async/await and make life simple.
private async void button1_Click(object sender, EventArgs e)
{
string url = "http://truyentranh8.com/danh_sach_truyen/";
var web = new HtmlWeb();
var doc = web.Load(url);
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//tbody/tr/td[#class='tit']/a[#class='tipsy']"))
{
textBox1.Text += node.InnerText + "\n";
await Task.Delay(1000);
}
}
If you are interested I have written article on this subject.

Do not use Thread.Sleep on an event thread for this task.
The problem is that the UI is not getting a chance to update as it redraws on the thread that is blocked. As such the UI update only appears after all the thread-blocking code ends and the Click handler is exited.
Use an appropriate Timer instead, or if feeling hackish, read up about DoEvents. Alternatively, consider doing the long running task in a BackgroundWorker - the UserState of the Progress event can be used to report partial updates, already marshaled back to the appropriate thread.

Use DoEvents to refresh the form every time you change something on design
private void button1_Click(object sender, EventArgs e)
{
string url = "http://truyentranh8.com/danh_sach_truyen/";
var web = new HtmlWeb();
var doc = web.Load(url);
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//tbody/tr/td[#class='tit']/a[#class='tipsy']"))
{
textBox1.Text += node.InnerText + "\n";
Application.DoEvents();
}
}

How to change the web browser control properties?

I am creating a web browser using c# winform. I am using webbrowser control for this. I am using this code. This is running good so far
// Declared Variables
private string[] SiteMemoryArray = new string[100];
private int count = 0;
// Page Load
private void Form1_Load(object sender, EventArgs e)
{
webBrowser.Navigate("http://www.google.com/"); // Goes To A Preset Site At Run Time
SiteMemoryArray[count] = urlTextBox.Text; // Saves URL To Memory
}
// Code For The ToolStrip
// URL TextBox
private void urlTextBox_Click(object sender, EventArgs e)
{
urlTextBox.SelectAll(); // Selects All The Text In The urlTexBox
}
// GO Button
private void goButton_Click(object sender, EventArgs e)
{
webBrowser.Navigate(urlTextBox.Text); // Navigates To The Site Typed In The urlTextBox
}
// Back Button
private void backButton_Click(object sender, EventArgs e)
{
if (count > 0) // Checks To Make Sure The Count Variable Is More Then 0
{
count = count - 1; // Subtracts 1 From Count Variable
urlTextBox.Text = SiteMemoryArray[count]; // Replace The Text In The urlTextBox With The Last URl
webBrowser.Navigate(urlTextBox.Text); // Navigates To The Site Typed In The urlTextBox
forwardButton.Enabled = true; // Enables The forwarButton
}
}
// Forward Button
private void forwardButton_Click(object sender, EventArgs e)
{
if (count < 100) // Checks To Make Sure The Count Variable Is Less Then 100
{
count = count + 1; // Adds 1 To Count Variable
urlTextBox.Text = SiteMemoryArray[count]; // Replace The Text In The urlTextBox With The Next URl
webBrowser.Navigate(urlTextBox.Text); // Navigates To The Site Typed In The urlTextBox
backButton.Enabled = true; // Enables The backButton
count = count + 1; // Adds 1 To Count Variable
if (SiteMemoryArray[count] == null) // Checks To See If The Next Variable In The SiteMemoryArray Is Null
{
forwardButton.Enabled = false; // Disables The forwarButton
}
count = count - 1; // Subtracts 1 From Count Variable
}
}
But after create this small application my friend who is php developer ask me to check browser name . For this he create a php script n give me url then i run this url on my this browser its show me the browser name Internet Explorer
Now I want my browser name whatever I give name Please tell me is it possible with this control. Is there any property by using i can change it ?

The web browser control is IE. If you want to create your own browser, it is a lot more work than this. You need to write code that is able to do following and more:
Understand and handle HTTP protocol.
Understand, parse and render HTML. Most browsers ignore certain HTML errors and still render pages accurately. Not sure if you want that kind of features.
Your application should be able apply CSS settings on the pages.
Your application should be able to apply JS, flash, video, audio and other items that may well be embedded on a page.
You would also need to provide features that are available standard browsers.
The question is: What is the purpose of this application? Are you trying to write your own browser?

Weather fetching is not working more than once

I have a problem with the Wunderground forecast that I am using to retrieve data in c# program.
When I click to retrieve data once everything is working correctly but when I hit the button once more I am getting this error:
Here is my code:
private void bweather_DoWork(object sender, DoWorkEventArgs e)
{
string lat = Math.Round(deciLat).ToString();
string lng = Math.Round(deciLon).ToString();
string latlong = String.Format("{0},{1}", lat.Replace(',', '.'), lng.Replace(',', '.'));
//Initialize Current as a new Day
dow.Current = new WeatherLib.WDay();
//Using Wunderground as the provider we populate the property with current data for the latlong entered into the textbox
try
{
dow = WeatherLib.WProvider.Wunderground(latlong);
writeToLogFile("Retrieve weather info successfully on: " + latlong);
}
catch (Exception ex)
{
writeToLogFile(ex.Message);
}
}
Here is the refresh button:
private void weather_refresh_Click(object sender, EventArgs e)
{
writeToLogFile("Weather button pressed");
weather_descripton.Clear();
weather_speed_textbox.Clear();
weather_tem_textbox.Clear();
weather_rain_text.Clear();
weather_wind_dir_textbox.Clear();
weather_descripton.AppendText("Searching.......");
if (!bweather.IsBusy)
{
bweather.CancelAsync();
}
bweather.RunWorkerAsync();
}
And here are the event handlers:
// Weather handlers
bweather.WorkerSupportsCancellation = true;
bweather.DoWork += bweather_DoWork;
bweather.RunWorkerCompleted += bweather_RunWorkerCompleted;
Any idea why is this not working as it should?
Thank you

Well the error message suggests that you're trying to use the same background worker more than once.
You're asking it to cancel if it's still busy, but that doesn't mean it'll cancel immediately. As far as I can tell, the BackgroundWorker code isn't even checking whether it's been cancelled, which means cancelling it won't really achieve anything useful.
I would suggest that if it's busy, you should instead just ignore the request. In fact, it might be better to disable the button completely when you start the operation, and only re-enable it when the operation completes.

ConvertCsvToXml does not exist in the current context

private void button2_Click(object sender, EventArgs e)
{
//read in file data and calculate record number
string csv = File.ReadAllText("File-Path");
int testing = 1;
XDocument doc = ConvertCsvToXml(csv, new[] { "\t" });
//create array of record number
WebService.Function[] xxx = new WebService.Function[1];
recordAmount = 0;
Hi there for some reason in the above code the ConvertCsvToXml line is bringing back an exception saying it does not exist, and the intelisense brings up nothing after typing XDocument.
What can I do to resolve this? There are no errors in my code other than this!
Thanks for your time.

You'll have to define the ConvertCsvToXml method in the same class as the button2_Click method.