We Keep Coding

Scraping web pages (including AJAX) from a .NET solution

Anyone has success with making scraping software in an azure function? I needs to be performed with some kind dynamic content loading like the web browser control or selenium where all content is loaded before scraping starts. Seems like Selenium is not an option due to the nature of azure functions.
I am trying to scrape some web pages and extract content. The pages are pretty dynamic. So first HTML is loaded and then through javascript data i lazy loaded. If using a standard http request I will not get the data. I could use the BrowserControl in .NET and wait for the Ready state, but the Browser control requires a browser and cannot be used in an Azure Function. Could be HtmlAgilityPack is the right answer. I tried it 5 years ago, and at the point it was pretty terrible in formatting html. I can see the have some kind of javascript library could be worth a try. Have you tried using that part of HtmlAgilityPack?

Your question is purely .NET-C#-ish (at least I assume you use .net c#).
Refer to this answer, please. If you achieve your goal in some way via .NET, you can do it in an Azure function - no restrictions on this side of the road.
For sure you will need an external third-party library that somehow simulates a web browser. I know that Selenium in a way uses browser "drivers" (not sure) - this could be an idea to research more thoroughly.
I was (and soon will be again) challenged with a similar request and I found no obvious solution. My personal expectations are that an external service (or something) should be developed and dedicated that then could send the result to an Azure HTTP Trigger function, which will proceed with the analysis. Even this so called "service" could have a Web API interface to be consumed from anywhere (e.g. Azure Function).

Targeting a specific Chrome process?

When I open the Chrome's Task Manager, I can see something like this?
Now, programatically, is it somehow possible to target the Chrome instance that is being used to run the Flash Plug-In? Maybe using System.Diagnostics.Process class or native API calls?

I was able to do this with native api calls. My solution worked perfectly on my machine. Here is what I did:
First, get Microsoft Process Explorer. Using this tool, I was able to get the flash plugin dll's name that chrome uses which is called pepflashplayer.dll in my case.
Next, a little search on Bing got me this article. It's C code so I converted to C# and after some coding I was able to retrieve the correct Chrome process ID that uses flash plug-in.
I have uploaded my solution to SkyDrive for you to check here. Note that I am not closing any handles in my sample so you'll have to add that for a real project.
There might be better solutions out there but this one should work. Hope it helps.

How to give Csharp Webbrowser default Path to download files

I am making a web browser in C Sharp, I want that all the files downloaded by the user on this web browser from any web sites, web browser saves it in one default folder (i.e C:\Users\Abc\Downloads)
Currently when i try to download file from any url it pops up a dialogue box asking for path, and it is annoying thing to have so i just wants to give one default path where it just saves the file automatically without asking user for the path.
Like we have default download path for Mozila firefox and google whenever user download any file from the web browser it saves in one default folder. so how can I achieve this in .net 4.0 Csharp web browser.

I'm afraid you can't with the webbrowser control,
maybe take a look at
http://www.mono-project.com/WebBrowser
first you don't use IE, and you can do more if I'm right
Regards,
Corné

If you're using the WebBrowser control in c# this can be somewhat of a challenge, but have a look at this link which might help you, I've tried it myself with good results.
http://www.codeproject.com/Articles/31163/Suppressing-Hosted-WebBrowser-Control-Dialogs
Good Luck! :)
Edit:
You might want to look into this solution to your issue as well:
Automated filedownload using WebBrowser without url
:)

Link Tracker C#

I am creating a download manager, and I need to be able to find out what link was clicked on (so my download manager can begin downloading the file), and also need to be able to stop the browser from launching its default download manager.
Does anyone know how to go about doing this, or know of any links to articles/tutorials/related docs?

As far as I know, the only way is to write a pluggin for IE which is done through Browser Helper Objects and that is unmanaged C++. Have a look at this:
http://msdn.microsoft.com/en-us/library/bb250436(VS.85).aspx

C# plugin for Google Chrome

Can I communicate to Google Chrome in C#?
For writing a chrome plugin for example.

<Spoilers>
Quick short answer: No, because Extensions require JSON, JavaScript, and HTML.
</Spoilers>
Hi Tony,
There are multiple templates on the internet that you can download to build a chrome extension project using Visual Studio.
Downloading one of these templates just gives you the files and folders that you need which I call "the setup".
That won't let you build a Google extension using C#.
Andrey mentioned that there are libraries like Bridge.NET that will compile your code into HTML/JavaScript. That will help you write HTML and JavaScript using C#. You still need a manifest.json file.
I don't recommend that. That library is not designed for Chrome Extensions. Also, you will have to compile the code into JavaScript and store that JavaScript code in a JavaScript file. I recommend using HTML/JavaScript with no compilers when building your Chrome Extension.
You need to keep in mind that there are 3 main parts in a chrome extension. These are:
manifest.json
A JavaScript file
HTML file
There are multiple steps and it's not hard to build a google chrome extension. This link will give you the complete tutorial on developing a chrome extension with detailed explanation. This tutorial installs a template so that you can develop it in Visual Studio just like I mentioned before.

What I have done to address is use Simple Message Host, it will trigger an executable on the local machine that you code in c#, sending stdin messages and listening to stdout messages so you can build this host to use as a bridge, but like I said, it needs to be on your local network at least, and you have to do some editing in the windows registry, so it has its limitations.
But for the system I am working with, this solution worked perfectly because I have a controlled environment that I can set up all these prerequisites.
So, just to clarify, what I did here is:
Create a chrome extension with background.js opening up the listener to the website's javascript.
Add a registry in windows registry pointing to the path of the executable.
Create the executable in C# doing all your logic.
Send a response from the executable to the extension and then back to the website.
There are several guides on how to do this, so I won't detail these steps here so I don't replicate it.
But for the moment, it is the best way to do what you want, if you have control of your environment that is.
So, if your plugin (extension or chrome app) will work on a controlled environment, this is the way to go, otherwise, I would consider something related to ClickOnce perhaps or WebAssembly but that's not fully released yet.

Chrome own extension manager supports extensions written in js and html.
that said, to execute c# code within the extension you have two options:
Compile c# code to javascript code which then can be added as a normal javascript extension to chrome (take a look at scriptsharp)
Use c# as a back-end system. just like most of download managers:
for case 2 you need a c# application installed in client device(or in the cloud) listing to a specific port (using httplistener or self hosted webapi (you can use netcore) which do these steps
Listing to incoming requests
parse request data eg. json and do something with it
return the result to javascript extension which can display it to user or do other things with it.

The topic is quite old, but I'd like to share that sample:
https://github.com/Retyped/Demos/tree/master/ChromeDemo
In a few words, the sample is implemented in C#. The Retyped.chrome NuGet package provides bindings (Chrome API) for Bridge.NET projects. So yes, now you can implement your logic in C#, then C# code will be transparently compiled into JavaScript by Bridge.NET compiler.
With that approach you can build your Chrome extension on top of .NET Framework as well as utilize thousands of JavaScript libraries.