Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I am (was) a Python developer who is building a GUI web scraping application. Recently I've decided to migrate to .NET framework and write the same application in C# (this decision wasn't mine).
In Python, I've used the Mechanize library. However, I can't seem to find anything similar in .NET. What I need is a browser that will run in a headless mode, which has the ability to fill out forms, submit them, etc. JavaScript parser is not a must, but it would be quite useful.
There are some options:
WebKit.Net (free)
Awesomium
It is based on Chrome/WebKit and works like a charm.
There is a free license available but also a commercial one and if need be you can buy the source code :-)
HTML Agility Pack (free) (An HTML Parser library, NOT a headless browser)
This helps with extracting information from HTML etc. and might be useful in your case (possibly in combination with HttpWebRequest)
More solutions:
PhantomJS - full featured headless web
browser. Often used in pair with Selenium which allows you to
access the browser from .NET application.
Optimus (nuget package)- lightweight headless web browser. It's in beta but it is sufficient for some cases.
I used to use both for web testing. But they are also suitable for web scraping.
You may be after TrifleJS (currently in beta), or something similar using the .NET WebBrowser class which communicates with IE via a windowless ActiveX/COM API.
You'll essentially be running a fully fledged browser (not a http request wrapper) using Internet Explorer's Trident engine, if you are not interested in the JavaScript API (a port of phantomjs) you may still be able to use some of the C# codebase to get around key concepts (custom headers, cookies, script execution, screenshot rendering etc).
Note that this can also emulate different versions of IE depending on what you have installed.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I'm developing a web crawler in .Net C# that works like this.
Step1 Visits main page of the site (let's call this page Main.aspx)
Step2 Use httpwebrequest to get the form page (Let's call this page Form.aspx)
Step3 Post the form to another page and get the results. (Let's call this page Results.aspx)
It's pretty straight forward in terms of web crawling.
The current problem is, I can't access Form.aspx page if I dont set a bunch of cookies before. All of these cookies are javascript generated by Main.aspx.
Whenever i try to directly get the Form.aspx page, i get redirected to the Main page. The code that generates the cookies have more than 20kb and its aboslutelly messy and insane, also it uses a lot of "document." references which would block a simple attempt to use JINT or Javascript.net
So after a lot of research i found out that a headless browser would be what I'm looking for, tried a lot of them, but it seems a lot of complication. I already have a class library project with all my web crawlers in there, i just wanted another dll to make it work. Any suggestions?
I'm trying to be as clear as possible, if you have any doubt, please post on comments before giving negative votes...
Use a .NET binding for PhantomJS, which is a headless webkit browser. You might consider going to a full-blown automation framework like Selenium, which is made for testing.
What you are asking for in not simple, though. You are asking for a lot of abstractions so that you can keep the amount of simplicity in your app that you have now.
If you didn't mind a "head-ful" browser, you could also use the Windows Forms "WebBrowser" control or remote control Internet Explorer through COM.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
The MSDN Library provides great documentation for .NET/C#. However, the primary delivery medium, HTML via a web browser is suboptimal. A rich native Windows client for browsing this documentation seems like a perfect opportunity to showcase WPF. Is there such a client available? Are there any alternative interfaces to the MSDN Library?
Visual Studio 2010 can download the help to your local hard drive and load it from local server (it actually does use a local server, so search etc. works). Everything is seamless - you browse the documentation directly in VS itself. Hitting F1 with caret on a symbol will direct the searches into this local browser.
The help is to some extent integrated into the VS editor itself (for example as parameter info). ReSharper goes slightly further and displays even more info:
You can also browse the .Net code with some documentation bits in Object Browser.
Other than these, I don't know about anything - but I also don't know what features you would like. I personally think the old documentation browser present in VS <=2008 was horrible.
There are some features I would like to see which are currently not available anywhere as far as I know (for example list of implementors of an interface), but the solutions I listed are mostly all I personally need.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
Can anyone indicate a good 3270 emulator (which can login, scrape screen, find text, send keys etc. in background) for .NET (win and web). Something very similar to http://www.zephyrcorp.com/legacy-integration/index.htm (apparently zephyr costs like $5k per year, which is quite a lot, for one PC).
I also tried http://open3270.codeplex.com/SourceControl/list/changesets but it's way old and buggy.
Attachmate Reflection is really good, it meets all the requirements you mentioned and it has a great .NET API you can use to easily automate it from your .NET code. They don't list prices on their website but I think it is under $300...
For Desktop: I liked the idea of using a stable and well known client and remote control this instance. Therefore I ended up using x3270 (http://x3270.bgp.nu/) as the full featured, mature client in combination with X3270.Rest (https://www.nuget.org/packages/X3270.Rest/) that let's you remote control it from within a .NET application via the exposed REST interface (http://x3270.bgp.nu/rest.html). Reimplementing a complete new .NET client was not really an option because of all the alternatives out there. All those pitfalls on the road that lead to an reinvented wheel... But with an appropriate interface like the one mentioned it was quite easy to automate tasks without using DDE/HLLAPI.
If running in the background means without visible window one could start the x3270 client from .NET via Process.Start with suitable arguments to avoid a window beeing created.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I like GWT but I prefer to use ASP.NET MVC for my projects, however, these two are not integrated and require me to write my code in two different platforms and two languages. Does Microsoft have any solutions comparable to GWT for compiling C# into JavaScript? I know there is Script# which is not supported by MS and the Volta project which was killed after its preview, but I was wondering if there is any good solutions available now or at least some good open source project that can integrate ASP.NET with GWT. Thanks.
Well, I can tell you what my preferred stack looks like these days. To me it is a nice balance of established tech with flexibility, though keep in mind I use this mostly to build single-page ajax "apps", not for the traditional collection of pages.
Sharp UI (full disclosure: this is one of my open source projects)
Script#
jQuery
I use a tool I wrote internally for generating "packet" classes shared by WCF and Script#.
WCF (in JSON)
ASP.NET (either Webforms or MVC)
I get compile-time type checking from Script#, UI control encapsulation from Sharp UI, fairly easy to maintain JSON service endpoints through WCF and my code generation tool, and ASP.NET for misc or traditional web pages. I'm firing on all 8 cylinders with this setup.
Bridge.NET is in this space. It describes itself as:
Open Source C# to JavaScript Compiler and Frameworks.
Run Your App On Any Device Using JavaScript.
The Microsoft driven solution is TypeScript which is a separate language made with input from the lead architect of C#, Anders Hejlsberg. It is also open source.
Good suggestion, but as AFAIK there is absolutely nothing like GWT in the .Net world.
I'm a Java and .Net programmer. I've battled infrequently with javascript for about 3 years, and never become comfortable with it. Since adopting GWT I'm producing Javascript=based web pages but coding in Java - I absolutely adore it ;-)
There's no great reason why there can't be a .Net equivalant of GWT. GWT doesn't do a 'literal copy' of Java to produce the Javascript - so it doesn't rely on the two languages having a 'similar' grammer. Any language could be converted. Mind you, it would take a lot of effort to duplicate the analysis and optimisation performed by the GWT compiler in producing it's js files.
A more effective route may be to find a C# to Java converter, and then pass the output to GWT.
SharpKit for C# .NET is like GWT but actually does much more. They even have a CLR written entirely in Javascript that provides Reflection, Generics, etc. on the client.
http://sharpkit.net
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
Last week i searched for good free or opensource solutions and component for GIS (Geographical Information Systems) I founded some system but no one fill my requirements
SharpMap is very buggy software
Gmap.net is very slow
MapWindow have a very complex structure and is very buggy.
I founded uDIG but is in java, i need a solution in vb.net or c#.
Anyone know a good solution that fill my requirements or have alternatives, i accept solutions?
You are limiting yourself a lot by insisting on .NET. I don't know of anything other than SharpMap or MapWinGIS ActiveX (MapWindow). Here are some free, but not .NET, options for Windows desktop applications.
If you'd consider writing your standalone application in Python or C++:
Mapnik
QGIS
Or if you'd consider writing a plug-in or a customisation for an existing GIS:
GRASS can be customised in Python, Perl, Ruby...
QGIS can be customised in Python
I think that you've covered it already. There really aren't any production quality open source GIS project out there using C#. Most of the good work is being done in Java, C/C++ or Python these days. If you must use the .NET Framework then I think the best of the bunch is indeed SharpMap.
Failing that you need to look at commercial products from companies like http://www.esri.com. Of course, it also depends on what you need: web services, Windows Forms control, WPF, etc. In the past I've managed to whip up some C# that constructed the right XML to send to a Java server-based mapping engine, so you could look at something like GeoServer and build your own client. Obviously not what you want to get in to but I don't see that you have many options beyond the ones you've listed.
I would recommend to look in to MapAround
Have you checked out SharpMap? It's available on codeplex.
MapSurfer.NET framework might be a good option.
MapSurfer.NET is free, modern cartographic framework which is able to provide maps of superior cartographic quality. This framework supports a bunch spatial data formats (e.g., Shape files, PostgreSQL, OSM, etc.) and web services (e.g., CartoDB, Mapzen, etc.). Furthermore, its setup includes MapSurfer.NET Studio application which allows creating and editing map styles (analogue of TileMill). Its symbology is inspired by both OGC specifications and other similar toolkits such as MapServer, GeoServer or Mapnik.
We use Mapzania (http://www.mapzania.com).
The best thing about it is that you load it into existing web-applications via a NuGet package and then you get a bunch of GIS functionality.
It uses Leaflet as its front-end and it has JS library that makes it easy to do stuff to Leaflet.
It also has a nice MapStyler for creating and editing maps.