Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I'm developing a web crawler in .Net C# that works like this.
Step1 Visits main page of the site (let's call this page Main.aspx)
Step2 Use httpwebrequest to get the form page (Let's call this page Form.aspx)
Step3 Post the form to another page and get the results. (Let's call this page Results.aspx)
It's pretty straight forward in terms of web crawling.
The current problem is, I can't access Form.aspx page if I dont set a bunch of cookies before. All of these cookies are javascript generated by Main.aspx.
Whenever i try to directly get the Form.aspx page, i get redirected to the Main page. The code that generates the cookies have more than 20kb and its aboslutelly messy and insane, also it uses a lot of "document." references which would block a simple attempt to use JINT or Javascript.net
So after a lot of research i found out that a headless browser would be what I'm looking for, tried a lot of them, but it seems a lot of complication. I already have a class library project with all my web crawlers in there, i just wanted another dll to make it work. Any suggestions?
I'm trying to be as clear as possible, if you have any doubt, please post on comments before giving negative votes...
Use a .NET binding for PhantomJS, which is a headless webkit browser. You might consider going to a full-blown automation framework like Selenium, which is made for testing.
What you are asking for in not simple, though. You are asking for a lot of abstractions so that you can keep the amount of simplicity in your app that you have now.
If you didn't mind a "head-ful" browser, you could also use the Windows Forms "WebBrowser" control or remote control Internet Explorer through COM.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 8 years ago.
Improve this question
I have a scenario I need to avoid my ASP.Net web application should not run in Firefox. If any one tries to open my application it will redirect to Chrome or Internet Explorer. I don't know how achieve this
situation.
Is it possible to handle this using JQuery or JavaScript ?
Simple answer: No
Longer answer...
Your question is fundamentally flawed - you simply cannot make a browser on a computer open a different browser on the same computer.
If you could to that, it would be a massive security flaw... and that is why you can't.
Although I don't like saying it - you could write an OCX (active X control) that would probably allow you to do it... but that would only work in Internet Explorer - and would require the user to install it.
But as you want to move from FireFox, this obviously doesn't answer your question.
No. It is not possible. You can't force a browser to open another application on the client machine (beside some browser plugins, like Adobe PDF).
Think what would happen if I could open your bank's application though javascript.
Everything is possible in technical sense. But if you try to do this way you may invite unnecessary issues from other browsers and loose focus on main objective.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I have a ASP.NET MVC website and I would like to profile all the method calls (including Javascript) which get invoked on a specific user action on a web page (button click for example). I would like to know which methods are taking longer and/or called frequently, so that I can look into the issue. I don't want to profile the whole website.. The event is handled by the Javascript first which makes an Ajax call to the server, gets back the response and does something with it. There is a lot going on both on the client and on the server when this event fires, so looking for profiler to show the root cause of the problem, instead of manually debugging and stepping through every single method.
The closest I can think of is http://miniprofiler.com/, it was originally developed for stack overflow if I understand correctly.
You can set the criteria (IP address, authentication, etc) for which to apply profiling on.
It doesn't look at client side profiling but then you can use firebug like another answer suggested.
I'm not aware of any tools that span both client scripting and server execution. I would suggest Firebug for profiling javascript and something like JetBrains dotTrace for the server-side.
It sounds like you are profiling for linear performance (just one user). So, you want to first get some data to see if you want to focus on the client javascript or server, then drill in.
You can use profiler shipped with VS. It could make Tier profiler, and you will get results of executing server-side methods, sql-queries, javascript used for generating response of each page.
Here is more about it http://msdn.microsoft.com/en-us/library/ms182372.aspx
Something like DynaTrace is what you need but you'll also need $$$
Take a look on UX Profiler. Its idea is to track and group all the browser events, network activity etc happening as a result of user action (click). It measures times of each activity and lets focus on the bottlenecks.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I am (was) a Python developer who is building a GUI web scraping application. Recently I've decided to migrate to .NET framework and write the same application in C# (this decision wasn't mine).
In Python, I've used the Mechanize library. However, I can't seem to find anything similar in .NET. What I need is a browser that will run in a headless mode, which has the ability to fill out forms, submit them, etc. JavaScript parser is not a must, but it would be quite useful.
There are some options:
WebKit.Net (free)
Awesomium
It is based on Chrome/WebKit and works like a charm.
There is a free license available but also a commercial one and if need be you can buy the source code :-)
HTML Agility Pack (free) (An HTML Parser library, NOT a headless browser)
This helps with extracting information from HTML etc. and might be useful in your case (possibly in combination with HttpWebRequest)
More solutions:
PhantomJS - full featured headless web
browser. Often used in pair with Selenium which allows you to
access the browser from .NET application.
Optimus (nuget package)- lightweight headless web browser. It's in beta but it is sufficient for some cases.
I used to use both for web testing. But they are also suitable for web scraping.
You may be after TrifleJS (currently in beta), or something similar using the .NET WebBrowser class which communicates with IE via a windowless ActiveX/COM API.
You'll essentially be running a fully fledged browser (not a http request wrapper) using Internet Explorer's Trident engine, if you are not interested in the JavaScript API (a port of phantomjs) you may still be able to use some of the C# codebase to get around key concepts (custom headers, cookies, script execution, screenshot rendering etc).
Note that this can also emulate different versions of IE depending on what you have installed.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
The MSDN Library provides great documentation for .NET/C#. However, the primary delivery medium, HTML via a web browser is suboptimal. A rich native Windows client for browsing this documentation seems like a perfect opportunity to showcase WPF. Is there such a client available? Are there any alternative interfaces to the MSDN Library?
Visual Studio 2010 can download the help to your local hard drive and load it from local server (it actually does use a local server, so search etc. works). Everything is seamless - you browse the documentation directly in VS itself. Hitting F1 with caret on a symbol will direct the searches into this local browser.
The help is to some extent integrated into the VS editor itself (for example as parameter info). ReSharper goes slightly further and displays even more info:
You can also browse the .Net code with some documentation bits in Object Browser.
Other than these, I don't know about anything - but I also don't know what features you would like. I personally think the old documentation browser present in VS <=2008 was horrible.
There are some features I would like to see which are currently not available anywhere as far as I know (for example list of implementors of an interface), but the solutions I listed are mostly all I personally need.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
Can anyone indicate a good 3270 emulator (which can login, scrape screen, find text, send keys etc. in background) for .NET (win and web). Something very similar to http://www.zephyrcorp.com/legacy-integration/index.htm (apparently zephyr costs like $5k per year, which is quite a lot, for one PC).
I also tried http://open3270.codeplex.com/SourceControl/list/changesets but it's way old and buggy.
Attachmate Reflection is really good, it meets all the requirements you mentioned and it has a great .NET API you can use to easily automate it from your .NET code. They don't list prices on their website but I think it is under $300...
For Desktop: I liked the idea of using a stable and well known client and remote control this instance. Therefore I ended up using x3270 (http://x3270.bgp.nu/) as the full featured, mature client in combination with X3270.Rest (https://www.nuget.org/packages/X3270.Rest/) that let's you remote control it from within a .NET application via the exposed REST interface (http://x3270.bgp.nu/rest.html). Reimplementing a complete new .NET client was not really an option because of all the alternatives out there. All those pitfalls on the road that lead to an reinvented wheel... But with an appropriate interface like the one mentioned it was quite easy to automate tasks without using DDE/HLLAPI.
If running in the background means without visible window one could start the x3270 client from .NET via Process.Start with suitable arguments to avoid a window beeing created.