How to extract the extra content loaded in a web page, which will not be visible in view page source. The extra content is being loaded using ajax. This data can be seen under NET tab using firebug. How to extract this data using c# code.
Two ways :
1- You can use webbrowser to load the same page and get the active document.
2- You can replicate the ajax call made, and use that to get the extra bits that are appended to the document.
And reading your linkedin example above:
When you select the checkbox a ajax call is made , which brings back results and populates the table.You can see that call using firebug console window and see the post parameter and replicate them to get the same result.
Depends on your application in the first place, if you are using c# application as the client for reading a web page, then the the ajax content may not be visible until you put in a javascript engine.
if you are serving the said pages, you only have to log the request response of the server.
More specific question would be appreciated
That extra content is dynamically generated by ajax (for eg: Gridview is generated as table), it is stored in browser's memory. and can be viewed by client side debugging tools (IE has developer tools option).
Once you do a post back, all the control's values are available for C#.
If you are saying extra content, can you please clarify what exactly you are trying to extract using c#?
Related
I am working on scrapping of website. so i make one desktop application for that.
I check website using inspect element then i can see whole data of website but when i try to check website data using page source(ctrl+U) then there is nothing.
means i can't find any website data in page source but can see in firebug(inspect element).
because of this when i am trying to get data using c# coding then i am getting only page source data which doesn't contains any website data only contains schema(structure) and js links.
see below image of firebug.
And this is page source image.
You met the js-powered site. The content is dynamicly loaded thru js, thus it's not visible in page-source. Turn to the scrape libraries that support js code evaluation. See here an example.
I am using the JQuery UI Modal Form Dialog and trying to save the old data and new data to a database. I am using C# (backend) and ASP.NET front end. I can delete new entries, I just don't know how to save the data. I have tried searching ways to pull pull the HTML data in, but couldn't get rid of the errors. Also Wasn't sure if there was a better method? Here is my JSFiddle
function addUser() {
I need to figure out a way to get the data from the table and post it to the server.
I think what you may be looking for is web methods, it allows the jquery (client side) to call the backend (c#) and for them to interact with each other.
Check out this link for reference:
http://encosia.com/using-jquery-to-directly-call-aspnet-ajax-page-methods/
Here is a brief summary from the website:
When it comes to lightweight client-side communication, I’ve noticed
that many of you prefer ASP.NET AJAX’s page methods to full ASMX web
services. In fact, page methods came up in the very first comment on
my article about using jQuery to consume ASMX web services. Given
their popularity, I’d like to give them their due attention. As a
result of Justin‘s question in those comments, I discovered that you
can call page methods via jQuery. In fact, it turns out that you can
even do it without involving the ScriptManager at all. In this post, I
will clarify exactly what is and isn’t necessary in order to use page
methods. Then, I’ll show you how to use jQuery to call a page method
without using the ScriptManager.
I found this:
This script gets data from the table that you can then parse through.
You could then pass it into a hidden field.
I have seen this on some survey websites. What is the C# code they use on the client side to keep the URL same, but when clicking the "Next" button, the same aspx page is maintained
without having any query string;
without any change even a character in the url; and
the grid, the data , the content, the questions keep changing?
Can anyone give a code-wise example how to achieve this?
My main query is how is this done in code-behind to change data of page and maintain same url.
Nothing simpler that a session, maintainted at the server side. Store a "current question number" in session, increment it at each succesfull postback and you have what you ask about.
Another possibility - a cookie which contains "current question number".
Both cookie and session are invisible in the query string of course.
"change data of page and maintain same url." Answer is Server.Transfer.
This method will preserve url.
The Next button may submit a form using the HTTP POST method. The form data may contain the session, question and response data. The site uses that to build a new response. Unlike a GET, a POST does not incorporate data into the URL.
Developers will typically accomplish this task by using AJAX. The basic premise behind it is that only a certain portion of the page (e.g. a grid or content area) will make a server call and retrieve the results (using Javascript). The effect achieved is that there has not been a full post back, which is why you don't see the URL or parameters changing.
It is possible to do this using jQuery, pure Javascript, or Microsoft's UpdatePanel.
oleksii's comment has some good links as well:
That's the AJAX magic. There are many JQuery plugings for this, for
example this one with a live demo. You can also program it easily
using JQuery Get or Post or any other wrapper that use XmlHttpRequest
object.
I have a crawler application which should parse all items from a Page with paging. Unfortunately the web site that my application crawls uses postbacks for paging. How can I get contents of second page programmatically for following URL in C#.
http://www.hurriyetemlak.com/coldwell-banker-dikey-gayrimenkul/konut-satilik/istanbul-basaksehir-bahcesehir/emlak-ofisleri-ilanlari/3OWB4lkhYFs=/9wZEBZ-ivFgmrA3ENMCIfQ==/qh.BgsUoTK4=/GmMGgVD5Wcc=/GmMGgVD5Wcc=?sParam=3OWB4lkhYFs%3d&sType=9wZEBZ-ivFgmrA3ENMCIfQ%3d%3d&ListIsBig=qh.BgsUoTK4%3d&sortType=GmMGgVD5Wcc%3d&pageSize=GmMGgVD5Wcc%3d
I've tried posting __EVENTTARGET hidden field along __VIEWSTATE and __EVENTVALIDATION but it didn't seem to work.
You can achieve this using screen scraping techniques (See HtmlAgilityPack). This will require you to read the response and reissue form posts to mimic what the user would do in a browser. Simple request replays will not work.
You may also need to pass __EVENTARGUMENT hidden field. And don't forget to set name attribute, as well as id attribute.
On a button click event I am required to POST to a page on an external website and redirect there. I get how to do this using a GET method
Reponse.Redirect("www.somesite.com?my=params&as=aget")
But how can I do this on as POST?
I don't want to post the entire form as this button event is called within a repeater
Depends.
If you want to post the exact input of a form you have on your site (that is, you just replicate a form the other site has), then just set the form's action to the URL you want to post to and the browser will do everything for you.
If however you want to post some values you generate on the server (perhaps based on the input from your form), I'm afraid it's not possible. You can't redirect using a POST. Redirect is GET by it's nature.
BUT you might be able to fake it by doing a POST (using something like System.Net.WebClient) and then a redirect (it depends on how the other site handles the GET - it might display the same thing that it did on the POST, or not).
One more option (for the second case) would be to to do an AJAX call to your server, which will compute the required values, then do the POST to the other server from Javascript.
You can build up the request using WebClient, adding the appropriate headers.
My inner forms don't contain the runat="server" attribute so I can do what I want. I do get this problem though ASP.Net First inner form in Server Form doesn’t POST.
Jquery is life saving in this situation. Used for one of my project and works like a charm. Give it a try : Peter Finch - Using Javascript to POST data between pages