I've been trying to see all day if it's possible to write a web app in C#/asp.net that uses the google search engine. I've been googling about it all day. I don't want the custom search api because I'm not looking to have a search engine search through a site. I want to have my web app pass input to the basic google web search that search the entire net not a particular site so I can then parse through the first page of the search results. I put that in bold because it seems like the custom search api is for searching a particular site (my own site) which is not what I want to do and yet the only thing I could find. (well for the most part at least) The closest answer I found to my question is this https://stackoverflow.com/a/4082976/5607333 Which might do the trick for me but I don't know how to do that. How do I send search input to google search and get results using html? (or in my case asp.net) If you think it's the answer to my question can you please post an example of how it's done? I say "think" because I'm not sure it's the answer to what I'm asking.
I hope this question isn't considered a dupe to the question I linked to as I have been way more specific than it.
Also if this task isn't possible in C#/asp.net but possible in another language can someone please post an example of how it's done in that language or a link to it?
Update: I figured out what an easy solution is to this it hit while I was looking at another question similar to my problem. The solution is to edit the url and then i assume you could just concatenate it in C# with the + sign.
Update: 2 Even though I figured what I specifically was having trouble with at the moment of writing this question I still doesn't why I can't find a google equivalent of this https://msdn.microsoft.com/en-us/library/dd251020.aspx that's not depreciated. I read an answer to another question on here where someone said it's because that's how they make their money off the ad results but if that's true it still surprises me.
Look this question:
Adding Google's standard search (not custom) to my website
you can use your own XML parser to customize the display for your search users.
with an http request like this:
GET /search?q=bill+material&output=xml&client=test&site=operations
But it has a limitation on number of requests per day, 500 or 1000 I guess
Related
Is there a way to programatically upload an image file to search in Google, and then downloading the first one (the one with best resolution)?
EDIT: The Google Search API would not work for me, as I would have much more than 100 requests per day, and I am not willing to pay, since I am not a company
Yes, there is. The Google Custom Search API allows you to submit queries (including images) and retrieve results programmatically. There are even client libraries available for multiple languages.
EDIT: After OP changed his question, basically saying that he doesn't want to use the Google API, I can only refer to this(a bit outdated) question and quote the Google Terms of Service:
1.4 Appropriate Conduct. You shall not, and shall not allow any third party to: ... (i) directly or indirectly generate queries, or
impressions of or clicks on Results, through any automated, deceptive,
fraudulent or other invalid means (including, but not limited to,
click spam, robots, macro programs, and Internet agents);
So to recap, it is possible, but it is only legal via the API I linked above.
My apologies if this has already been requested/answered before, but I'm having a hard time trying to find information on the web about this. I've been tasked to find a way to directly update the content of a Google Sites page (Such as a text box or image) using the API.
I've been looking about the web for examples of doing this using C#, and have not found much to go by. I have some experience with writing/calling REST and SOAP APIs, but I can't seem to work out what Google uses for its API.
I'm struggling as to how I approach this, as the few examples I have found don't really walk through the code, so they leave me little to understand in order to manipulate it to what I want to achieve. One site I did find was Sites API Demo
Does anybody have any examples, or able to provide a simple example of updating content on a Google Site?
I appreciate any help you can give.
Mark
I realise this is an old question, but I have successfully updated the content of a google sites page using the library provided here, and following the example in the wiki.
I compiled the source as a .dll and added it as a reference to my project.
Ok. I thought I would fix this rather easy, but I'm not getting anything to work for some reason. What I want to do is simple; I want to create a web part displaying the latest tweets from a specific user. I'm coding C#.
I found a great post here but it's using php.
Can anyone help me to perform this simple task?
You can find alot examples over internet
Web:
http://www.microsoft.com/web/post/learning-the-basics-of-using-the-twitter-api-in-aspnet-web-pages-with-razor-syntax
Console:
http://www.d80.co.uk/post/2011/02/13/A-Simple-Twitter-Client-in-C-with-OAUTH-using-TweetSharp.aspx
:)
Can I use C# to auto search websites, then return the search results?
Is there a web crawler that would do the same thing if I give it a top domain (ex: I tell it find the word "funny" on stackoverflow.com, and it would tell me all the times "funny" appeared)?
These web sites allow searching via their search bar.
Do I need the web sites cooperation to automate searches?
NOTE: I only plan to be doing about one or two searches a day, so I doubt I'll be blocked, or asked to authenticate myself.
If your planning on crawling through an entire website to count words like that if you dont cache it you will get blocked, youll be requesting every page of the website essentially. Perhaps consider integrating google domain search's instead?
Here is a link to googles page detailing how to interface with c#
http://code.google.com/apis/gdata/client-cs.html
EDIT: Sorry that wasn't quite right : http://gsalib.codeplex.com/
http://answers.oreilly.com/topic/2165-how-to-search-google-and-bing-in-c/
I would look into building an RSS aggregator. RSS is standardized, so that's probably the most reliable way to collect search results from various sources.
EDIT: For sites that don't support RSS
For the sites that don't support RSS, you can look into using a screen scraper. Check out this article on The Code Project to get you started:
http://www.codeproject.com/KB/aspnet/weather.aspx
...web sites allow searching via their search bar ... Can I use C# to auto search websites, then return the search results?
Yes, if the website provides a URL where the search-term is provided as a query-string argument to a URL.
http://yourTargetDomain?searchterm=foo
But unless the website has specifically designed the search results from that URL to be structured data, the website won't "tell [you] all the times 'funny' appeared" but will send you back a search response that is suitable for a browser to display, so you would have to parse the results out of this stream of HTML.
For example:
http://philadelphia.craigslist.org/search/tls?query=ladder&srchType=A&minAsk=&maxAsk=
First of all, I hope my question doesn't bother you. I really need to get and idea of how I can accomplish that, but unfortunatelly, I'm really a beginner, I'm crawling when it comes to programming. I'm struggling to learn it the best way I can. I'll thank you for any help you give me.
Here's the task: I was ordered to find a way to collect some data from a website using a c# application. This will be done everyday, in order to update the data which we'll use to calculate some financial index.
I know my question might sound vague, anyway, even telling me how I can be more precise will help me. I know I seem to know desperate, but putting appart all the personell issues, my scholarship kind of depends on it.
Thanks in advance! (Please, don't mind the bad English, I'm brasilian and my English might not be that good yet.)
First, your English is fine. In fact, I thought you were a native speaker until you said otherwise.
The term you're looking for is 'site scraping'. Observe this question: Options for HTML scraping?. The second answer points to an HTML agility pack library you can use.
Now, there are two possibilities here. The first is you have to parse the HTML and scrape your data out of it. This is more computationally intensive and depends on the layout of the page. If they change the way the site looks, it could break the scraper.
The second possibility is they provide some XML or JSON web service you can consume. In this case you aren't scraping anything, but are rather using a true data feed. If the layout of the site changes, you will not break. Whether your target site supports this form of data feed is up to the site.
If I understand your question, you're being asked to do some Web Scraping, where you 1) download the contents of a web page and 2) try to parse data from that content.
For step #1, you should look into using a WebClient object in C# to download the HTML from the web page. You can give a WebClient object the URL you want to download the content from and obtain a String containing the content (probably HTML) of the URL.
How you go about doing step #2 depends on what content is present at the web site. If you know of certain patterns you're looking for in the HTML, you can search the HTML string using various methods. A more general solution for parsing HTML data can be found through using the Html Agility Pack, which will let you handle the HTML as a tree structure (DOM).
Use the WebClient class to get the page.
Turn the html into xml.
Use XPath to select the data you are interested in.
Ok, this is a pretty straightforward app design, and a lot of the code exists that you can reuse. Since you're a beginner, I'll break down into steps of what you need to do and recommend approaches.
1) You will use classes from System.Net to pull the web pages (WebClient being the easiest to usse). You will want to have this part of the program run on a timer if you can (using the scheduled jobs feature of the OS) and have it just pull the pages and drop them in a folder.
2) You have a second job which will run separately, pulling unread files from that folder, parsing them (using the HtmlAgility pack library is best) and then storing them in an index of some kind (Lucene is best for that)
3) You have a front end application of some sort (web or desktop) which queries that index for the information you're looking for.