Can I use C# to auto search websites, then return the search results?
Is there a web crawler that would do the same thing if I give it a top domain (ex: I tell it find the word "funny" on stackoverflow.com, and it would tell me all the times "funny" appeared)?
These web sites allow searching via their search bar.
Do I need the web sites cooperation to automate searches?
NOTE: I only plan to be doing about one or two searches a day, so I doubt I'll be blocked, or asked to authenticate myself.
If your planning on crawling through an entire website to count words like that if you dont cache it you will get blocked, youll be requesting every page of the website essentially. Perhaps consider integrating google domain search's instead?
Here is a link to googles page detailing how to interface with c#
http://code.google.com/apis/gdata/client-cs.html
EDIT: Sorry that wasn't quite right : http://gsalib.codeplex.com/
http://answers.oreilly.com/topic/2165-how-to-search-google-and-bing-in-c/
I would look into building an RSS aggregator. RSS is standardized, so that's probably the most reliable way to collect search results from various sources.
EDIT: For sites that don't support RSS
For the sites that don't support RSS, you can look into using a screen scraper. Check out this article on The Code Project to get you started:
http://www.codeproject.com/KB/aspnet/weather.aspx
...web sites allow searching via their search bar ... Can I use C# to auto search websites, then return the search results?
Yes, if the website provides a URL where the search-term is provided as a query-string argument to a URL.
http://yourTargetDomain?searchterm=foo
But unless the website has specifically designed the search results from that URL to be structured data, the website won't "tell [you] all the times 'funny' appeared" but will send you back a search response that is suitable for a browser to display, so you would have to parse the results out of this stream of HTML.
For example:
http://philadelphia.craigslist.org/search/tls?query=ladder&srchType=A&minAsk=&maxAsk=
Related
I've been trying to see all day if it's possible to write a web app in C#/asp.net that uses the google search engine. I've been googling about it all day. I don't want the custom search api because I'm not looking to have a search engine search through a site. I want to have my web app pass input to the basic google web search that search the entire net not a particular site so I can then parse through the first page of the search results. I put that in bold because it seems like the custom search api is for searching a particular site (my own site) which is not what I want to do and yet the only thing I could find. (well for the most part at least) The closest answer I found to my question is this https://stackoverflow.com/a/4082976/5607333 Which might do the trick for me but I don't know how to do that. How do I send search input to google search and get results using html? (or in my case asp.net) If you think it's the answer to my question can you please post an example of how it's done? I say "think" because I'm not sure it's the answer to what I'm asking.
I hope this question isn't considered a dupe to the question I linked to as I have been way more specific than it.
Also if this task isn't possible in C#/asp.net but possible in another language can someone please post an example of how it's done in that language or a link to it?
Update: I figured out what an easy solution is to this it hit while I was looking at another question similar to my problem. The solution is to edit the url and then i assume you could just concatenate it in C# with the + sign.
Update: 2 Even though I figured what I specifically was having trouble with at the moment of writing this question I still doesn't why I can't find a google equivalent of this https://msdn.microsoft.com/en-us/library/dd251020.aspx that's not depreciated. I read an answer to another question on here where someone said it's because that's how they make their money off the ad results but if that's true it still surprises me.
Look this question:
Adding Google's standard search (not custom) to my website
you can use your own XML parser to customize the display for your search users.
with an http request like this:
GET /search?q=bill+material&output=xml&client=test&site=operations
But it has a limitation on number of requests per day, 500 or 1000 I guess
Is there a way to programatically upload an image file to search in Google, and then downloading the first one (the one with best resolution)?
EDIT: The Google Search API would not work for me, as I would have much more than 100 requests per day, and I am not willing to pay, since I am not a company
Yes, there is. The Google Custom Search API allows you to submit queries (including images) and retrieve results programmatically. There are even client libraries available for multiple languages.
EDIT: After OP changed his question, basically saying that he doesn't want to use the Google API, I can only refer to this(a bit outdated) question and quote the Google Terms of Service:
1.4 Appropriate Conduct. You shall not, and shall not allow any third party to: ... (i) directly or indirectly generate queries, or
impressions of or clicks on Results, through any automated, deceptive,
fraudulent or other invalid means (including, but not limited to,
click spam, robots, macro programs, and Internet agents);
So to recap, it is possible, but it is only legal via the API I linked above.
Want to develop a query refinement tool just like Google Suggest so that i will get recommended options when i enter keywords in the search textbox i.e( if i type "car"
i will get the recommended options as rental cars,used cars,cars for sale etc) want to develop the tool in c#/ASP.Net dont know where to start.please give some suggestions.
This can be achieved by using javascript and let's say some web service / page method. You send the values that user gave you to a web service. The web service translates the text, make some database (or other storage) request and return the results to the page as json (or some other format that suits you).
Look at this article if you need some example code:
http://www.dotnetcurry.com/ShowArticle.aspx?ID=515
By the way, if you need more articles, just put "asp.net autocomplete" into Google. That should give you plenty of resources.
im starting the pseudo code of a new site, and want it to be as SEO friendly as possible.
the site i am creating is a booking agency site with c# and asp.net. essentially bands will register on the site with their availability and other info, and fill out their profile information with images etc. this info will be stored in a db.
creating this is not a problem, but i want the site to be a SEO friendly as possible.
I know google loves huge sites with great content. And all of these profile pages would be an excellent addition to my site for seo purposes. i also hear that google cannot see dynamically generated content when crawling a site.
i want to find a method of coding these pages, so google can see the content when it crawls them.
i need a pointer in the right direction for a solution for this. nothing is off limits - i will basically code my entire site around this principle, i just have no idea where to start looking for a solution. im not looking for a code solution, just what i should be researching to solve this issue.
Thanks in advance
i also hear that google cannot see dynamically generated content when crawling a site.
Google can see anything you can retrieve via http GET request (ie: there's a specific URL for it) and that someone either linked to or is listed in a published xml site map file.
To make sure that your profile pages fit this, you will want to make sure that profiles are all rendered via a single asp.net *.aspx file that determines which page is shown via a url parameter. Something that looks like this:
http://example.com/profiles.aspx?profile=SomeBandName
Now, you probably also want a friendly URL, that looks like this:
http://example.com/profiles/SomeBandName
To do that, you need to set up routing.
In order to crawl and index your pages by google or other search engine properly. Follow the following guidelines.
i: Page title must be precise and according to content available in page.
ii: Page url should be user friendly.
iii: Content is king (useful content)
iv: No ajax or javascript oriented way to load contents.
v: No flash or other media files. if exist must have description via alt tag.
vi: Create url sitemap of all static and dynamically generated contents.
vii: Submit sitemap to google and keep tracking how google crawl and index your pages.
fix issues contineously if google found via crawling.
In this way your most pages and content will be index properly and fastly.
I'd look into dynamic URL Rewriting.
Basically instead of having one page say http://localhost/Profile.aspx you'll have a bunch of simulated urls like
http://localhost/profiles/Band1
http://localhost/profiles/Band2
http://localhost/profiles/Band3
etc.
All of those will then map to back to the orgial profile.aspx page with a parameter so internally in your code it would look like http://localhost/Profile.aspx?Name=Band1, http://localhost/Profile.aspx?Name=Band2, etc
Basically your website appears to have a bunch of pages for each band but in reality they are all getting mapped back to the same asp.net page but have different parameters.
This is article I read about it some time back. http://weblogs.asp.net/scottgu/archive/2007/02/26/tip-trick-url-rewriting-with-asp-net.aspx
i also hear that google cannot see dynamically generated content when crawling a site.
you could create a sitemap.xml with the urls pointing to the dynamic profile pages. using google webmaster tools you can submit and monitor the crawling progress.
you may also create an index page or something similar ('browse by category' pages) that link to matching profile pages.
a reference for seo I regularly use is http://www.seomoz.org/learn-seo
I'm working on a program in Visual C#.NET and I need some help.
I need it to be able to take in some text through a text box, then somehow send that text to google, and bring back the resulting URLs (not the full results, just the URLs) and then display those in my program. How would I do that?
Use the WebClient class to send the query to Google and read the response.
Alternatively, use a .NET library that interacts with the Google search API, like this one (this was just the first Google result).
There are also REST libraries for .NET, if you go with the newer custom search.
Unfortunately the Google Web Search API is deprecated and no longer available. However the next best thing IMO is Google Custom Search Engine.