I've been searching for help with screen scraping for the windows phone 7 but cannot find any help relevant to what i want. The basis of my application is to take a phone number typed in an input box on the device -> pass it to a website's searchbox -> search the website's SQL database -> pass the raw results back to the phone and display them in a table.
I have permission from the website owner to use his online database for this purpose.
Is this possible and, if so, how would I go about doing this?
Thanks in advance!
EDIT: After some extra research I've found that with using the POST method I can send the data needed to the search box on the website and the results are successfully found but I am unsure on how to display the results onto the application itself? I know the data is successfully sent via packets viewed in WireShark. Thanks again.
Code for POST:
InitializeComponent();
Dictionary<string, object> parameters = new Dictionary<string, object>();
parameters.Add("search_name", "Google"); //Test Search
parameters.Add("submit", "Search");
PostClient proxy = new PostClient(parameters);
proxy.DownloadStringCompleted += (sender, e) =>
{
if (e.Error == null)
{
//Process the result...
data = e.Result;
}
};
proxy.DownloadStringAsync(new Uri("http://www.SITE.com/search.php", UriKind.Absolute));
webBrowser1.Navigate(new Uri(, UriKind.Absolute));`
Rather than using the website's UI and screen scraping the results, I would create an HTTP request similar to or the same as the request generated by the web page (this will probably be a POST request containing form data). I would then send this to the web server and use something like HtmlAgilityPack to parse the required data from the response.
Effectively, the website presents you with an HTTP API using HTML as the message format. Use this directly, rather than using a client side rendering of these messages, which ultimately is designed for user interaction rather than code interaction.
As far as I know it is not possible/allowed to access other applications running on the Windows Phone.
So this is not possible unless the publishers of the other applications connect to your website/webservice.
NB: When Windows Phone 8 come out and it is the same as or similar to Windows 8 there might be contracts available that allow the wiring up of application like this. (This is just guessing)
Related
My question is rather simple, but I can't find any information about this on the internet.
I am developing a windows phone application and I want to use the web api (from MVC 4) to get, set, and update.
I already made all the GET methods and they work fine. My question is: How can I perform a POST from a url (and add data to my database)?
Something like this: http://someurl.com/api/post/username/parameter1/parameter2
Is this even possible? And otherwise how else can I resolve this problem?
Simply use RestSharp for all your WebApi work in Windows Phone.
Believe me when I say that it will save you development time!
(To say the truth, I almost never use WebRequest directly in my apps, and just go ahead with RestSharp...)
There are two alternatives:
WebClient.UploadStringAsync
WebClient client = new WebClient();
client.UploadStringCompleted += OnUploadStringCompleted;
client.UploadStringAsync(new Uri("http://someurl.com/api/post/username/parameter1/parameter2", UriKind.Absolute), "Data to upload goes here");
or
HttpWebRequest.BeginGetRequestStream
There's a complete example on this MSDN page.
Hope that helps!
I'm sorry if this subject has already been answered, but I couldn't find what I needed (yet).
I'm working on a program that downloads files from university websites that use the same infrastructure. It's an open source project which I'm trying to support in my free time
(hosted in goodle code: http://code.google.com/p/highlearner/)
Until now we used GET and POST requests to login into the right page and download stuff. But the universities keep changing their websites and every little change requires teaking in Highlearner, which requires a new version, auto-updating all users, etc. Also, every university has its own login page, requiring me to tailor a login sequences..
So I'm looking for a more robust solution. Instead of manually redirecting and setting the HTTP parameters. Is there some kind of mini browser that supports with HTML + Javascript? No GUI is needed, I just need the engine.
This way, I will simply need to fill out the form parameters and let the browser do the work.
Thanks,
Nitay
You could try to automate the process with WatiN library . It allows you to click buttons, submit forms, etc.
using (var ie = new IE(loginUrl))
{
if (ie.TextField("username").Exists
&& ie.TextField("password").Exists)
{
ie.TextField("username").Value = "username";
ie.TextField("password").Value = "password";
ie.Button(Find.ByName("submit")).Click();
}
}
I hope the question is self-describing.
I'm currently developing an asp.net website which uses a MS SqlServer database in the data layer.
And I was thinking what are my options to get a mobile version (most importantly supports BlackBerry and iPhone and hopefully every mobile device!) and when used on blackberry I want to be able to let it run at the BB's background.
I was thinking about asp.net mobile controls but the projects page seems like a dead/not-updated framework and not sure exactly if supports only windows mobiles or what!
Edit
Thank you for your questions, but they all covered my problem from only one respective .. I mean how this is going to let me use the BlackBerry Appliction options like letting my website run at the device background or sending notifications to my users!
This is mostly going to be a product of styling. Mobile websites work just like regular websites these days, except you want to use CSS and images that work well on a mobile device. You can use a product like 51 Degrees that will give you a bunch of information on what type of device is connected, so you can customize your output based on resolution or any number of other things if you so desire.
You could also try a book on mobile design, such as "Mobile Web Design" by Cameron Moll.
If you use ASP.Net MVC to create your app and create regular and mobile views. You can use jQuery Mobile to help with the mobile views too.
This question covers how to change your view based on the device type,
If you use WebForms, you can change your MasterPage depending on the browser thus giving you the ability to swap to mobile versions more easily:
protected void Page_PreInit(object sender, EventArgs e)
{
if (Request.Browser.IsMobileDevice)
MasterPageFile = "~/Mobile.Master";
}
Or use a Global.asax to redirect mobile requests completely:
void Session_Start(object sender, EventArgs e)
{
// Redirect mobile users to the mobile home page
HttpRequest httpRequest = HttpContext.Current.Request;
if (httpRequest.Browser.IsMobileDevice)
{
string path = httpRequest.Url.PathAndQuery;
bool isOnMobilePage = path.StartsWith("/Mobile/",
StringComparison.OrdinalIgnoreCase);
if (!isOnMobilePage)
{
string redirectTo = "~/Mobile/";
// Could also add special logic to redirect from certain
// recognized pages to the mobile equivalents of those
// pages (where they exist). For example,
// if (HttpContext.Current.Handler is UserRegistration)
// redirectTo = "~/Mobile/Register.aspx";
HttpContext.Current.Response.Redirect(redirectTo);
}
}
}
Either way read this article: http://www.asp.net/learn/whitepapers/add-mobile-pages-to-your-aspnet-web-forms-mvc-application
You don't really need to do anything special; Just create an alternative stylesheet that is optimized for 320px width viewport. You can serve this stylesheet through a separate stylesheet using the "media" attribute of the LINK element, or you can use CSS Media Queries within your mater stylesheet. Some relevant info:
http://googlewebmastercentral.blogspot.com/2011/02/making-websites-mobile-friendly.html
http://www.css3.info/preview/media-queries/
If you are using asp.net MVC be sure to check out
Web Application Toolkit for Mobile Web Applications
I have implemented a method which manually scrapes the Search Twitter page and gets the tweets on different pages.
But since there is a fast refresh rate, the method triggers an exception.
Therefore I have decided to use TweetSharp API instead
var search = FluentTwitter.CreateRequest()
.AuthenticateAs(TWITTER_USERNAME, TWITTER_PASSWORD)
.Users()
.SearchFor("dumbledore");
var result = search.Request();
var users = result.AsUsers();
this code was on the site.
Does anyone know how I can avoid giving my credentials and retrieve from all users and not just the ones I have as friends?
Thanks!
What you want to do is interface with the Twitter Streaming API. This API allows you to open a persistent connection with Twitter and Twitter will then stream results to you as they come in.
(taken from the Twitter Streaming API page)
That said, TweetSharp doesn't currently support the Streaming API. However, it's not difficult to open a connection to Twitter in .NET and process the responses as they're received (however, I'd recommend using the HttpClient class to process this asynchronously, as well as using a proper JSON parsing library, like Json.NET).
Note the third column in the diagram "Streaming connection process", specifically the middle part:
Receives streamed Tweets, performs processing and stores result
As well as the "HTTP Server process" column:
Server pulls processed result from data store and renders view.
While not explicitly mentioned, you are best off just persisting the Tweet as you get it into a data store and then having another process handle the Tweets; the volume of Tweets you might get is so high that performing any processing when you get the Tweet will backlog the receiving of new Tweets.
For your specific case, you'll want to access the Public Streams with a POST filter of "dumbledore".
Please consider the following scenario,
There are two web applications App1 & App2. A user would submit his information on App1 though a form. On click of a specific button/link on App1, the same data should be posted to a page on App2 and the user should also be redirected to the same page on App2.
I would like some help in finding out the best way to implement this functionality.
One of the approaches that I have already tried out is by creating a temporary HTML form at runtime, setting the action attribute of the form to the App2 Page and get the form posted by using javascript submit. The data can then be fetched on App2 page by using the response.form object.
This approach works well, but i was still wondering if there is any other way to implement the required functionality.
I would really appriciate if you can give some insights on using RESTful webservices to implement this, or else, using some HttpModule to intercept requests at App1 and modify redirect response to app2 or any other approach that you might find fit for the purpose.
Edit:
Using querystring isnt an option for me.
I've had a need to do similar things with feed agregation and building rss feeds from web page content on different domains.
User Gets app1 page, fills in details and submits then on the server for app1 I have a method that looks like this ...
HTMLDocument FetchURL( string url )
{
WebClient wc = new WebClient();
string remoteContent = wc.DownloadString(url);
// mshtml api is very weird but lets just say you have to do things this way ...
HtmlDocument doc = new HTMLDocument();
IHTMLDocument2 doc2 = (IHTMLDocument2)doc;
doc2.write(new object[] { remoteContent });
return (HTMLDocument)doc2;
}
This function does 2 things of use ...
It gets the page of content at "url"
It parses that content in to a HTMLDocument object
Once you have this function you can then call it passing it the url to the remote page and get back a html doucment.
The functions in the HTMLDocument object will allow you to do javascript like dom queries such as :
docObject.GetElementById("id");
I then have different functions that do different things with this object based on the page / site i'm returning data from.
There is however one fatal flaw here ...
This is likely to work really well with sites that don't change much in structure and are built by code but not so well on less dynamic sites.
With stackoverflow for example its easy to pull out a question and the accepted answer for that question so I could use this code to pull and publish content from here on my own web site.
However ...
This is not going to help you for user / login related details as this sort of information is not shared to generally everyone.
It's bit like me going and trying this to link facebook profiles to my own website, I would have to go through some form of api that asked the user to authenticate their details before making the request.
simply pulling a web page based on a url only will give the other site no authentication information unless that site accepts the user login details in the quesrystring and you already have them.
You may however be able to chain requests by ripping apart my sample method, requesting the login page parsing the results, filling in the form, then posting back using the same web client instance to login then requesting the url.
The idea being that you would have a form that asks the user to put in their login details for the remote site on your site then you go and find their profile page based on that.
This would be best farmed out to a class rather than just a simple method like i have here.
In my case though i was only after something simple (the bbc top 40 uk charts) which i pulled information from not only the bbc but places like amazon, google, and youtube, then i built a page :)
It's neat but serves no functional purpose other than pulling all your other fave sources of info on to 1 page.
If you are already committed to using javascript, then why not an ajax post, and change the window.location based on the response?
You can use HttpServerUtility.Transfer this will preserve your form contents and transfer the user to the new page.
http://msdn.microsoft.com/en-us/library/system.web.httpserverutility.transfer.aspx
I have built something like what you are describing, and I found that using a <form> tag to POST to app2 is the most reliable way... basically, the way you found that worked well.
If App2 is residing on a different domain, it's usually best to create your own interface for the submission, and have that interface handle the posting from App1 to App2.
(Browser) -> Submits form to App1 ->
(App1) -> validate input
-> stores local info
-> creates an HttpRequest/POST object
-> posts to App2
(App2) -> handles the post
<- returns the response
-> confirms the results of App2
<- returns the results to the browser.
In essense, you want to control and proxy requests from your Applications domain to any outside interfaces as much as possible.
Note: I'm answering my own question
just to have a correct answers marked
against it. All the suggestions
provided by various members here are
correct in their own way, but they
were not apt for my requirements.
Hence, I cant accept any of them as
correct.
The way I have Implemented is by creating a custom control which would have a configurable property containing the URL to post data and another one accepting a dictionary object as the data input to be posted.
This control would internally create a HTML form with action attribute set to the URL specified by the user and have the data feilds created out of the dictionary object. This form would then be posted on the button click event on the page hosting this control.