I have implemented a method which manually scrapes the Search Twitter page and gets the tweets on different pages.
But since there is a fast refresh rate, the method triggers an exception.
Therefore I have decided to use TweetSharp API instead
var search = FluentTwitter.CreateRequest()
.AuthenticateAs(TWITTER_USERNAME, TWITTER_PASSWORD)
.Users()
.SearchFor("dumbledore");
var result = search.Request();
var users = result.AsUsers();
this code was on the site.
Does anyone know how I can avoid giving my credentials and retrieve from all users and not just the ones I have as friends?
Thanks!
What you want to do is interface with the Twitter Streaming API. This API allows you to open a persistent connection with Twitter and Twitter will then stream results to you as they come in.
(taken from the Twitter Streaming API page)
That said, TweetSharp doesn't currently support the Streaming API. However, it's not difficult to open a connection to Twitter in .NET and process the responses as they're received (however, I'd recommend using the HttpClient class to process this asynchronously, as well as using a proper JSON parsing library, like Json.NET).
Note the third column in the diagram "Streaming connection process", specifically the middle part:
Receives streamed Tweets, performs processing and stores result
As well as the "HTTP Server process" column:
Server pulls processed result from data store and renders view.
While not explicitly mentioned, you are best off just persisting the Tweet as you get it into a data store and then having another process handle the Tweets; the volume of Tweets you might get is so high that performing any processing when you get the Tweet will backlog the receiving of new Tweets.
For your specific case, you'll want to access the Public Streams with a POST filter of "dumbledore".
Related
I am trying to get all threaded messages from a group of yammer, but it seems that something wrong happened in the call.
The retrive all messages from the group I use this call:
/api/v1/messages/in_group/{groupId}.json?threaded=true
And when this call ends, I save the last message retrieved from the result and I execute the following call recursively until there is no more messages in the group:
/api/v1/messages/in_group/{groupId}.json?threaded=true&older_than={messageId}
It seems that the process works correctly, but when you look the data that you have retrieved from Yammer, there is some messages that appear in the Yammer group Wall that have been not retrieved using the REST API.
Do someone know why the REST API is not getting all Yammer data?
Thank you so much!
Aleu
From the docs https://developer.yammer.com/docs/messagesjson, here is the intended functionality of threaded=true: "threaded=true will only return the thread starter (first message) for each thread. This parameter is intended for apps which need to display message threads collapsed. threaded=extended will return the thread starter messages and the two most recent messages all ordered by activity, as they are viewed in the default view on the Yammer web interface."
Based on your question, perhaps threaded=extended would give you what you need.
I have a pretty big video file I upload to a web service via multipart/form-data.
It takes ~ 30 seconds to arrive and I would prefer not waiting that long simply to access parameters I send along with the file.
My question is simple, can I access parameters sent with the form without waiting for the video payload to be uploaded?
Can this be done using headers or any other methods?
Streaming vs. Buffering
It's about how the webserver is set up. For IIS you can enable Streaming.
Otherwise, by default, IIS will use 'buffering' - the whole request is loaded into memory first (IIS's memory that you can't get to) before your app running in IIS can get it.
Not using IIS? You have to figure out how to get the webserver to do the same thing.
How to stream using IIS:
Streaming large file uploads to ASP.NET MVC
Note the way the file is read in the inner loop:
while ((cbRead = clientRequest.InputStream.Read(rgbBody, 0, rgbBody.Length)) > 0)
{
fileStream.Write(rgbBody, 0, cbRead);
}
Here instead of just saving the data like that question does, you will have to parse any xml/json/etc or whatever contains the file parameters you speak of ... and expect the video to be sent afterwards. You can process them right away if it's a quick process ... then get the rest of the video ... or you can send them to a background thread.
You probably won't be able to parse it just dumping what you have to a json or xml parser, there will be an unclosed tag or } at the top that isn't closed til after the video data is uploaded (however that is done). Or if it's multipart data from a form submission, as you imply, you will have to parse that partial upload yourself, instead of just asking IIS for the post data.
So this will be tricky, you can first start by writing 1k at a time to a log file with a time stamp to prove that you're getting the data as it comes. after that it's just a coding headache.
Getting this to work also means you'll have to have some control over the client and how it sends the data.
That's because you'll at least have to ensure it sends the file parameters FIRST!
Which concerns me, because, if you have control of the client, why can't you take the simple route (as Nobody and Nkosi imply) and use 2 requests? You mention you need one. Why not write js client code to send the parameters first in an XHR and then the file in a second request, using a correlation ID in both to tie them together? (the server could return this from the first request and you could send it in the 2nd).
Obviously, if you're just having a form with some inputs and a file upload and doing submit, then you need one request ;-) But if you have control over the client side you're not stuck with that.
Good luck, there is some advanced programming here, but nothing super high-tech. You will make it work!!!
If you don't have control over the server code, you are probably stuck, if the server app's webserver is buffering, the server app won't get anything, of course, if you wanted to do something with the file parameters first, this really implies you have control of the server side ;-)
I've searched some time, looking for easy way to connect with some other sites WebAPI. There are some solutions, but they are made in very complicated way.
What I want to do:
Connect with server using URL adress
Provide login and password to get some data
Get data as JSON/XML
Save this data in an "easy-to-read" way. I mean: save it to C# variable which could be easy to modify.
Currently, API that I want to work with is Bing Search, but I'm looking for some universal way. I found an example, but it doesn't work for me and in my app I can't use this class: "DataServiceQuery" because it doesn't exsist.
How do you usually do it? Do you have your favourite solutions? Are there some universal ways or it depends on type of API that you work with?
I'm currently working on .NET MVC app (in case it could make any difference)
From server side
You can use that like below.
// Create an HttpClient instance
HttpClient client = new HttpClient();
// Send a request asynchronously continue when complete
client.GetAsync(_address).ContinueWith(
(requestTask) =>
{
// Get HTTP response from completed task.
HttpResponseMessage response = requestTask.Result;
// Check that response was successful or throw exception
response.EnsureSuccessStatusCode();
// Read response asynchronously as JsonValue
response.Content.ReadAsAsync<JsonArray>().ContinueWith(
(readTask) =>
{
var result = readTask.Result
//Do something with the result
});
});
You can see example on following link.
https://code.msdn.microsoft.com/Introduction-to-HttpClient-4a2d9cee
For JavaScirpt:
You could use jQuery and WebAPI both together to do your stuff.
There are few steps to it.
Call web api with Ajax jquery call.
Get reponse in JSON
Write javascript code to manipulate that response and do your stuff.
This is the easiest way.
See following link for reference:
http://www.codeproject.com/Articles/424461/Implementing-Consuming-ASP-NET-WEB-API-from-JQuery
It entirely depends on the type of API you want to use. From a .Net point of view, there could be .Net 2 Web Services, WCF Services and Web API Services.
Web APIs today are following the REST standard and RMM. Some APIs need API Keys provided as url parameters, others require you to put in request's header. Even some more robust APIs, use authentication schemes such as OAuth 2. And some companies have devised their own standards and conventions.
So, the short answer is that there is no universal way. The long answer comes from documentation of each API and differs from one to another.
I got one question. I am going to use RESTfull web service in my mobile app(which is based on C#, monotouch). I have to send big data using either json or xml. Can I make sure that in case connection is lost I want to stop pulling data and cancel rest of the data which has been pulled.
Basically. my target is that I don't want to pull partially. I need whole data without loosing it.
Any other idea how can I achieve this?
Just wanted to ask another question. How can I secure my url which refer to data. If I use RESTfull webservice
If you use NSURLConnection as follows
NSURLConnection *theConnection=[[NSURLConnection alloc] initWithRequest:[NSMutableURLRequest requestWithURL:[NSURL URLWithString:#"http://www.google.com"]] delegate:self];
Then, you can use the following two delegate methods to check if the data downloaded fully.
-(void)connection:(NSURLConnection *)connection didFailWithError:(NSError *)error {
//Download failed
}
-(void)connectionDidFinishLoading:(NSURLConnection *)connection {
//Download success
}
I've been searching for help with screen scraping for the windows phone 7 but cannot find any help relevant to what i want. The basis of my application is to take a phone number typed in an input box on the device -> pass it to a website's searchbox -> search the website's SQL database -> pass the raw results back to the phone and display them in a table.
I have permission from the website owner to use his online database for this purpose.
Is this possible and, if so, how would I go about doing this?
Thanks in advance!
EDIT: After some extra research I've found that with using the POST method I can send the data needed to the search box on the website and the results are successfully found but I am unsure on how to display the results onto the application itself? I know the data is successfully sent via packets viewed in WireShark. Thanks again.
Code for POST:
InitializeComponent();
Dictionary<string, object> parameters = new Dictionary<string, object>();
parameters.Add("search_name", "Google"); //Test Search
parameters.Add("submit", "Search");
PostClient proxy = new PostClient(parameters);
proxy.DownloadStringCompleted += (sender, e) =>
{
if (e.Error == null)
{
//Process the result...
data = e.Result;
}
};
proxy.DownloadStringAsync(new Uri("http://www.SITE.com/search.php", UriKind.Absolute));
webBrowser1.Navigate(new Uri(, UriKind.Absolute));`
Rather than using the website's UI and screen scraping the results, I would create an HTTP request similar to or the same as the request generated by the web page (this will probably be a POST request containing form data). I would then send this to the web server and use something like HtmlAgilityPack to parse the required data from the response.
Effectively, the website presents you with an HTTP API using HTML as the message format. Use this directly, rather than using a client side rendering of these messages, which ultimately is designed for user interaction rather than code interaction.
As far as I know it is not possible/allowed to access other applications running on the Windows Phone.
So this is not possible unless the publishers of the other applications connect to your website/webservice.
NB: When Windows Phone 8 come out and it is the same as or similar to Windows 8 there might be contracts available that allow the wiring up of application like this. (This is just guessing)