I am writing a very simple RSS reader - all it needs to do is get the xml doc, and print to the console the title and publish date of every item. I got started using these two questions:
How can I get started making a C# RSS Reader?
Reading the Stack Overflow RSS feed
I'm trying to figure out how to subscribe, and as far as I can figure you do it one of two ways. Send an HTTP request to the feed site so it pushes you updates as they come, or poll the site every X seconds and simply print the new ones.
I find it difficult the believe that there is no way to subscribe due to the millions of RSS readers running at any given moment, popular RSS sites like facebook, twitter, or myspace would be hit hundreds of millions of times per second due to all the RSS readers "subscribed" to it and look like a DOS attack.
So what is the "standard" way to subscribe to an RSS feed, if such a standard truely exists?
The standard way is to poll. Not every x seconds but every x minutes or x hours.
The reasoning behind RSS is to keep the feed extremely simple. Small download and the same file can be served to all subscribers (easy to cache in memory and no processing overhead to find out exactly what and when to send to each client).
Not sure you quite understand the concept of RSS feeds.
It is simple:
You application (RSS reader) sends an HTTP GET request to given RSS feed url.
You get XML in return.
You parse that XML and show that data on your UI.
And generally, the websites you mentioned are smart enough to identify DOS attacks (for example, frequent requests from same IP in very short time). So, you don't have to worry about that.
Also, while creating an RSS reader, every time you get new XML from feed url, you have to identify new posts from old ones (that you already have on your UI). Timestamps are generally used to identify posts, but, there no standard way of doing that.
RSS on a site / server does not manage any suscriptions. The suscription is only a concept in the RSS reader. That keeps stuff simple on the RSS server side, as there's no need for suscription management which made the protocol easy to adopt.
You have to periodically poll the RSS feed by an HTTP GET to the feed URL. You get a XML document in the RSS format in return. Then you parse it and display the infos you like. Voila.
Related
How do I retrieve RSS Feeds based on a date range?
Specifically, how do I prepare the url so that I can get items that were published past a certain date?
www.pwop.com/feed.aspx?show=dotnetrocks&filetype=master&tags=Craftsmanship
Your questions is more related to the HTTP API of the site, not RSS it self.
RSS is a predefined XML data format.
Most RSS urls doesn't support filters and introduce simple URL which returns in RSS format the last X results (x is usually between 10 to 50 results).
Some URL allow to specify categories or Tags like in your example, so the reutrn RSS XML will contain only results from this tags.
If you don't want to miss results, you need to keep query the RSS URL every X minutes/hours depends on the update speeds of the results.
Other option is to contact the site and request a full API access or even to implement a feature to filter by date.
Not all websites support it, but maybe there is a solution that can work:
Websites usually have a sitemap.xml (or sitemap.xml.gz or sitemap.gz) file that contains all the urls in bulk or grouped in some way (e.g., by category, tag, month). The sitemap.xml can contain links to additional xmls and so on.
The main sitemap is typically located in the root of the site (e.g., https://news.bitcoin.com/sitemap.xml), but you can find more information about sitemaps here: https://www.sitemaps.org/protocol.html.
If a website has such an xml file, perhaps processing it will make it easier to extract the needed information without any special site crawler or API.
I am thinking now how the rss applications are parsing the feeds, basically if I just want to parse XML from the feed I will use XMLReader
http://content.warframe.com/dynamic/rss.php
Based on this feed I will get exception about illegal path (this is less important), BUT I can put this link to another application (link at the bottom) and it will work...
There is w3c validator which shows many error
http://validator.w3.org/feed/check.cgi?url=http%3A%2F%2Fcontent.warframe.com%2Fdynamic%2Frss.php
If this rss feed really have so many "errors", why it is working with other applications? On example when i put it into:
http://feed.mikle.com/
We are trying to find out any technologies/libraries available in .NET stack (even wrappers on top of 3rd party dlls) that'll help us to build an app that can
1 - Capture an image from a user's video device
2 - Upload it realtime to a server
3 - Process the video (in the server) - eg: Adding a watermark to the video
4 - Stream it back to the user/other users
Preferably, the time delay/latency between step2 and 4 should be minimal
The first requirement (capturing) seems pretty straight forward. The challenge is identifying a suitable way to do the upload, do the processing, and stream it back. Any valid suggestions or ideas?
Recently came acrsoss FFmpeg library, and it has a C# wrapper. Does FFmpeg can be used to do the processing side?
I would go about it this way:
Use silverlight or flash to capture the video camera input, e.g. as detailed here.
You can send the byte-stream over a socket that your server is listening to.
On the receiving end, just use the socket-accepting program as a router-program with a number of listening workers connected. Between workers and router-program, e.g. AMQP with RabbitMQ. Send asynchronous messages (e.g. with reactive extensions) with e.g. the stream encoding to the rabbit-node, which then can either further all messages to one single computer as a part of a conversation/user-session, or interleave between the available workers. Here's the manual. As the video is encoded, it is streamed asynchronously over the message bus back. According to intel tests the bus itself should work well at high throughputs, but they had to use the interleaved tcp channel mode (they tested on a gigabit lan). Other users here have suggested FFlib. You might also look into having the workers convert into webM, but if FFlib works, that might be a lot easier. Each worker publishes over AMQP the next encoded video piece. A server-running program, e.g. the router program I talked about before, starts sending to the client (see no. 4)
Have a client-program, e.g. silverlight/flash connect (for example over the same socket that you opened for client->server data, or over HTTP), and read the byte-stream with a decoder. Render the output.
VideoLab from Mitov can accomplish all of this and is free for personal use (not so free for commercial use, but pricing is not too heavy).
I have bought and use the Delphi version and know it works extremely well, so I'm pretty sure the .NET version will do what you need.
This kind of task is not trivial (as seen by the lack of responses here), so expect to struggle considerably with DirectX/Microsoft Media Encoder- but with this toolkit and some help from the author, you will eventually succeed.
http://www.mitov.com/html/videolab.html
It seems that Splicer can process static video and convert it - I'm not sure about processing a realtime uploaded video - http://splicer.codeplex.com/
Take a look at Video.Show by Vertigo. It's an open source website for user-generated video content. It uses the Expression Encoder to handle compression/video editing. It's not exactly what you need, but it's a good start!
You could use Silverlight for capture as is mentioned above, and then use Expression Encoder to push it to a stream server or stream from there directly.
It should have everything you need:
Smart encoding/smart recompression for
WMV if the source is also WMV and no
frame operations are performed [4],
cuts editing, serial batch encoding,
Live encoding from webcams and DV
camcorders
Decoding/import format support because
of DirectShow
Smooth streaming (720p+ video using
HTTP) with optimized client
(Silverlight) and server (IIS with
smooth streaming)
WebDAV publishing, publishing plugins
for Silverlight Streaming, Amazon S3
Importing XAML overlays created in
Expression Design and customizing
their timing, animation, opacity,
placement and looping
JavaScript trigger events
Windows Media 11 SDK and VC-1 SDK
integration, native MPEG-2 decoder
Adding captions to videos using SAMI
or W3C Timed Text format
Previewing and comparing encoding
settings in real time
Screen capture
Object model for the encoding engine,
SDK downloadable separately
The question is kind of short on details (is this a web server, what os is the server? etc) but I'll take a stab based on what I think you're trying to do.
One thing you might consider is doing the capture and process at one time. If the user is running your client app, have that do the capture and processing via DirectShow. Then all you need to do is upload the video and you can skip the entire server process. This is assuming that the 'user' is under your control - that this is not some random person out there uploading video, but an employee or someone otherwise trusted.
If this isnt the case, then ffmpeg can certainly be used to watermark video on your server. You dont really need 'wrappers' for it. You can just call it as a command line app from your server application and wait for it to finish.
The process really isnt that complex... its the details that are going to matter (for example - what does 'stream' mean to you? Do you really mean 'stream', or is this via http? Thats a huge topic right there)
I have seen many of website are displaying RSS Feeds on their website.
Example:
1) compgroups.net
2) velocityreviews.com
3) bytes.com
4) eggheadcafe.com
And many other websites.
What i observe is Google is even giving them good rank despite of duplicate content.
What i want to know is...
How can I find RSS Feeds? Also where can i found RSS Feeds for Yahoogroups?
Read here: http://help.yahoo.com/l/ca/yahoo/groups/rss/rss-03.html
The source of an RSS feed is XML. You can request a feed from c# using the HttpRequest, providing the url of the feed.
You read in the XML, process it and show its contents in your webpage.
I want to read all items of a feed in C#. The solutions I've found are only for the latest items like just the last 10 days.
Anybody has a good solution for this?
Libraries for reading feeds typically read all the data in the feed, but feeds typically only contain recent data - you need a source of data that includes older items, not a better library for reading the data you have.
Most entities publish feeds to allow people to track when new content is published, not to make all their data available in a more convenient machine readable format. For this purpose, publishing recent data only makes sense as it saves on bandwidth.
If you can tie into something like Google Reader, which archives old feed items (although I'm not sure it's a permanent archive or not), then perhaps you can accomplish this.
Most RSS feeds are only written to deliver a relatively short period of time - 'all' items in a feed generally need you to have created your own archive over time.
Extending thomas' answer, The two google-related archives of feed data you can find are the official one:
Google AJAX Feed API
http://code.google.com/apis/ajaxfeeds/
which will limit you to 250 items, and the unofficial one:
Google Reader API
http://www.niallkennedy.com/blog/2005/12/google-reader-api.html
which will give you unlimited (i think) items but you will need to work around their authentication (something with cookies) and pray they don't change or drop the API (as it is undocumented).
I tried Google Reader, but ther archive was incomplete. I know the people who run the blogs, so I just asked them for a export.