We Keep Coding

Buffer size is not sufficient or queue is full when making many Http Requests

I am currently trying to get a lot of data about video games out of Wikipedia using their public API. I've gotten some of the way. I can currently get all the pageid I need with their associated article title. But then I need to get their Unique Identifiers (Qxxxx where x are numbers) and that takes quite a while...possibly because I have to make single queries for every title (there are 22031) or because I don't understand Wikipedia Queries.
So I thought "Why not just make multiple queries at once?" so I started working on that, but I've run into the issue in the title. After the program has run for a while (usually 3-4 minutes) about a minute passes then the application crashes with the error in the title. I think it's because my approach is just bad:
ConcurrentBag<Entry> entrybag = new ConcurrentBag<Entry>(entries);
Console.WriteLine("Getting Wikibase Item Ids...");
Parallel.ForEach<Entry>(entrybag, (entry) =>
{
entry.WikibaseItemId = GetWikibaseItemId(entry).Result;
});
Here is the method that is called:
async static Task<String> GetWikibaseItemId(Entry entry)
{
using (var client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate }))
{
client.BaseAddress = new Uri("https://en.wikipedia.org/w/api.php");
entry.Title.Replace("+", "Plus");
entry.Title.Replace("&", "and");
String queryString = "?action=query&prop=pageprops&ppprop=wikibase_item&format=json&redirects=1&titles=" + entry.Title;
HttpResponseMessage response = await client.GetAsync(queryString);
response.EnsureSuccessStatusCode();
String result = response.Content.ReadAsStringAsync().Result;
dynamic deserialized = JsonConvert.DeserializeObject(result);
String data = deserialized.ToString();
try
{
if (data.Contains("wikibase_item"))
{
return deserialized["query"]["pages"]["" + entry.PageId + ""]["pageprops"]["wikibase_item"].ToString();
}
else
{
return "NONE";
}
}
catch (RuntimeBinderException)
{
return "NULL";
}
catch (Exception)
{
return "ERROR";
}
}
}
And just for good measure, here is the Entry Class:
public class Entry
{
public EntryCategory Category { get; set; }
public int PageId { get; set; }
public String Title { get; set; }
public String WikibaseItemId { get; set; }
}
Could anyone perhaps help out? Do I just need to change how I query or something else?

Initiating roughly 22000 http requests in parallel from one process is just too much. If your machine had unlimited resources and internet connection bandwidth, this would come close to a denial-of-service attack.
What you see is either TCP/IP port exhaustion or queue contention. To resolve it, process your array in smaller chunks, for example fetch 10 items, process those in parallel, then fetch the next ten, and so on.
Specifically Wikimedia sites have a recommendation to process requests serially:
There is no hard and fast limit on read requests, but we ask that you be considerate and try not to take a site down. Most sysadmins reserve the right to unceremoniously block you if you do endanger the stability of their site.
If you make your requests in series rather than in parallel (i.e. wait for the one request to finish before sending a new request, such that you're never making more than one request at the same time), then you should definitely be fine.
Be sure to check their API terms of service to learn whether and how many parallel requests would be in compliance.

SignalR chat room with Acknowledgement of message received after saving it to Sql data Base

Hello every one I am new to signalR I need little help to follow right approach in my chat module .
My Refernce :
http://www.codeproject.com/Articles/562023/Asp-Net-SignalR-Chat-Room but this article does not uses dataBase.
Steps which I have used to build my chat module are :
1 - Created chatHub class inherited from Hub class.
2 - On Connect(string enryptedId) function in chatHub class I am adding to List<UserDetail>.
3 - On SendPrivateMessage(string toData) I am saving it to database if it saved successfully to dataBase without any exception then sending it to both sender and reciever and binding on their communication messages <div>.
Problem in this approach - If after saving it to database if sender got disconnected due to network problem then sender will not recieve message from chathub class to client function so message is not appended to <div> which shows user communications but actually message sent successfully. Please can any one tell me the right approach to do this.

I would do it something like this, keep in mind its Pseduo code not tested or anything
private Dictionary<Guid, TaskCompletionSource<bool>> transactions = new Dictionary<Guid, TaskCompletionSource<bool>>();
public Task SendPrivateMessage(string content)
{
var taskCompletion = new TaskCompletionSource<bool>();
var transactionId = Guid.NewGuid();
transactions[transactionId] = taskCompletion;
var message = new Message
{
TransactionId = transactionId,
Content = content
};
GlobalHost
.ConnectionManager
.GetHubContext<MyHub>()
.Clients
.Client(connectionId)
.OnMessage(message);
return taskCompletion.Task;
}
public void OnTransactionConfirmed(Guid transactionId)
{
var taskCompletion = transactions[transactionId];
transactions.Remove(transactionId);
taskCompletion.SetResult(true);
}
Message is just a DTO
public class Message
{
public Guid TransactionId { get; set; }
public string Content { get; set; }
}
Basically you send a message to the client using the client method called OnMessage then you wait asynchronously until client confirms with the Hub method called OnTransactionConfirmed
Using a task makes it easier to work asynchronous in a synchronous way, the consumer of SendPrivateMessage could be asynchronous itself (If its called from a WebApi method or similar).
public Task MyWebApiMethod()
{
return myChatLogic.SendPrivateMessage("foo");
}
Or in a synchronous manner
public void MySyncMethod()
{
myChatLogic.SendPrivateMessage("foo").Wait(); //Thread will wait here until client asnwers
}
Note
To make this code failsafe you need to timeout the wait for confirmation and complete the task and remove the transactionId etc

your send message, receive message and display message should be handled separately. you did well with the send message except that you need to add a datetime stamp generated by the Server (do not use time from client) to each message and store it in sql. whenever there is a new message received, the Server should send this message to all online senders' and receivers' client browser or app.
here I said senderS because user A may login using a phone and at the same time login using office desktop browser, if user A send a message to user B mobile phone using the desktop browser, this message should be appear in 3 client device: user A desktop browser, user A mobile phone and user B mobile phone.
for the receiver, whenever go from offline to online, check the last message datetime stored in client side using cookie or file or clientside sqlite or clientside internal storage, with the Server database and get all related messages since last offline by comparing the last message datetime from clientside and message dt from server database.
for the display message, only read from client side storage so that the apps can run offline.

When sending a message, append a random string of 8 characters to the start of the message. On your message received function, cut out the first 8 characters and send those back to the original sender.
If you receive the same characters back from the receiver, then you know that receiver has received the message and has not been disconnected.
You can use this method to generate a random string.
private static IEnumerable<string> RandomStrings(
string allowedChars,
int minLength,
int maxLength,
Random rng)
{
char[] chars = new char[maxLength];
int setLength = allowedChars.Length;
int length = rng.Next(minLength, maxLength + 1);
for (int i = 0; i < length; ++i)
{
chars[i] = allowedChars[rng.Next(setLength)];
}
return new string(chars, 0, length);
}
you will can call the method like this
private string GetRandomStrings(){
const string AllowedChars =
"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz##$^*()";
Random rng = new Random();
return RandomStrings(AllowedChars, 8, 8, 1, rng)
}

Programing HTML5 websocket chat with rooms

I want to create html5 web multiroom chat, based on HTML5 websocket.
But I need some little help to start.
I want to do server side code in c#, but I can not find any tutorials how to do chat websocket server with multi room in c#.
Is there any server which is already implemented in .net, or which I can update to multi room chat ?
It is a little project, one room for 10 peoples. Could you help to me how to start ?
Thank you very much !
I prepare example code structure:
Main server class:
class Program
{
// List of courses, which are currentli avalible ( REPRESENT CHAT ROOM)
protected static ConcurrentDictionary<Course, string> OnlineUsers = new ConcurrentDictionary<Course, string>();
static void Main(string[] args)
{
// Initialize the server on port 81, accept any IPs, and bind events.
var aServer = new WebSocketServer(81, IPAddress.Any)
{
OnReceive = OnReceive,
OnSend = OnSend,
OnConnected = OnConnect,
OnDisconnect = OnDisconnect,
TimeOut = new TimeSpan(0, 5, 0)
};
aServer.Start();
// Accept commands on the console and keep it alive
var command = string.Empty;
while (command != "exit")
{
command = Console.ReadLine();
}
aServer.Stop();
}
// event when the clients connect to server
// Server send to client list of Lessons which are avalible, after
private static void OnConnect(UserContext context)
{
throw new NotImplementedException();
}
// event whent the client, want to disconnect from server
private static void OnDisconnect(UserContext context)
{
throw new NotImplementedException();
}
// event, when client is sending some data
private static void OnSend(UserContext context)
{
throw new NotImplementedException();
}
// event, when server receive data from client
// client choose which room want to join and, we add cleint to list of lessons which he choose
// another method ... Register, Rename, LogOff ...
private static void OnReceive(UserContext context)
{
throw new NotImplementedException();
}
}
Course class: (ROOMS)
class Course
{
// Every course has list of active users
protected static ConcurrentDictionary<User, string> OnlineUsers = new ConcurrentDictionary<User, string>();
// Name of course
public String CourseName { get; set; }
}
User class:
class User
{
// Name of User
public string Name = String.Empty;
// UserContext - Contains data we will export to the Event Delegates.
public UserContext Context { get; set; }
}
It is good structure for my purpose ? I have many courses (room), with one teacher, in one course can be 20 pupils example .. In one course the pupils can talk with techer using chat (web socket) and drawing board ..

That's how I would build the object hierarchy:
The chat server should have a list of ChatRooms.
Each ChatRoom should have a list of ChatUsers.
Each ChatUser should have one or no ChatRoom and an outbound socket.
(this assumes that a user is only in one room at a time. Allowing multiple rooms would make things a bit more complex)
That's how room selection could work:
When a client connects, a ChatUser is created. The first thing the server does is send the list of chatrooms to the user. The client then responds with the name of the chatroom it wants to join. When a chatroom of that name doesn't exist, it is created and added to the global list of rooms.
The client is then added to the room, and the room is set on the client.
That's how chatting could work:
When the socket of the user receives a chat message, it should call a SendToAllClients method on the room the user is currently in (when the room is null, it should return an error message to the user that they must join a room first).
The SendToAll method of the room should then call a SendToClient of all users which are on its list of users.
The SendToClient method of the class would then send the chat message to the client.
How to expand this for multiple chatrooms per user
To allow a client to join multiple chatrooms at once and have separate conversations in them, the client must be able to:
request a list of rooms at any time, not just at startup
join rooms at any time, not just at startup
leave rooms
specify the room when sending a message
That means that the action the client wants to perform can not be deduced from the state it is currently in. You need to add this information to the messages of the user. You could, for example, do this as prefixes. Like
!list
requests the list of rooms
!join:asdf
join/create the room asdf
_asdf:Hello
sends the message Hello to the room asdf.
The messages from the server should have similar prefixes, so that the client can deduce if a message is a room list or a chat message and from what room it originates.

You should try to look into SignalR for ASP.NET (example : jabbr.net/). This may be more helpful and handy.

SignalR notification system

This is my first time playing around with SignalR. I am trying to build a notification system where the server checks at regular intervals to see if there is something (query database) to broadcast and if there is then it broadcasts it to all the clients.
I came across this post on Stackoverflow and was wondering if modifying the code to make a DB call at a particular interval was indeed the right way to do it. If not is there a better way to do it?
I did see a lot of Notification related questions posted here but none with any code in it. Hence this post.
This is the exact code that I am using:
public class NotificationHub : Hub
{
public void Start()
{
Thread thread = new Thread(Notify);
thread.Start();
}
public void Notify()
{
List<CDCNotification> notifications = new List<CDCNotification>();
while (true)
{
notifications.Clear();
notifications.Add(new CDCNotification()
{
Server = "Server A", Application = "Some App",
Message = "This is a long ass message and amesaadfasd asdf message",
ImgURL = "../Content/Images/accept-icon.png"
});
Clients.shownotification(notifications);
Thread.Sleep(20000);
}
}
}
I am already seeing some weird behaviour where the notifications come more often than they are supposed to. Even though I am supposed to get it every 20s I get it around 4-5 secs and I get multiple messages.
Here is my client:
var notifier = $.connection.notificationHub;
notifier.shownotification = function (data) {
$.each(data, function (i, sample) {
var output = Mustache.render("<img class='pull-left' src='{{ImgURL}}'/> <div><strong>{{Application}}</strong></div><em>{{Server}}</em> <p>{{Message}}</p>", sample);
$.sticky(output);
});
};
$.connection.hub.start(function () { notifier.start(); });

Couple of notes:
As soon as a second client connects to your server there will be 2 threads sending the notifications, therefore if you ave more than one client you will have intervals smaller than 20s
Handling thread manually within ASP.NET is considered bad practice, you should avoid this if possible
In general this smells a lot like polling which is kind of the thing SignalR lets you get rid of since you don't need to signal the server/client
In order to solve this you need todo something like this (again, threads in a web application are generally not a good idea):
public class NotificationHub : Hub
{
public static bool initialized = false;
public static object initLock = new object();
public void Start()
{
if(initialized)
return;
lock(initLock)
{
if(initialized)
return;
Thread thread = new Thread(Notify);
thread.Start();
initialized = true;
}
}
public void Notify()
{
List<CDCNotification> notifications = new List<CDCNotification>();
while (true)
{
notifications.Clear();
notifications.Add(new CDCNotification() { Server = "Server A", Application = "Some App", Message = "This is a long ass message and amesaadfasd asdf message", ImgURL = "../Content/Images/accept-icon.png" });
Clients.shownotification(notifications);
Thread.Sleep(20000);
}
}
}
The static initialized flag prevents multiple threads from being created. The locking around it is to ensure that the flag is only set once.

I am working on the same task over here. Instead of continuously checking the database, I created my own events and listener, where an event is RAISED when a NOTIFICATION IS ADDED :) What do you think about that?

Ajax server process with live interface progress status?

I have a process that retrieves html from a remote site and parses it. I pass several URL's into the method, so I would like to ajaxify the process and give a screen notification each time a URL completes parsing. For example, this is what I am trying to do:
List<string> urls = ...//load up with arbitary # of urls
foreach (var url in urls)
{
string html = GetContent(url);
//DO SOMETHING
//COMPLETED.. SEND NOTIFICATION TO SCREEN (HOW DO I DO THIS)
}
public static string GetContent(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
using (var stream = request.GetResponse().GetResponseStream())
{
using (var reader = new StreamReader(stream, Encoding.UTF8))
{
return reader.ReadToEnd();
}
}
}
In each iteration in the loop, I want to show the URL was completed and moving on to the next one. How can I accomplish this?

The first thing you need to worry about is the fact (I'm assuming) that you're running a potentially long-running operation in ASP.NET code. This will become a problem when you run in to IIS timeouts. (By default, 90 seconds.) Assume you're processing ten URLs, each of which takes 15 seconds to complete reader.ReadToEnd() – your code will time out and get killed after the sixth URL.
You might be thinking "I can just crank up the timeout," but that's not really a good answer; you're still under time pressure.
The way I solve problems like this is to move long-running operations into a stand-alone Windows Service, then use WCF to communicate between ASP.NET code and the Service. The Service can run a thread pool that executes requests to process a group of URLs. (Here's an implementation that allows you to queue work items.)
Now, from your web page, you can poll for status updates via AJAX requests. The handler in your ASP.NET code can use WCF to pull the status information from the Service process.
A way to do this might be to assign each submitted work unit a unique ID and return that ID to the client. The client can then poll for status by sending an AJAX request for the status of work unit n. In the Service, keep a List of work units with their statuses (locking it as appropriate to avoid concurrency problems).
public class WorkUnit {
public int ID { get; set; }
public List<string> URLs { get; set; }
public int Processed { get; set; }
}
private var workUnits = new List<WorkUnit>();
private void ExecuteWorkUnit(int id) {
var unit = GetWorkUnit(id);
foreach (var url in unit.URLs) {
string html = GetContent(url);
// do whatever else...
lock (workUnits) unit.Processed++;
}
}
public WorkUnit GetWorkUnit(int id) {
lock (workUnits) {
// Left as an exercise for the reader
}
}
You'll need to fill in methods to add a work unit, return the status of a given work unit, and deal with the thread pool.
I've used a similar architecture with great success.