Given a list of IP addresses:
List<string> ipList = new List<string>(); //example: 192.168.0.1, 192.168.0.2, 192.168.0.3 etc.
I am attempting to loop over each IP in the list, in a parallel fashion and then print a meaningful message to screen:
foreach (PingReply pingReply in ipList.AsParallel().WithDegreeOfParallelism(64).Select(ip => new Ping().Send(ip)))
{
Console.WriteLine($"Ping status: {pingReply.Status} for the target IP address: {ip}");
}
I am unable to access ip in that context. I would really like to understand how I could go about accessing each relative ip as I am sending them out?
I have explored the PingReply object but PingReply.Address as an example contains the host (sender) IP, so it cannot help with this requirement. I really wish the PingReply object contained the Ip that was pinged!
UPDATE
As per example provided by #haim770 and #MindSwipe I ended up using:
foreach (var pingResponseData in ipList.AsParallel().WithDegreeOfParallelism(64).Select(ip => new { ip, pingReply = new Ping().Send(ip) }))
{
Console.WriteLine($"Ping status: {pingResponseData.pingReply.Status} for the target IP address: {pingResponseData.ip}");
}
UPDATE 2
As per comment from #pinkfloydx33 regarding use of ValueTuple I have done as per the following example:
foreach (var (ip, reply) in ipList.AsParallel().WithDegreeOfParallelism(ipList.Count).Select(ip => (ip, new Ping().Send(ip, 150))))
{
Console.WriteLine($"Ping status: {reply.Status} for the target IP address: {ip}");
}
You're currently only selecting the pingReply, not the ip and the pingReply, to do that you'll need to select a new anonymous type and iterate over that. Like so:
foreach (var (pingReply, ip) in ipList.AsParallel().WithDegreeOfParallelism(64).Select(ip => (ip, Ping().Send(ip))))
{
// Here 'i' is an object with the properties 'ip' and 'pingReply'
Console.WriteLine($"Ping status: {i.pingReply.Status} for the target IP address: {i.ip}");
}
Edit: Just noticed now that haim770 posted basically this in their comment
Edit 2: Thanks pinkfloydx33 for pointing out I could use tuple deconsturcting
Calling the synchronous Ping.Send in parallel with a degree of parallelism = 64 is quite inefficient, because as many as 64 threads are blocked during the parallel execution (provided that the ThreadPool has enough threads available to satisfy the demand, which is doubtful). A more efficient way to do the pinging is to use the asynchronous Ping.SendPingAsync method. To invoke this method in a way that no more than 64 asynchronous operations will be simultaneously in-flight, you will need a Parallel.ForEach equivalent that works with asynchronous delegates. Currently there is no such thing available built-in (it will probably be available in .NET 6), but you can find lots of custom implementations if you search for ForEachAsync. There is one here for example. Then you will be able to do this:
var ipList = new List<string>() {"192.168.0.1", "192.168.0.2", "192.168.0.3"}; // etc
ipList.ForEachAsync(async ip =>
{
var ping = new Ping();
var reply = await ping.SendPingAsync(ip);
Console.WriteLine($"IP '{ip}' ping reply status: {reply.Status}");
}, dop: 64).Wait();
You could also await the completion of the operation, instead of using the blocking Wait, provided that you are calling it from an async method.
Related
My client is attempting to send messages to the receiver. However I noticed that the receiver sometimes does not receive all the messages sent by the client thus missing a few messages (not sure where the problem is ? Client or the receiver).
Any suggestions on why that might be happening. This is what I am currently doing
On the receiver side this is what I am doing.
This is the Event Processor
async Task IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
}
}
This is how the client connects to the event hub
var StrBuilder = new EventHubsConnectionStringBuilder(eventHubConnectionString)
{
EntityPath = eventHubName,
};
this.eventHubClient = EventHubClient.CreateFromConnectionString(StrBuilder.ToString());
How do I direct my messages to specific consumers
I'm using this sample code from eventhub official doc, for sending and receiving.
And I have 2 consumer groups: $Default and newcg. Suppose you have 2 clients, the client_1 are using the default consumer group($Default), and client_2 are using the other consumer group(newcg)
First, after create the send client, in the SendMessagesToEventHub method, we need to add a property with value. The value should be the consumer group name. Sample code like below:
private static async Task SendMessagesToEventHub(int numMessagesToSend)
{
for (var i = 0; i < numMessagesToSend; i++)
{
try
{
var message = "444 Message";
Console.WriteLine($"Sending message: {message}");
EventData mydata = new EventData(Encoding.UTF8.GetBytes(message));
//here, we add a property named "cg", it's value is the consumer group. By setting this property, then we can read this message via this specified consumer group.
mydata.Properties.Add("cg", "newcg");
await eventHubClient.SendAsync(mydata);
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} > Exception: {exception.Message}");
}
await Task.Delay(10);
}
Console.WriteLine($"{numMessagesToSend} messages sent.");
}
Then in the client_1, after create the receiver project, which use the default consumer group($Default)
-> in the SimpleEventProcessor class -> ProcessEventsAsync method, we can filter out the unnecessary event data. Sample code for ProcessEventsAsync method:
public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
//filter the data here
if (eventData.Properties["cg"].ToString() == "$Default")
{
var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
Console.WriteLine($"Message received. Partition: '{context.PartitionId}', Data: '{data}'");
Console.WriteLine(context.ConsumerGroupName);
}
}
return context.CheckpointAsync();
}
And in another client, like client_2, which use another consumer group, like it's name is newcg, we can follow the steps in client_1, just a little changes in ProcessEventsAsync method, like below:
public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (var eventData in messages)
{
//filter the data here, using another consumer group name
if (eventData.Properties["cg"].ToString() == "newcg")
{
//other code
}
}
return context.CheckpointAsync();
}
This happens only when there are 2 or more Event Processor Host reading from same consumer group.
If you have event hub with 32 partitions and 2 event processor host reading from same consumer group. Then each event processor host will read from 16 partition and so on.
Similarly if 4 Event processor host parallelly reading from same consumer group then each will read from 8 partitions.
Check if you have 2 or more event processor host running on same consumer group.
I have tested your code and slightly modified it(different overload of EventProcessorHost constructor, and added CheckpointAsync after consuming the messages), and then did some tests.
By using the default implementation and default EventProcessorOptions(EventProcessorOptions.DefaultOptions) I can say that I did experience some latency when it comes to consuming messages, but all messages were processed successfully.
So, sometimes it seems like I am not getting the messages from the certain partition, but after a certain period of time, all messages arrive:
Here you can find the actual modified code that worked for me. It is a simple console app that prints to the console if something arrives.
string processorHostName = Guid.NewGuid().ToString();
var Options = new EventProcessorOptions()
{
MaxBatchSize = 1, //not required to make it working, just for testing
};
Options.SetExceptionHandler((ex) =>
{
System.Diagnostics.Debug.WriteLine($"Exception : {ex}");
});
var eventHubCS = "event hub connection string";
var storageCS = "storage connection string";
var containerName = "test";
var eventHubname = "test2";
EventProcessorHost eventProcessorHost = new EventProcessorHost(eventHubname, "$Default", eventHubCS, storageCS, containerName);
eventProcessorHost.RegisterEventProcessorAsync<MyEventProcessor>(Options).Wait();
For sending the messages to the event hub and testing I used this message publisher app.
I'm developing an application that manages devices in the network, at a certain point in the applicaiton, I must ping (actually it's not a ping, it's a SNMP get) all computers in the network to check if it's type is of my managed device.
My problem is that pinging all computers in the network is very slow (specially because most of them won't respond to my message and will simply timeout) and has to be done asynchronously.
I tried to use TLP to do this with the following code:
public static void FindDevices(Action<IPAddress> callback)
{
//Returns a list of all host names with a net view command
List<string> hosts = FindHosts();
foreach (string host in hosts)
{
Task.Run(() =>
{
CheckDevice(host, callback);
});
}
}
But it runs VERY slow, and when I paused execution I checked threads window and saw that it only had one thread pinging the network and was thus, running tasks synchronously.
When I use normal threads it runs a lot faster, but Tasks were supposed to be better, I'd like to know why aren't my Tasks optimizing parallelism.
**EDIT**
Comments asked for code on CheckDevice, so here it goes:
private static void CheckDevice(string host, Action<IPAddress> callback)
{
int commlength, miblength, datatype, datalength, datastart;
string output;
SNMP conn = new SNMP();
IPHostEntry ihe;
try
{
ihe = Dns.Resolve(host);
}
catch (Exception)
{
return;
}
// Send sysLocation SNMP request
byte[] response = conn.get("get", ihe.AddressList[0], "MyDevice", "1.3.6.1.2.1.1.6.0");
if (response[0] != 0xff)
{
// If response, get the community name and MIB lengths
commlength = Convert.ToInt16(response[6]);
miblength = Convert.ToInt16(response[23 + commlength]);
// Extract the MIB data from the SNMP response
datatype = Convert.ToInt16(response[24 + commlength + miblength]);
datalength = Convert.ToInt16(response[25 + commlength + miblength]);
datastart = 26 + commlength + miblength;
output = Encoding.ASCII.GetString(response, datastart, datalength);
if (output.StartsWith("MyDevice"))
{
callback(ihe.AddressList[0]);
}
}
}
Your issue is that you are iterating a none thread safe item the List.
If you replace it with a thread safe object like the ConcurrentBag you should find the threads will run in parallel.
I was a bit confused as to why this was only running one thread, I believe it is this line of code:
try
{
ihe = Dns.Resolve(host);
}
catch (Exception)
{
return;
}
I think this is throwing exceptions and returning; hence you only see one thread. This also ties into your observation that if you added a sleep it worked correctly.
Remember that when you pass a string your passing the reference to the string in memory, not the value. Anyway, the ConcurrentBag seems to resolve your issue. This answer might also be relevant
I'm having an issue with ZeroMQ, which I believe is because I'm not very familiar with it.
I'm trying to build a very simple service where multiple clients connect to a server and sends a query. The server responds to this query.
When I use REQ-REP socket combination (client using REQ, server binding to a REP socket) I'm able to get close to 60,000 messages per second at server side (when client and server are on the same machine). When distributed across machines, each new instance of client on a different machine linearly increases the messages per second at the server and easily reaches 40,000+ with enough client instances.
Now REP socket is blocking, so I followed ZeroMQ guide and used the rrbroker pattern (http://zguide.zeromq.org/cs:rrbroker):
REQ (client) <----> [server ROUTER -- DEALER --- REP (workers running on different threads)]
However, this completely screws up the performance. I'm getting only around 4000 messages per second at the server when running across machines. Not only that, each new client started on a different machine reduces the throughput of every other client.
I'm pretty sure I'm doing something stupid. I'm wondering if ZeroMQ experts here can point out any obvious mistakes. Thanks!
Edit: Adding code as per advice. I'm using the clrzmq nuget package (https://www.nuget.org/packages/clrzmq-x64/)
Here's the client code. A timer counts how many responses are received every second.
for (int i = 0; i < numTasks; i++) { Task.Factory.StartNew(() => Client(), TaskCreationOptions.LongRunning); }
void Client()
{
using (var ctx = new Context())
{
Socket socket = ctx.Socket(SocketType.REQ);
socket.Connect("tcp://192.168.1.10:1234");
while (true)
{
socket.Send("ping", Encoding.Unicode);
string res = socket.Recv(Encoding.Unicode);
}
}
}
Server - case 1: The server keeps track of how many requests are received per second
using (var zmqContext = new Context())
{
Socket socket = zmqContext.Socket(SocketType.REP);
socket.Bind("tcp://*:1234");
while (true)
{
string q = socket.Recv(Encoding.Unicode);
if (q.CompareTo("ping") == 0) {
socket.Send("pong", Encoding.Unicode);
}
}
}
With this setup, at server side, I can see around 60,000 requests received per second (when client is on the same machine). When on different machines, each new client increases number of requests received at server as expected.
Server Case 2: This is essentially rrbroker from ZMQ guide.
void ReceiveMessages(Context zmqContext, string zmqConnectionString, int numWorkers)
{
List<PollItem> pollItemsList = new List<PollItem>();
routerSocket = zmqContext.Socket(SocketType.ROUTER);
try
{
routerSocket.Bind(zmqConnectionString);
PollItem pollItem = routerSocket.CreatePollItem(IOMultiPlex.POLLIN);
pollItem.PollInHandler += RouterSocket_PollInHandler;
pollItemsList.Add(pollItem);
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("{0}", ze.Message);
return;
}
dealerSocket = zmqContext.Socket(SocketType.DEALER);
try
{
dealerSocket.Bind("inproc://workers");
PollItem pollItem = dealerSocket.CreatePollItem(IOMultiPlex.POLLIN);
pollItem.PollInHandler += DealerSocket_PollInHandler;
pollItemsList.Add(pollItem);
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("{0}", ze.Message);
return;
}
// Start the worker pool; cant connect
// to inproc socket before binding.
workerPool.Start(numWorkers);
while (true)
{
zmqContext.Poll(pollItemsList.ToArray());
}
}
void RouterSocket_PollInHandler(Socket socket, IOMultiPlex revents)
{
RelayMessage(routerSocket, dealerSocket);
}
void DealerSocket_PollInHandler(Socket socket, IOMultiPlex revents)
{
RelayMessage(dealerSocket, routerSocket);
}
void RelayMessage(Socket source, Socket destination)
{
bool hasMore = true;
while (hasMore)
{
byte[] message = source.Recv();
hasMore = source.RcvMore;
destination.Send(message, message.Length, hasMore ? SendRecvOpt.SNDMORE : SendRecvOpt.NONE);
}
}
Where the worker pool's start method is:
public void Start(int numWorkerTasks=8)
{
for (int i = 0; i < numWorkerTasks; i++)
{
QueryWorker worker = new QueryWorker(this.zmqContext);
Task task = Task.Factory.StartNew(() =>
worker.Start(),
TaskCreationOptions.LongRunning);
}
Console.WriteLine("Started {0} with {1} workers.", this.GetType().Name, numWorkerTasks);
}
public class QueryWorker
{
Context zmqContext;
public QueryWorker(Context zmqContext)
{
this.zmqContext = zmqContext;
}
public void Start()
{
Socket socket = this.zmqContext.Socket(SocketType.REP);
try
{
socket.Connect("inproc://workers");
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("Could not create worker, error: {0}", ze.Message);
return;
}
while (true)
{
try
{
string message = socket.Recv(Encoding.Unicode);
if (message.CompareTo("ping") == 0)
{
socket.Send("pong", Encoding.Unicode);
}
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("Could not receive message, error: " + ze.ToString());
}
}
}
}
Could you post some source code or at least a more detailed explanation of your test case? In general the way to build out your design is to make one change at a time, and measure at each change. You can always move stepwise from a known working design to more complex ones.
Most probably the 'ROUTER' is the bottleneck.
Check out these related questions on this:
Client maintenance in ZMQ ROUTER
Load testing ZeroMQ (ZMQ_STREAM) for finding the maximum simultaneous users it can handle
ROUTER (and ZMQ_STREAM, which is just a variant of ROUTER) internally has to maintain the client mapping, hence IMO it can accept limited connections from a particular client. It looks like ROUTER can multiplex multiple clients, only as long as, each client has only one active connection.
I could be wrong here - but I am not seeing much proof to the contrary (simple working code that scales to multi-clients with multi-connections with ROUTER or STREAM).
There certainly is a very severe restriction on concurrent connections with ZeroMQ, though it looks like no one know what is causing it.
I have done done performance testing on calling a native unmanaged DLL function with various methods from C#:
1. C++/CLI wrapper
2. PInvoke
3. ZeroMQ/clrzmq
The last might be interesting for you.
My finding at the end of my performance test was that using the ZMQ binding clrzmq was not useful and produced a factor of 100 performance overhead after I tried to optimize the PInvoke calls within the source code of the binding. Therefore I have used the ZMQ without a binding but with PInvoke calls.these calls must be done with the cdecl convention and with the option "SuppressUnmanagedCodeSecurity" to get most speed.
I had to import just 5 functions which was fairly easy.
At the end the speed was a bit slower than a PInvoke call but with the ZMQ-in my case over "inproc".
This may give you the hint to try it without the binding, if speed is interesting for you.
This is not a direct answer for your question but may help you to increase performance in general.
I want to use Fleck for my WebSocket project,
Server side looks pretty straightforward, but how to I differentiate opened connections. is there some sort of ID? The only way I can think of is to create GUID in OnOpen event and pass it back to client. is there a smarter solution?
Basic server set up:
socket.OnOpen = () =>
{
Console.WriteLine("Open!");
allSockets.Add(socket);
};
socket.OnClose = () =>
{
Console.WriteLine("Close!");
allSockets.Remove(socket);
};
socket.OnMessage = message =>
{
Console.WriteLine(message);
allSockets.ToList().ForEach(s => s.Send("Echo: " + message));
};
E.G. how would I make a chat room so all connection receive message except for the one sending.
Fleck server here: https://github.com/statianzo/Fleck
Fleck now creates a Guid Id on the WebSocketConnectionInfo for every connected client. This will help in cases where multiple connections are using the same IP. See the related commit here:
https://github.com/statianzo/Fleck/commit/fc037f49362bb41a2bc753a5ff51cc9da40ad824
Ask the user for a user name, and attach the username to the the socket address:
var wsimpl = window.WebSocket || window.mozWebSocket;
window.ws = new wsimpl('http://localhost:8080/myApp_' + userName, myProtocol);
then strip out the userName on the service side after the socket has been opened with socket.WebSocketConnectionInfo.path. There is also a clientip property that can be used also. I am doing this and it works great.
I started something similar to what you are asking I am doing. In my case I am using the ConnectionInfo.Path to differ between what my sockets are doing.
You can gain a lot of information already out of the ConnectionInfo
socket.ConnectionInfo.{Host|Path|Origin|SubProtocol|ClientIPAddress|Cookies}
So to answer your question to give it to everyone but the sender you can differentiate each socket based on the ConnectionInfo (if applicable you could create a UID out of this info also)
As a very basic example:
If you know each Client will have a different IP something like the following would work:
socket.OnMessage = message =>
{
foreach (IWebSocketConnection socketConnection in allSockets.Where(socketConnection => socket.ConnectionInfo.ClientIpAddress != socketConnection.ConnectionInfo.ClientIpAddress))
{
socketConnection.Send("Echo: " + message);
}
};
It's a different socket instance per client, so I would have thought you should be able to do:
allSockets.Where(x => x != socket).ToList().ForEach(s => s.Send("Echo: " + message));
Related to How to get the IP address of a WCF remote endpoint?
I am using this code to retrieve the remote IP address when a workflow method is invoked:
private static string GetRemoteIP()
{
var oc = OperationContext.Current;
var mp = oc.IncomingMessageProperties;
var remp = mp[RemoteEndpointMessageProperty.Name] as RemoteEndpointMessageProperty;
return remp == null ? "(unknown)" : remp.Address;
}
However, the address I get back is "::1". I don't want the IPv6 address, I want the IPv4 one (127.0.0.1) - any way of forcing this?
No, I don't think so. You basically just read out a property set by the client at the time of the call. Your only option would be to instruct the client (through some config) to use IPv4 instead of IPv6 at all times (i.e. turn off IPv6 all together).
I'm not aware of any WCF setting to enforce that - you'd have to dig into the network stack and see if there's any way to make it use IPv4 addresses instead of IPv6.
You're seeing ::1 because you're connecting to the service by resolving the name "localhost" instead of addressing it as "127.0.0.1". Modern versions of Windows that have the IPv6 stack enabled will use IPv6 first.
You can disable the IPv6 stack, but that's roughly the same as making like an ostrich and sticking your head in the sand. IPv6 is here, and people are using it on their networks, so your application should be prepared to support it.
The workaround proposed by Murat will not work.
The MSDN says - if you pass the IP address to the GetHostAddresses method this address is returned in an array without querying the DNS.
To get it working you will need to query for the host name first, using GetHostEntry method. And then, using the host name, use GetHostAddresses. However, even the GetHostEntry may have the list of addresses that will be enough for you.
Here is a workaround: (You can store the values in a hashtable to avoid multiple DNS operations)
static string GetClientIP()
{
var context = OperationContext.Current;
var mp = context.IncomingMessageProperties;
var propName = RemoteEndpointMessageProperty.Name;
var prop = (RemoteEndpointMessageProperty) mp[propName];
string remoteIP = prop.Address;
if(remoteIP.IndexOf(":") > -1)
{
IPAddress[] addresses = Dns.GetHostAddresses(remoteIP);
for (int i = 0; i < addresses.Length; i++)
{
if(addresses[i].ToString().IndexOf(".")>-1)
return addresses[i].ToString();
}
return remoteIP;
}
else
{
return remoteIP;
}
}
You could use AddressFamily.InterNetwork == address.AddressFamily and AddressFamily.InterNetworkV6 as a test instead of looking for ":" or "." in the addresses strings.