Related
Scenario:
I am building an Android app using Xamarin.Forms that will be deployed to a group of devices. All but one of the devices will be doing some data collection, and the remaining device will be the "hub" to aggregate all of the data and do some reporting. I am using Bluetooth for the device-to-device communication. The 'hub', labelled the master, acts as the client, and all of the collectors act as the server. I have a prototype working with a single server and client...almost.
Occasionally the client/master will be unable to read from the server/collector. I am struggling to find the reason for why this is and would appreciate any help.
Symptoms:
The client's call to .Read() from the InputStream will occasionally block indefinitely, even though the server has written to the output stream. I've added a timeout to this call to prevent the app from getting stuck entirely.
This happens intermittently, but I've found some pattern to when it works and when it doesn't
It seems to be related to the 'server' app, and not the client. The client can remain open, running, and initiate the request to connect to the server as often as needed.
It always works the first time the 'server' app is launched and connected to. It ususally works the second time. By the third connection, .Read() will consistently block/timeout. Closing and reopening the app on the server "cleans the slate" so to speak and it will work again.
Once it starts failing, it seems to be 'stuck' in a failed state.
Removing the app from the foreground (but not closing/killing it) seems to correct the faulted state, and the connection/read will happen successfully as long as the app/UI remains in the background. Once restored to the foreground, it starts failing again.
Code:
All of the bluetooth handling is done by a single class/service that I'm injecting using Xamarin.Forms DependencyService. All of the devices will, on startup (via the constructor of this class), loop indefinitely on a background thread, waiting for connections and repeating. Much of this bluetooth code is based on the Bluetooth Chat example, as well as some other online resources I've found (some android native/java, some Xamarin/C#)
The master will, on demand (triggered by press of a button in the UI), attempt to connect to any collectors (via bonded bluetooth devices) and read data from them. There is also a simple UI component which essentially serves as a console log.
Here is the service class in its entirety.
public class GameDataSyncService : IGameDataSyncService
{
private const string UUID = "8e99f5f1-4a07-4268-9686-3a288326e0a2";
private static Task acceptLoopTask;
private static Task syncDataTask;
private static readonly object locker = new object();
private static bool running = false;
public event EventHandler<DataSyncMessage> MessageBroadcast;
public GameDataSyncService()
{
// Every device will listen and accept incoming connections. The master will make the connections.
lock (locker)
{
if (acceptLoopTask == null)
{
acceptLoopTask = Task.Factory.StartNew(AcceptLoopWorker, TaskCreationOptions.LongRunning);
}
}
}
public void SyncData()
{
lock (locker)
{
if (running)
{
BroadcastMessage("Previous data sync is still running.", DataSyncMessageType.Warning);
return;
}
else
{
running = true;
syncDataTask = Task.Factory.StartNew(SyncDataWorker);
}
}
}
private void BroadcastMessage(string message, DataSyncMessageType type = DataSyncMessageType.Info)
{
MessageBroadcast?.Invoke(this, new DataSyncMessage { Text = message, Type = type });
}
private async Task AcceptLoopWorker()
{
int count = 0;
while (true)
{
BluetoothServerSocket serverSocket = null;
BluetoothSocket clientSocket = null;
try
{
BroadcastMessage($"Listening for incoming connection...", DataSyncMessageType.Debug);
serverSocket = BluetoothAdapter.DefaultAdapter.ListenUsingRfcommWithServiceRecord(nameof(GameDataSyncService), Java.Util.UUID.FromString(UUID));
clientSocket = serverSocket.Accept(); // This call blocks until a connection is established.
BroadcastMessage($"Connection received from {clientSocket.RemoteDevice.Name}. Sending data...", DataSyncMessageType.Info);
var bytes = Encoding.UTF8.GetBytes($"Hello World - {string.Join(" ", Enumerable.Repeat(Guid.NewGuid(), ++count))}");
await clientSocket.OutputStream.WriteAsync(bytes, 0, bytes.Length);
clientSocket.OutputStream.Flush();
// Give the master some time to close the connection from their end
await Task.Delay(1000*3);
}
catch (Exception ex)
{
BroadcastMessage($"{ex.GetType().FullName}: {ex.Message}", DataSyncMessageType.Debug);
}
finally
{
try { clientSocket?.InputStream?.Close(); } catch { }
try { clientSocket?.InputStream?.Dispose(); } catch { }
try { clientSocket?.OutputStream?.Close(); } catch { }
try { clientSocket?.OutputStream?.Dispose(); } catch { }
try { clientSocket?.Close(); } catch { }
try { clientSocket?.Dispose(); } catch { }
try { serverSocket?.Close(); } catch { }
try { serverSocket?.Dispose(); } catch { }
BroadcastMessage($"Connection closed.", DataSyncMessageType.Debug);
}
}
}
private async Task SyncDataWorker()
{
BroadcastMessage($"Beginning data sync...");
foreach (var bondedDevice in BluetoothAdapter.DefaultAdapter.BondedDevices.OrderBy(d => d.Name))
{
BluetoothSocket clientSocket = null;
try
{
clientSocket = bondedDevice.CreateRfcommSocketToServiceRecord(Java.Util.UUID.FromString(UUID));
BroadcastMessage($"Connecting to {bondedDevice.Name}...");
try
{
clientSocket.Connect();
}
catch
{
BroadcastMessage($"Connection to {bondedDevice.Name} failed.", DataSyncMessageType.Error);
}
while (clientSocket.IsConnected)
{
byte[] buffer = new byte[1024];
var readTask = clientSocket.InputStream.ReadAsync(buffer, 0, buffer.Length);
if (await Task.WhenAny(readTask, Task.Delay(1000)) != readTask)
{
BroadcastMessage($"Read timeout...", DataSyncMessageType.Error);
break;
}
int bytes = readTask.Result;
BroadcastMessage($"Read {bytes} bytes.", DataSyncMessageType.Success);
if (bytes > 0)
{
var text = Encoding.UTF8.GetString(buffer.Take(bytes).ToArray());
BroadcastMessage(text, DataSyncMessageType.Success);
break;
}
}
}
catch (Exception ex)
{
BroadcastMessage($"{ex.GetType().FullName}: {ex.Message}", DataSyncMessageType.Debug);
}
finally
{
try { clientSocket?.InputStream?.Close(); } catch { }
try { clientSocket?.InputStream?.Dispose(); } catch { }
try { clientSocket?.OutputStream?.Close(); } catch { }
try { clientSocket?.OutputStream?.Dispose(); } catch { }
try { clientSocket?.Close(); } catch { }
try { clientSocket?.Dispose(); } catch { }
}
}
await Task.Delay(1000 * 3);
BroadcastMessage($"Data sync complete!");
lock (locker)
{
running = false;
}
}
}
What I've tried (nothing below has had any effect):
Most of these were from 'solutions' from other stackoverflow posts.
Adding arbitrary delays into the mix
Making sure to explicitly close/dispose everything, in order, including the streams
Tried replacing the socket handling with their 'Insecure' counterparts.
Adjusting my read timeout to something arbitrarily long, in case a second wasn't enough.
Disabling/Re-enabling bluetooth on the server/collector before .Accept() ing a new connection (resorted to trying random stuff by this point)
Video:
I took a video of this happening.
The tablet in the back is the collector/server The tablet in the foreground is the master/client. When the video starts, the client is displaying some previous attempts, and the server app is in the background (but running). I demonstrate that the .Read works when the collector/server app is in the background, but not the foreground. Each request to begin data sync has a corresponding entry to the "console" (or a warning if I pressed it too soon)
https://youtu.be/NGuGa7upCU4
Summary:
To the best of my knowledge, my code is correct. I have no idea what else to change/fix to get this working more reliably. The actual connection seems like it is successful (based on logs from the server/collector, unfortunately not shown in the video), but the issue lies somewhere in the .Write (or .Read). ANy help, suggestions, or insight would be awesome.
Try the following, changed all to using:
private async Task AcceptLoopWorker()
{
int count = 0;
while (true)
{
try
{
BroadcastMessage("Listening for incoming connection...", DataSyncMessageType.Debug);
using (var serverSocket = BluetoothAdapter.DefaultAdapter.ListenUsingRfcommWithServiceRecord(nameof(GameDataSyncService), Java.Util.UUID.FromString(UUID)))
using (var clientSocket = serverSocket.Accept()) // This call blocks until a connection is established.
{
BroadcastMessage(string.Format("Connection received from {0}. Sending data...", clientSocket.RemoteDevice.Name), DataSyncMessageType.Info);
var bytes = System.Text.Encoding.UTF8.GetBytes(string.Format("Hello World - {0}", string.Join(" ", Enumerable.Repeat(Guid.NewGuid(), ++count))));
await clientSocket.OutputStream.WriteAsync(bytes, 0, bytes.Length);
}
await Task.Delay(1000 * 3); // Give the master some time to close the connection from their end
}
catch (Java.IO.IOException ex)
{
BroadcastMessage(string.Format("IOException {0}: {1}", ex.GetType().FullName, ex.Message), DataSyncMessageType.Debug);
}
catch (Java.Lang.Exception ex)
{
BroadcastMessage(string.Format("Exception {0}: {1}", ex.GetType().FullName, ex.Message), DataSyncMessageType.Debug);
}
}
}
private async Task SyncDataWorker()
{
BroadcastMessage("Beginning data sync...");
foreach (var bondedDevice in BluetoothAdapter.DefaultAdapter.BondedDevices.OrderBy(d => d.Name))
{
try
{
using (var clientSocket = bondedDevice.CreateRfcommSocketToServiceRecord(Java.Util.UUID.FromString(UUID)))
{
BroadcastMessage(string.Format("Connecting to {0}...", bondedDevice.Name));
if (!clientSocket.IsConnected)
{
clientSocket.Connect();
}
if (clientSocket.IsConnected)
{
byte[] buffer = new byte[1024];
var readTask = clientSocket.InputStream.ReadAsync(buffer, 0, buffer.Length);
if (await Task.WhenAny(readTask, Task.Delay(1000)) != readTask)
{
BroadcastMessage("Read timeout...", DataSyncMessageType.Error);
break;
}
int bytes = readTask.Result;
BroadcastMessage(string.Format("Read {0} bytes.", bytes), DataSyncMessageType.Success);
if (bytes > 0)
{
var text = System.Text.Encoding.UTF8.GetString(buffer.Take(bytes).ToArray());
BroadcastMessage(text, DataSyncMessageType.Success);
break;
}
}
else
{
BroadcastMessage("Not Connected...", DataSyncMessageType.Error);
}
}
}
catch (Java.IO.IOException ex)
{
BroadcastMessage(string.Format("IOException {0}: {1}", ex.GetType().FullName, ex.Message), DataSyncMessageType.Debug);
}
catch (Java.Lang.Exception ex)
{
BroadcastMessage(string.Format("Exception {0}: {1}", ex.GetType().FullName, ex.Message), DataSyncMessageType.Debug);
}
}
await Task.Delay(1000 * 3);
BroadcastMessage("Data sync complete!");
lock (locker)
{
running = false;
}
}
I have a requirement where I am continuously sending data to IoT hub and If Internet connectivity is went down I need to store it on Local database like SQL.
To Send data to IoT hub we have asyn method that is "deviceClient.SendEventAsync"
where device client is the object of DeviceClient class.
Now since this is asyn method it is not throwing any exception when no internet connection is there hence I am not able to catch this and store into local sql db.
Inside method my code is like
try
{
await deviceClient.SendEventAsync(message) ;
}
catch(AggregateException)
{
//If exception found then store same msg to local database.
}
catch(Exception ex)
{
//If exception found then store same msg to local database.
}
But I am never getting any exception in case of any fault or no internet connectivity and the execution of code is keep going.
Please help me to tackle this issue.
Also let me know in case there are any other ways to capture exception while calling any asyn method.
Please find the entire code structure that I am using for this operation.
namespace D2CDeviceDataSender
{
class Program
{
private static DeviceClient deviceClient;
private static string iotHubUri = string.Empty;
private static string deviceKey = string.Empty;
private static string deviceName = string.Empty;
static void Main(string[] args)
{
try
{
iotHubUri = ConfigurationManager.AppSettings["IoTHubURI"].ToString();
deviceName = ConfigurationManager.AppSettings["DeviceName"].ToString();
deviceKey = ConfigurationManager.AppSettings["DeviceKey"].ToString();
deviceClient = DeviceClient.Create(iotHubUri, new DeviceAuthenticationWithRegistrySymmetricKey(deviceName, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);
Task.Factory.StartNew(() => SendDeviceToCloudMessagesAsync1());
Console.ReadLine();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine("Press any key to exit.....");
Console.ReadLine();
}
}
public static async Task SendDeviceToCloudMessagesAsync1()
{
while (true)
{
try
{
double avgWindSpeed = 10; // m/s
Random rand = new Random();
double currentWindSpeed = avgWindSpeed + rand.Next();
var telemetryDataPoint = new
{
DeviceCode = "myFirstDevice",
Date = DateTime.UtcNow,
};
string messageString = JsonConvert.SerializeObject(telemetryDataPoint);
var message = new Message(Encoding.ASCII.GetBytes(messageString));
Task taskresult = deviceClient.SendEventAsync(message);
await taskresult;
Console.WriteLine("Data sent to IoT hub :" + DateTime.Now + " " + messageString);
}
catch (IotHubCommunicationException ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.InnerException);
Console.WriteLine("Internet connectivity down insert data to local database !");
}
catch (AggregateException ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.InnerException);
Console.WriteLine("Internet connectivity down insert data to local database !");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.InnerException);
Console.WriteLine("Internet connectivity down insert data to local database !");
}
Thread.Sleep(5000);
}
}
}
}
Below are my two observation that is:
This code will throwing exception only one time when internet
connectivity get down.
From next iteration the "await taskresult;" method in stop
responding and from on ward iteration so I am not able to capture any
exception.
Please revert back with your feedback.
I have the following code in a loop that continuously sends messages to IoT Hub:
try
{
await deviceClient.SendEventAsync(message);
Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, messageString);
}
catch (AggregateException e)
{
Console.WriteLine("{0} > Error Sending message: {1}", DateTime.Now, messageString);
Console.WriteLine("AggregateException: {0} ", e.Message);
}
If I set this running, and then disconnect the computer from the network, I see this in the output:
19/04/2017 09:11:47 > Sending message: {"deviceId":"myFirstDevice","windSpeed":10.042793008518775}
19/04/2017 09:11:52 > Sending message: {"deviceId":"myFirstDevice","windSpeed":8.5088816902175921}
19/04/2017 09:11:58 > Sending message: {"deviceId":"myFirstDevice","windSpeed":10.399840490147398}
19/04/2017 09:12:22 > Error Sending message: {"deviceId":"myFirstDevice","windSpeed":10.388208727533094}
AggregateException: One or more errors occurred.
This shows that you can catch AggregateExceptions from the SendEventAsync method. What platform are you running your code on?
We're currently using Redis 2.8.4 and StackExchange.Redis (and loving it) but don't have any sort of protection against hardware failures etc at the moment. I'm trying to get the solution working whereby we have master/slaves and sentinel monitoring but can't quite get there and I'm unable to find any real pointers after searching.
So currently we have got this far:
We have 3 redis servers and sentinel on each node (setup by the Linux guys):
devredis01:6383 (master)
devredis02:6383 (slave)
devredis03:6383 (slave)
devredis01:26379 (sentinel)
devredis02:26379 (sentinel)
devredis03:26379 (sentinel)
I am able to connect the StackExchange client to the redis servers and write/read and verify that the data is being replicated across all redis instances using Redis Desktop Manager.
I can also connect to the sentinel services using a different ConnectionMultiplexer, query the config, ask for master redis node, ask for slaves etc.
We can also kill the master redis node and verify that one of the slaves is promoted to master and replication to the other slave continues to work. We can observe the redis connection trying to reconnect to the master, and also if I recreate the ConnectionMultiplexer I can write/read again to the newly promoted master and read from the slave.
So far so good!
The bit I'm missing is how do you bring it all together in a production system?
Should I be getting the redis endpoints from sentinel and using 2 ConnectionMultiplexers?
What exactly do I have to do to detect that a node has gone down?
Can StackExchange do this for me automatically or does it pass an event so I can reconnect my redis ConnectionMultiplexer?
Should I handle the ConnectionFailed event and then reconnect in order for the ConnectionMuliplexer to find out what the new master is?
Presumably while I am reconnecting any attempts to write will be lost?
I hope I'm not missing something very obvious here I'm just struggling to put it all together.
Thanks in advance!
I was able to spend some time last week with the Linux guys testing scenarios and working on the C# side of this implementation and am using the following approach:
Read the sentinel addresses from config and create a ConnectionMultiplexer to connect to them
Subscribe to the +switch-master channel
Ask each sentinel server in turn what they think the master redis and slaves are, compare them all to make sure they all agree
Create a new ConnectionMultiplexer with the redis server addresses read from sentinel and connect, add event handler to ConnectionFailed and ConnectionRestored.
When I receive the +switch-master message I call Configure() on the redis ConnectionMultiplexer
As a belt and braces approach I always call Configure() on the redis ConnectionMultiplexer 12 seconds after receiving a connectionFailed or connectionRestored event when the connection type is ConnectionType.Interactive.
I find that generally I am working and reconfigured after about 5 seconds of losing the redis master. During this time I can't write but I can read (since you can read off a slave). 5 seconds is ok for us since our data updates very quickly and becomes stale after a few seconds (and is subsequently overwritten).
One thing I wasn't sure about was whether or not I should remove the redis server from the redis ConnectionMultiplexer when an instance goes down, or let it continue to retry the connection. I decided to leave it retrying as it comes back into the mix as a slave as soon as it comes back up. I did some performance testing with and without a connection being retried and it seemed to make little difference. Maybe someone can clarify whether this is the correct approach.
Every now and then bringing back an instance that was previously a master did seem to cause some confusion - a few seconds after it came back up I would receive an exception from writing - "READONLY" suggesting I can't write to a slave. This was rare but I found that my "catch-all" approach of calling Configure() 12 seconds after a connection state change caught this problem. Calling Configure() seems very cheap and therefore calling it twice regardless of whether or not it's necessary seemed OK.
Now that I have slaves I have offloaded some of my data cleanup code which does key scans to the slaves, which makes me happy.
All in all I'm pretty satisfied, it's not perfect but for something that should very rarely happen it's more than good enough.
I am including our Redis wrapper, it has changed somewhat from the original answer, for various reasons:
We wanted to use pub/sub
Sentinel didn't always appear to give us the master changed message at the 'right' time (i.e. it we called Configure() and ended up thinking a slave was a master)
The connectionMultiplexer didn't always seem to restore connctions every time, affecting pub/sub
I rather suspect this is down to our sentinel/redis configuration more than anything else. Either way, it just wasn't perfectly reliable despite destructive testing. Added to which, the master changed message took a long time since we had to increase timeouts due to sentinel being "too sensitive" and calling failovers when there weren't any. I think running in a virtual environment also exacerbated the problem.
Instead of listening to subscriptions now we simply attempt a write test every 5 seconds, and also have a "last message received" check for pub/sub. If we encounter any problems we strip down completely the connections and rebuild them. It seems overkill but actually it's pretty fast and still faster than waiting for the master changed message from sentinel...
This won't compile without various extension methods and other classes etc, but you get the idea.
namespace Smartodds.Framework.Redis
{
public class RedisClient : IDisposable
{
public RedisClient(RedisEnvironmentElement environment, Int32 databaseId)
{
m_ConnectTimeout = environment.ConnectTimeout;
m_Timeout = environment.Timeout;
m_DatabaseId = databaseId;
m_ReconnectTime = environment.ReconnectTime;
m_CheckSubscriptionsTime = environment.CheckSubscriptions;
if (environment.TestWrite == true)
{
m_CheckWriteTime = environment.TestWriteTime;
}
environment.Password.ToCharArray().ForEach((c) => m_Password.AppendChar(c));
foreach (var server in environment.Servers)
{
if (server.Type == ServerType.Redis)
{
// will be ignored if sentinel servers are used
m_RedisServers.Add(new RedisConnection { Address = server.Host, Port = server.Port });
}
else
{
m_SentinelServers.Add(new RedisConnection { Address = server.Host, Port = server.Port });
}
}
}
public bool IsSentinel { get { return m_SentinelServers.Count > 0; } }
public IDatabase Database { get { return _Redis.GetDatabase(m_DatabaseId); } }
private ConnectionMultiplexer _Redis
{
get
{
if (m_Connecting == true)
{
throw new RedisConnectionNotReadyException();
}
ConnectionMultiplexer redis = m_Redis;
if (redis == null)
{
throw new RedisConnectionNotReadyException();
}
return redis;
}
}
private ConnectionMultiplexer _Sentinel
{
get
{
if (m_Connecting == true)
{
throw new RedisConnectionNotReadyException("Sentinel connection not ready");
}
ConnectionMultiplexer sentinel = m_Sentinel;
if (sentinel == null)
{
throw new RedisConnectionNotReadyException("Sentinel connection not ready");
}
return sentinel;
}
}
public void RegisterSubscription(string channel, Action<RedisChannel, RedisValue> handler, Int32 maxNoReceiveSeconds)
{
m_Subscriptions.Add(channel, new RedisSubscription
{
Channel = channel,
Handler = handler,
MaxNoReceiveSeconds = maxNoReceiveSeconds,
LastUsed = DateTime.UtcNow,
});
}
public void Connect()
{
_Connect(true);
}
private void _Connect(object state)
{
bool throwException = (bool)state;
// if a reconnect is already being attempted, don't hang around waiting
if (Monitor.TryEnter(m_ConnectionLocker) == false)
{
return;
}
// we took the lock, notify everything we are connecting
m_Connecting = true;
try
{
Stopwatch sw = Stopwatch.StartNew();
LoggerQueue.Debug(">>>>>> REDIS CONNECTING... >>>>>>");
// if this is a reconnect, make absolutely sure everything is cleaned up first
_KillTimers();
_KillRedisClient();
if (this.IsSentinel == true && m_Sentinel == null)
{
LoggerQueue.Debug(">>>>>> CONNECTING TO SENTINEL >>>>>> - " + sw.Elapsed);
// we'll be getting the redis servers from sentinel
ConfigurationOptions sentinelConnection = _CreateRedisConfiguration(CommandMap.Sentinel, null, m_SentinelServers);
m_Sentinel = ConnectionMultiplexer.Connect(sentinelConnection);
LoggerQueue.Debug(">>>>>> CONNECTED TO SENTINEL >>>>>> - " + sw.Elapsed);
_OutputConfigurationFromSentinel();
// get all the redis servers from sentinel and ignore any set by caller
m_RedisServers.Clear();
m_RedisServers.AddRange(_GetAllRedisServersFromSentinel());
if (m_RedisServers.Count == 0)
{
throw new RedisException("Sentinel found no redis servers");
}
}
LoggerQueue.Debug(">>>>>> CONNECTING TO REDIS >>>>>> - " + sw.Elapsed);
// try to connect to all redis servers
ConfigurationOptions connection = _CreateRedisConfiguration(CommandMap.Default, _SecureStringToString(m_Password), m_RedisServers);
m_Redis = ConnectionMultiplexer.Connect(connection);
LoggerQueue.Debug(">>>>>> CONNECTED TO REDIS >>>>>> - " + sw.Elapsed);
// register subscription channels
m_Subscriptions.ForEach(s =>
{
m_Redis.GetSubscriber().Subscribe(s.Key, (channel, value) => _SubscriptionHandler(channel, value));
s.Value.LastUsed = DateTime.UtcNow;
});
if (this.IsSentinel == true)
{
// check subscriptions have been sending messages
if (m_Subscriptions.Count > 0)
{
m_CheckSubscriptionsTimer = new Timer(_CheckSubscriptions, null, 30000, m_CheckSubscriptionsTime);
}
if (m_CheckWriteTime != null)
{
// check that we can write to redis
m_CheckWriteTimer = new Timer(_CheckWrite, null, 32000, m_CheckWriteTime.Value);
}
// monitor for connection status change to any redis servers
m_Redis.ConnectionFailed += _ConnectionFailure;
m_Redis.ConnectionRestored += _ConnectionRestored;
}
LoggerQueue.Debug(string.Format(">>>>>> ALL REDIS CONNECTED ({0}) >>>>>>", sw.Elapsed));
}
catch (Exception ex)
{
LoggerQueue.Error(">>>>>> REDIS CONNECT FAILURE >>>>>>", ex);
if (throwException == true)
{
throw;
}
else
{
// internal reconnect, the reconnect has failed so might as well clean everything and try again
_KillTimers();
_KillRedisClient();
// faster than usual reconnect if failure
_ReconnectTimer(1000);
}
}
finally
{
// finished connection attempt, notify everything and remove lock
m_Connecting = false;
Monitor.Exit(m_ConnectionLocker);
}
}
private ConfigurationOptions _CreateRedisConfiguration(CommandMap commandMap, string password, List<RedisConnection> connections)
{
ConfigurationOptions connection = new ConfigurationOptions
{
CommandMap = commandMap,
AbortOnConnectFail = true,
AllowAdmin = true,
ConnectTimeout = m_ConnectTimeout,
SyncTimeout = m_Timeout,
ServiceName = "master",
TieBreaker = string.Empty,
Password = password,
};
connections.ForEach(s =>
{
connection.EndPoints.Add(s.Address, s.Port);
});
return connection;
}
private void _OutputConfigurationFromSentinel()
{
m_SentinelServers.ForEach(s =>
{
try
{
IServer server = m_Sentinel.GetServer(s.Address, s.Port);
if (server.IsConnected == true)
{
try
{
IPEndPoint master = server.SentinelGetMasterAddressByName("master") as IPEndPoint;
var slaves = server.SentinelSlaves("master");
StringBuilder sb = new StringBuilder();
sb.Append(">>>>>> _OutputConfigurationFromSentinel Server ");
sb.Append(s.Address);
sb.Append(" thinks that master is ");
sb.Append(master);
sb.Append(" and slaves are ");
foreach (var slave in slaves)
{
string name = slave.Where(i => i.Key == "name").Single().Value;
bool up = slave.Where(i => i.Key == "flags").Single().Value.Contains("disconnected") == false;
sb.Append(name);
sb.Append("(");
sb.Append(up == true ? "connected" : "down");
sb.Append(") ");
}
sb.Append(">>>>>>");
LoggerQueue.Debug(sb.ToString());
}
catch (Exception ex)
{
LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Could not get configuration from sentinel server ({0}) >>>>>>", s.Address), ex);
}
}
else
{
LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Sentinel server {0} was not connected", s.Address));
}
}
catch (Exception ex)
{
LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Could not get IServer from sentinel ({0}) >>>>>>", s.Address), ex);
}
});
}
private RedisConnection[] _GetAllRedisServersFromSentinel()
{
// ask each sentinel server for its configuration
List<RedisConnection> redisServers = new List<RedisConnection>();
m_SentinelServers.ForEach(s =>
{
try
{
IServer server = m_Sentinel.GetServer(s.Address, s.Port);
if (server.IsConnected == true)
{
try
{
// store master in list
IPEndPoint master = server.SentinelGetMasterAddressByName("master") as IPEndPoint;
redisServers.Add(new RedisConnection { Address = master.Address.ToString(), Port = master.Port });
var slaves = server.SentinelSlaves("master");
foreach (var slave in slaves)
{
string address = slave.Where(i => i.Key == "ip").Single().Value;
string port = slave.Where(i => i.Key == "port").Single().Value;
redisServers.Add(new RedisConnection { Address = address, Port = Convert.ToInt32(port) });
}
}
catch (Exception ex)
{
LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Could not get redis servers from sentinel server ({0}) >>>>>>", s.Address), ex);
}
}
else
{
LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Sentinel server {0} was not connected", s.Address));
}
}
catch (Exception ex)
{
LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Could not get IServer from sentinel ({0}) >>>>>>", s.Address), ex);
}
});
return redisServers.Distinct().ToArray();
}
private IServer _GetRedisMasterFromSentinel()
{
// ask each sentinel server for its configuration
foreach (RedisConnection sentinel in m_SentinelServers)
{
IServer sentinelServer = _Sentinel.GetServer(sentinel.Address, sentinel.Port);
if (sentinelServer.IsConnected == true)
{
try
{
IPEndPoint master = sentinelServer.SentinelGetMasterAddressByName("master") as IPEndPoint;
return _Redis.GetServer(master);
}
catch (Exception ex)
{
LoggerQueue.Error(string.Format(">>>>>> Could not get redis master from sentinel server ({0}) >>>>>>", sentinel.Address), ex);
}
}
}
throw new InvalidOperationException("No sentinel server available to get master");
}
private void _ReconnectTimer(Nullable<Int32> reconnectMilliseconds)
{
try
{
lock (m_ReconnectLocker)
{
if (m_ReconnectTimer != null)
{
m_ReconnectTimer.Dispose();
m_ReconnectTimer = null;
}
// since a reconnect will definately occur we can stop the check timers for now until reconnect succeeds (where they are recreated)
_KillTimers();
LoggerQueue.Warn(">>>>>> REDIS STARTING RECONNECT TIMER >>>>>>");
m_ReconnectTimer = new Timer(_Connect, false, reconnectMilliseconds.GetValueOrDefault(m_ReconnectTime), Timeout.Infinite);
}
}
catch (Exception ex)
{
LoggerQueue.Error("Error during _ReconnectTimer", ex);
}
}
private void _CheckSubscriptions(object state)
{
if (Monitor.TryEnter(m_ConnectionLocker, TimeSpan.FromSeconds(1)) == false)
{
return;
}
try
{
DateTime now = DateTime.UtcNow;
foreach (RedisSubscription subscription in m_Subscriptions.Values)
{
if ((now - subscription.LastUsed) > TimeSpan.FromSeconds(subscription.MaxNoReceiveSeconds))
{
try
{
EndPoint endpoint = m_Redis.GetSubscriber().IdentifyEndpoint(subscription.Channel);
EndPoint subscribedEndpoint = m_Redis.GetSubscriber().SubscribedEndpoint(subscription.Channel);
LoggerQueue.Warn(string.Format(">>>>>> REDIS Channel '{0}' has not been used for longer than {1}s, IsConnected: {2}, IsConnectedChannel: {3}, EndPoint: {4}, SubscribedEndPoint: {5}, reconnecting...", subscription.Channel, subscription.MaxNoReceiveSeconds, m_Redis.GetSubscriber().IsConnected(), m_Redis.GetSubscriber().IsConnected(subscription.Channel), endpoint != null ? endpoint.ToString() : "null", subscribedEndpoint != null ? subscribedEndpoint.ToString() : "null"));
}
catch (Exception ex)
{
LoggerQueue.Error(string.Format(">>>>>> REDIS Error logging out details of Channel '{0}' reconnect", subscription.Channel), ex);
}
_ReconnectTimer(null);
return;
}
}
}
catch (Exception ex)
{
LoggerQueue.Error(">>>>>> REDIS Exception ERROR during _CheckSubscriptions", ex);
}
finally
{
Monitor.Exit(m_ConnectionLocker);
}
}
private void _CheckWrite(object state)
{
if (Monitor.TryEnter(m_ConnectionLocker, TimeSpan.FromSeconds(1)) == false)
{
return;
}
try
{
this.Database.HashSet(Environment.MachineName + "SmartoddsWriteCheck", m_CheckWriteGuid.ToString(), DateTime.UtcNow.Ticks);
}
catch (RedisConnectionNotReadyException)
{
LoggerQueue.Warn(">>>>>> REDIS RedisConnectionNotReadyException ERROR DURING _CheckWrite");
}
catch (RedisServerException ex)
{
LoggerQueue.Warn(">>>>>> REDIS RedisServerException ERROR DURING _CheckWrite, reconnecting... - " + ex.Message);
_ReconnectTimer(null);
}
catch (RedisConnectionException ex)
{
LoggerQueue.Warn(">>>>>> REDIS RedisConnectionException ERROR DURING _CheckWrite, reconnecting... - " + ex.Message);
_ReconnectTimer(null);
}
catch (TimeoutException ex)
{
LoggerQueue.Warn(">>>>>> REDIS TimeoutException ERROR DURING _CheckWrite - " + ex.Message);
}
catch (Exception ex)
{
LoggerQueue.Error(">>>>>> REDIS Exception ERROR during _CheckWrite", ex);
}
finally
{
Monitor.Exit(m_ConnectionLocker);
}
}
private void _ConnectionFailure(object sender, ConnectionFailedEventArgs e)
{
LoggerQueue.Warn(string.Format(">>>>>> REDIS CONNECTION FAILURE, {0}, {1}, {2} >>>>>>", e.ConnectionType, e.EndPoint.ToString(), e.FailureType));
}
private void _ConnectionRestored(object sender, ConnectionFailedEventArgs e)
{
LoggerQueue.Warn(string.Format(">>>>>> REDIS CONNECTION RESTORED, {0}, {1}, {2} >>>>>>", e.ConnectionType, e.EndPoint.ToString(), e.FailureType));
}
private void _SubscriptionHandler(string channel, RedisValue value)
{
// get handler lookup
RedisSubscription subscription = null;
if (m_Subscriptions.TryGetValue(channel, out subscription) == false || subscription == null)
{
return;
}
// update last used
subscription.LastUsed = DateTime.UtcNow;
// call handler
subscription.Handler(channel, value);
}
public Int64 Publish(string channel, RedisValue message)
{
try
{
return _Redis.GetSubscriber().Publish(channel, message);
}
catch (RedisConnectionNotReadyException)
{
LoggerQueue.Error("REDIS RedisConnectionNotReadyException ERROR DURING Publish");
throw;
}
catch (RedisServerException ex)
{
LoggerQueue.Error("REDIS RedisServerException ERROR DURING Publish - " + ex.Message);
throw;
}
catch (RedisConnectionException ex)
{
LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING Publish - " + ex.Message);
throw;
}
catch (TimeoutException ex)
{
LoggerQueue.Error("REDIS TimeoutException ERROR DURING Publish - " + ex.Message);
throw;
}
catch (Exception ex)
{
LoggerQueue.Error("REDIS Exception ERROR DURING Publish", ex);
throw;
}
}
public bool LockTake(RedisKey key, RedisValue value, TimeSpan expiry)
{
return _Execute(() => this.Database.LockTake(key, value, expiry));
}
public bool LockExtend(RedisKey key, RedisValue value, TimeSpan extension)
{
return _Execute(() => this.Database.LockExtend(key, value, extension));
}
public bool LockRelease(RedisKey key, RedisValue value)
{
return _Execute(() => this.Database.LockRelease(key, value));
}
private void _Execute(Action action)
{
try
{
action.Invoke();
}
catch (RedisServerException ex)
{
LoggerQueue.Error("REDIS RedisServerException ERROR DURING _Execute - " + ex.Message);
throw;
}
catch (RedisConnectionException ex)
{
LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING _Execute - " + ex.Message);
throw;
}
catch (TimeoutException ex)
{
LoggerQueue.Error("REDIS TimeoutException ERROR DURING _Execute - " + ex.Message);
throw;
}
catch (Exception ex)
{
LoggerQueue.Error("REDIS Exception ERROR DURING _Execute", ex);
throw;
}
}
private TResult _Execute<TResult>(Func<TResult> function)
{
try
{
return function.Invoke();
}
catch (RedisServerException ex)
{
LoggerQueue.Error("REDIS RedisServerException ERROR DURING _Execute - " + ex.Message);
throw;
}
catch (RedisConnectionException ex)
{
LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING _Execute - " + ex.Message);
throw;
}
catch (TimeoutException ex)
{
LoggerQueue.Error("REDIS TimeoutException ERROR DURING _Execute - " + ex.Message);
throw;
}
catch (Exception ex)
{
LoggerQueue.Error("REDIS ERROR DURING _Execute", ex);
throw;
}
}
public string[] GetAllKeys(string pattern)
{
if (m_Sentinel != null)
{
return _GetAnyRedisSlaveFromSentinel().Keys(m_DatabaseId, pattern).Select(k => (string)k).ToArray();
}
else
{
return _Redis.GetServer(_Redis.GetEndPoints().First()).Keys(m_DatabaseId, pattern).Select(k => (string)k).ToArray();
}
}
private void _KillSentinelClient()
{
try
{
if (m_Sentinel != null)
{
LoggerQueue.Debug(">>>>>> KILLING SENTINEL CONNECTION >>>>>>");
ConnectionMultiplexer sentinel = m_Sentinel;
m_Sentinel = null;
sentinel.Close(false);
sentinel.Dispose();
}
}
catch (Exception ex)
{
LoggerQueue.Error(">>>>>> Error during _KillSentinelClient", ex);
}
}
private void _KillRedisClient()
{
try
{
if (m_Redis != null)
{
Stopwatch sw = Stopwatch.StartNew();
LoggerQueue.Debug(">>>>>> KILLING REDIS CONNECTION >>>>>>");
ConnectionMultiplexer redis = m_Redis;
m_Redis = null;
if (this.IsSentinel == true)
{
redis.ConnectionFailed -= _ConnectionFailure;
redis.ConnectionRestored -= _ConnectionRestored;
}
redis.Close(false);
redis.Dispose();
LoggerQueue.Debug(">>>>>> KILLED REDIS CONNECTION >>>>>> " + sw.Elapsed);
}
}
catch (Exception ex)
{
LoggerQueue.Error(">>>>>> Error during _KillRedisClient", ex);
}
}
private void _KillClients()
{
lock (m_ConnectionLocker)
{
_KillSentinelClient();
_KillRedisClient();
}
}
private void _KillTimers()
{
if (m_CheckSubscriptionsTimer != null)
{
m_CheckSubscriptionsTimer.Dispose();
m_CheckSubscriptionsTimer = null;
}
if (m_CheckWriteTimer != null)
{
m_CheckWriteTimer.Dispose();
m_CheckWriteTimer = null;
}
}
public void Dispose()
{
_KillClients();
_KillTimers();
}
}
}
I just asked this question, and found a similar question to yours and mine which I believe answers the question of how does our code (the client) know now which is the new master server when the current master goes down?
How to tell a Client where the new Redis master is using Sentinel
Apparently you just have to subscribe and listen to events from the Sentinels. Makes sense.. I just figured there was a more streamlined way.
I read something about the Twemproxy for Linux which acts as a proxy and probably does this for you? But I was on redis for Windows and was trying to find a Windows option. We might just moved to Linux if that's the approved way it should be done.
Today (I just configured StackExchange.Redis 2.1.58 to use sentinel) it's enough to specify a sentinel endpoint and serviceName in the redis connection string or Configuration. All the rest has been encapsulated as a part of this commit. So you just point stackexchange.redis to your sentinel nodes and ConnectionMuliplexer gives you up and running IDatabase each time you call GetDatabase().
var conn = ConnectionMultiplexer.Connect("sentinel:26379,serviceName=mymaster");
var db = conn.GetDatabase();
db.StringSet("key", "value");
I am getting a lot of files that are not being uploaded to my AWS bucket but it the code below always shows complete. Anyone know whats up with this? How am I supposed to find out if the upload failed or not if it always comes back complete? I did see something about using transferutility.EndUpload but I am unsure on how to use it. Also how would I implement a retry? Pass the object state to BeginUpload? Any help?
public class S3Upload
{
private string awsAccessKeyId = "XXXXXX";
private string awsSecretAccessKey = "XXXXXX";
private string bucketName = System.Configuration.ConfigurationManager.AppSettings.Get("BucketName");
private Amazon.S3.Transfer.TransferUtility transferUtility;
private static log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
public S3Upload()
{
// Initialize log4net.
log4net.Config.XmlConfigurator.Configure();
this.transferUtility = new Amazon.S3.Transfer.TransferUtility(awsAccessKeyId, awsSecretAccessKey);
log.Info("S3 instance initiated");
}
public void UploadFile(string filePath, string toPath)
{
try
{
AsyncCallback callback = new AsyncCallback(uploadComplete);
log.Info("S3 upload started...");
log.InfoFormat("S3 filePath: {0}", filePath);
log.InfoFormat("S3 toPath: {0}", toPath);
var uploadRequest = new Amazon.S3.Transfer.TransferUtilityUploadRequest();
uploadRequest.FilePath = filePath;
uploadRequest.BucketName = bucketName;
uploadRequest.Key = toPath;
uploadRequest.StorageClass = Amazon.S3.Model.S3StorageClass.ReducedRedundancy;
uploadRequest.AddHeader("x-amz-acl", "public-read");
transferUtility.BeginUpload(uploadRequest, callback, null);
}
catch (AmazonS3Exception amazonS3Exception)
{
log.ErrorFormat("An Error, number {0}, occurred when creating a bucket with the message '{1}", amazonS3Exception.ErrorCode, amazonS3Exception.Message);
}
}
private void uploadComplete(IAsyncResult result)
{
var x = result;
if (x.IsCompleted)
{
log.Info("S3 upload completed...");
}
}
}
Found a fix! I added transferUtility.EndUpload(ar) and if it error's it will write the exception to my log and retry the put request.
Error making request PutObject.
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. --->
public class S3Upload
{
private string awsAccessKeyId = "XXXXX";
private string awsSecretAccessKey = "XXXXX";
private string bucketName = System.Configuration.ConfigurationManager.AppSettings.Get("BucketName");
private Amazon.S3.Transfer.TransferUtility transferUtility;
private static log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
public S3Upload()
{
// Initialize log4net.
log4net.Config.XmlConfigurator.Configure();
this.transferUtility = new Amazon.S3.Transfer.TransferUtility(awsAccessKeyId, awsSecretAccessKey);
log.Info("S3 instance initiated");
}
public void UploadFile(string filePath, string toPath)
{
try
{
AsyncCallback callback = new AsyncCallback(uploadComplete);
log.Info("S3 upload started...");
log.InfoFormat("S3 filePath: {0}", filePath);
log.InfoFormat("S3 toPath: {0}", toPath);
var uploadRequest = new Amazon.S3.Transfer.TransferUtilityUploadRequest();
uploadRequest.FilePath = filePath;
uploadRequest.BucketName = bucketName;
uploadRequest.Key = toPath;
uploadRequest.StorageClass = Amazon.S3.Model.S3StorageClass.ReducedRedundancy;
uploadRequest.AddHeader("x-amz-acl", "public-read");
IAsyncResult ar = transferUtility.BeginUpload(uploadRequest, callback, null);
transferUtility.EndUpload(ar);
}
catch (AmazonS3Exception amazonS3Exception)
{
log.ErrorFormat("An Error, number {0}, occurred when creating a bucket with the message '{1}", amazonS3Exception.ErrorCode, amazonS3Exception.Message);
}
}
private void uploadComplete(IAsyncResult result)
{
var x = result;
}
}
The BeginUpload function starts a new thread that initiates and performs the upload. Calling EndUpload causes the current thread to block until the upload is complete. Also, the EndUpload function catches any exceptions that you'll get from the upload.
If you're going to call BeginUpload, you need to call EndUpload() from a separate thread so that you're not blocking your main thread. If you plan on blocking your main thread, just call the Upload() function instead of the APM versions.
For example, spawn a new thread that will sit and wait for the upload to complete. (Note that it doesn't have to do anything specific, just catch the errors.)
public void UploadFile(string filePath, string toPath)
{
try
{
AsyncCallback callback = new AsyncCallback(uploadComplete);
log.Info("S3 upload started...");
log.InfoFormat("S3 filePath: {0}", filePath);
log.InfoFormat("S3 toPath: {0}", toPath);
var uploadRequest = new Amazon.S3.Transfer.TransferUtilityUploadRequest();
uploadRequest.FilePath = filePath;
uploadRequest.BucketName = bucketName;
uploadRequest.Key = toPath;
uploadRequest.StorageClass = Amazon.S3.Model.S3StorageClass.ReducedRedundancy;
uploadRequest.AddHeader("x-amz-acl", "public-read");
IAsyncResult ar = transferUtility.BeginUpload(uploadRequest, callback, null);
// transferUtility.EndUpload(ar);
ThreadPool.QueueUserWorkItem(c =>
{
try
{
transferUtility.EndUpload(ar);
}
catch (AmazonS3Exception ex)
{
// Retry, log the error, etc.
}
catch (WebException ex)
{
// Retry, log the error, etc.
}
});
}
catch (AmazonS3Exception amazonS3Exception)
{
log.ErrorFormat("An Error, number {0}, occurred when creating a bucket with the message '{1}", amazonS3Exception.ErrorCode, amazonS3Exception.Message);
}
}
Also, that thread checks for WebException. I don't know if you can get one of these from the Beginupload or not, but some code I'm working on seems to watch for those. (Better safe then sorry.)
Also, I found you can test this with the CurrPorts Utility. That utility lets you kill the open port to amazon (although it will retry up until its configured MaxRetries).
I've got some network code to process an arbitary TCP connection.
It all seems to work as expected but seems slow. When i've profiled the code the it seems to spend a good 600 ms in NetworkStream.Read() and I'm wondering how to improve it. I've fiddled with the buffer sizes and alternated between a massive buffer to read all of the data in one go or a small one which should concatenate the data into a StringBuilder. Currently the client i'm using is a web-browser but this code is generic and it may well not be HTTP data that is being sent to it. Any ideas?
My code is this:
public void StartListening()
{
try
{
lock (oSyncRoot)
{
oTCPListener = new TcpListener(oIPaddress, nPort);
// fire up the server
oTCPListener.Start();
// set listening bit
bIsListening = true;
}
// Enter the listening loop.
do
{
// Wait for connection
TcpClient newClient = oTCPListener.AcceptTcpClient();
// queue a request to take care of the client
oThreadPool.QueueUserWorkItem(new WaitCallback(ProcessConnection), newClient);
}
while (bIsListening);
}
catch (SocketException se)
{
Logger.Write(new TCPLogEntry("SocketException: " + se.ToString()));
}
finally
{
// shut it down
StopListening();
}
}
private void ProcessConnection(object oClient)
{
TcpClient oTCPClient = (TcpClient)oClient;
try
{
byte[] abBuffer = new byte[1024];
StringBuilder sbReceivedData = new StringBuilder();
using (NetworkStream oNetworkStream = oTCPClient.GetStream())
{
// set initial read timeout to nInitialTimeoutMS to allow for connection
oNetworkStream.ReadTimeout = nInitialTimeoutMS;
int nBytesRead = 0;
do
{
try
{
bool bDataAvailable = oNetworkStream.DataAvailable;
while (!bDataAvailable)
{
Thread.Sleep(5);
bDataAvailable = oNetworkStream.DataAvailable;
}
nBytesRead = oNetworkStream.Read(abBuffer, 0, abBuffer.Length);
if (nBytesRead > 0)
{
// Translate data bytes to an ASCII string and append
sbReceivedData.Append(Encoding.UTF8.GetString(abBuffer, 0, nBytesRead));
// decrease read timeout to nReadTimeoutMS second now that data is coming in
oNetworkStream.ReadTimeout = nReadTimeoutMS;
}
}
catch (IOException)
{
// read timed out, all data has been retrieved
nBytesRead = 0;
}
}
while (nBytesRead > 0);
//send the data to the callback and get the response back
byte[] abResponse = oClientHandlerDelegate(sbReceivedData.ToString(), oTCPClient);
if (abResponse != null)
{
oNetworkStream.Write(abResponse, 0, abResponse.Length);
oNetworkStream.Flush();
}
}
}
catch (Exception e)
{
Logger.Write(new TCPLogEntry("Caught Exception " + e.StackTrace));
}
finally
{
// stop talking to client
if (oTCPClient != null)
{
oTCPClient.Close();
}
}
}
Edit: I get roughly the same figures on two entirely seperate machines (my XP development machine and a 2003 box in a colo). I've put some timing into the code around the relevant parts (using System.Diagnostic.StopWatch) and dump it to a log:
7/6/2009 3:44:50 PM : Debug : While DataAvailable took 0 ms
7/6/2009 3:44:50 PM : Debug : Read took 531 ms
7/6/2009 3:44:50 PM : Debug : ProcessConnection took 577 ms
I recommend you use Microsoft Network Monitor or something like it to see what's going on in terms of those 600ms. NetworkStream is a piece of networking software - when looking at its behavior, always consider what the network is doing.
Another vote for the use of network monitoring software. Either Network Monitor or WireShark should do. Make sure you record what time the networkstream.read call begins and ends in your program so you can know where in the recorded network traffic your program events happened.
Also, I'd recommend waiting for the NetworkStream.DataAvailable property to become true before you call the Read method, and record the time it becomes true as well. If your network monitor shows data arriving 600 ms before your program indicates it can be read, something else on your computer may be holding up the packet - e.g. antivirus or your firewall.
Addendum 2009/7/6 3:12 PM EDT:
The extra timing information you posted is interesting. If data is available, why is it taking so long to read? I ran your code on my development machine, and both waiting for dataavailable and the read function itself comes out as 0 milliseconds. Are you sure you have the latest service packs, etc. installed? I'm running Visual Studio Professional 2005 with .NET 2.0.50727. I also have .NET 3.0 and 3.5 installed, but I don't think VS 2005 is using those. Do you have a fresh OS installation (real or virtual machine) with no extra programs (even/especially ones "required" by corporate IT) that you could try this on?
Here's the code I ran:
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Threading;
using System.Diagnostics;
namespace stackoverflowtest
{
class Program
{
static private object oSyncRoot = new object();
static private TcpListener oTCPListener;
static private IPAddress oIPaddress = IPAddress.Parse("10.1.1.109");
static private int nPort = 8009;
static bool bIsListening = true;
static void Main(string[] args)
{
StartListening();
Thread.Sleep(500000);
bIsListening = false;
}
public static void StartListening()
{
try
{
lock (oSyncRoot)
{
oTCPListener = new TcpListener(oIPaddress, nPort);
// fire up the server
oTCPListener.Start();
// set listening bit
bIsListening = true;
}
// Enter the listening loop.
do
{
// Wait for connection
TcpClient newClient = oTCPListener.AcceptTcpClient();
// queue a request to take care of the client
ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessConnection), newClient);
}
while (bIsListening);
}
catch (SocketException se)
{
Console.WriteLine("SocketException: " + se.ToString());
}
finally
{
// shut it down
//StopListening();
}
}
private static void ProcessConnection(object oClient)
{
TcpClient oTCPClient = (TcpClient)oClient;
try
{
byte[] abBuffer = new byte[1024];
StringBuilder sbReceivedData = new StringBuilder();
using (NetworkStream oNetworkStream = oTCPClient.GetStream())
{
int nInitialTimeoutMS = 1000;
// set initial read timeout to nInitialTimeoutMS to allow for connection
oNetworkStream.ReadTimeout = nInitialTimeoutMS;
int nBytesRead = 0;
do
{
try
{
bool bDataAvailable = oNetworkStream.DataAvailable;
Stopwatch sw = new Stopwatch();
while (!bDataAvailable)
{
Thread.Sleep(5);
bDataAvailable = oNetworkStream.DataAvailable;
}
Console.WriteLine("DataAvailable loop took " + sw.ElapsedMilliseconds);
sw.Reset();
nBytesRead = oNetworkStream.Read(abBuffer, 0, abBuffer.Length);
Console.WriteLine("Reading " + nBytesRead + " took " + sw.ElapsedMilliseconds);
if (nBytesRead > 0)
{
// Translate data bytes to an ASCII string and append
sbReceivedData.Append(Encoding.UTF8.GetString(abBuffer, 0, nBytesRead));
// decrease read timeout to nReadTimeoutMS second now that data is coming in
int nReadTimeoutMS = 100;
oNetworkStream.ReadTimeout = nReadTimeoutMS;
}
}
catch (IOException)
{
// read timed out, all data has been retrieved
nBytesRead = 0;
}
}
while (nBytesRead > 0);
byte[] abResponse = new byte[1024];
for (int i = 0; i < abResponse.Length; i++)
{
abResponse[i] = (byte)i;
}
oNetworkStream.Write(abResponse, 0, abResponse.Length);
oNetworkStream.Flush();
//send the data to the callback and get the response back
//byte[] abResponse = oClientHandlerDelegate(sbReceivedData.ToString(), oTCPClient);
//if (abResponse != null)
//{
// oNetworkStream.Write(abResponse, 0, abResponse.Length);
// oNetworkStream.Flush();
//}
}
}
catch (Exception e)
{
Console.WriteLine("Caught Exception " + e.StackTrace);
}
finally
{
// stop talking to client
if (oTCPClient != null)
{
oTCPClient.Close();
}
}
}
}
}
After some more research it seems that the only way to speed this up is to break after the first x bytes have been read. The delay seems to be on the second read. If I change the buffer to be 8096 bytes (probably the max my application will be sent at any one go) and break here:
if (nBytesRead > 0)
{
// Translate data bytes to an ASCII string and append
sbReceivedData.Append(Encoding.UTF8.GetString(abBuffer, 0, nBytesRead));
if (bTurboMode)
{
break;
}
else
{
// decrease read timeout to nReadTimeoutMS second now that data is coming in
oNetworkStream.ReadTimeout = nReadTimeoutMS;
}
}
Then the response time goes from 600ms to about 80ms. This is an acceptable solution for me currently. I can toggle the bTurboMode from the calling application and speed things up substantially for this case