I am developing a game in which I need to retrieve data from a stream (that hasn't end).
I have a class called StreamingChannel which creates the streaming channel
public StreamingChannel (){
//stuff to set the stream
webResponse = (HttpWebResponse) webRequest.GetResponse();
responseStream = new StreamReader (webResponse.GetResponseStream (), encode);
}
and to read from it i have this method
public string Read(){
try{
string jsonText = responseStream.ReadLine();
return jsonText;
}catch(ObjectDisposedException){
return null;
}
}
I perform the reading every tot secs with an InvokeRepeating and I do that for the whole game.
It works great except that for the fact that my stream lasts for about a couple of minute. After that it throws an ObjectDisposedException.
At first I wanted to restore the connection but I didn't manage to do that without reinstantiate the whole connection. In this case the problem is that the game lags for about a seconds.
So how can I tell the StreamReader that has to leave open the channel?
ps I cannot use the constructor
public StreamReader(
Stream stream,
Encoding encoding,
bool detectEncodingFromByteOrderMarks,
int bufferSize,
bool leaveOpen)
because it has been introduced in the version 4.5 of the .NET Framework, and Unity doesn't support that.
A streaming API expects your code to pull data out of Stream pretty aggressively. You may not be able to wait for Unity to schedule your ReadLine method. I think a better model is to use a separate thread to pull data as fast as possible from the Stream and store it in a buffer. (I think this is possible in Unity.) Then you can pull the stream data out of your buffer in the standard Unity thread without worrying about the pull rate. A ConcurrentQueue would be a great buffer, but Unity doesn't support it, so I've used a locked List.
Using a separate thread also allows you to restart after failures without blocking the main game.
using System.Collections.Generic;
using System.Threading;
public class StreamingChannel
{
private readonly List<string> backgroundLinesList;
private readonly object listLock = new object();
private Thread streamReaderThread;
public StreamingChannel()
{
streamReaderThread = new Thread(this.ReadWebStream);
streamReaderThread.Start();
}
public List<string> Read()
{
if (!streamReaderThread.IsAlive)
{
streamReaderThread = new Thread(this.ReadWebStream);
streamReaderThread.Start();
}
List<string> lines = null;
lock (listLock)
{
if (backgroundLinesList != null)
{
lines = backgroundLinesList;
backgroundLinesList = null;
}
}
return lines;
}
private void ReadWebStream()
{
try
{
//stuff to set the stream
HttpWebRequest webRequest;
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
StreamReader responseStream = new StreamReader(webResponse.GetResponseStream(), encode);
while (!responseStream.EndOfStream)
{
var line = responseStream.ReadLine()
lock (listLock)
{
if (backgroundLinesList == null)
{
backgroundLinesList = new List<string>();
}
backgroundLinesList.Add(line);
}
}
log.Debug("Stream closed");
}
catch (Exception e)
{
log.Debug("WebStream thread failure: " + e + " Stack: " + e.StackTrace);
}
}
}
Related
Here is how i read data from my stream now:
public List<ServerClient> clients = new List<ServerClient>();
while (true)
{
Update();
}
private void Update()
{
//Console.WriteLine("Call");
if (!serverStarted)
{
return;
}
foreach (ServerClient c in clients.ToList())
{
// Is the client still connected?
if (!IsConnected(c.tcp))
{
c.tcp.Close();
disconnectList.Add(c);
Console.WriteLine(c.connectionId + " has disconnected.");
CharacterLogout(c.connectionId);
continue;
//Console.WriteLine("Check for connection?\n");
}
else
{
// Check for message from Client.
NetworkStream s = c.tcp.GetStream();
if (s.DataAvailable)
{
string data = c.streamReader.ReadLine();
if (data != null)
{
OnIncomingData(c, data);
}
}
//continue;
}
}
for (int i = 0; i < disconnectList.Count - 1; i++)
{
clients.Remove(disconnectList[i]);
disconnectList.RemoveAt(i);
}
}
When data is read it is send to OnIncomingData function which is processing the data. I don't have problems there.
Here is how i send data to the stream:
public void Send(string header, Dictionary data)
{
if (stream.CanRead)
{
socketReady = true;
}
if (!socketReady)
{
return;
}
JsonData SendData = new JsonData();
SendData.header = "1x" + header;
foreach (var item in data)
{
SendData.data.Add(item.Key.ToString(), item.Value.ToString());
}
SendData.connectionId = connectionId;
string json = JsonConvert.SerializeObject(SendData);
var howManyBytes = json.Length * sizeof(Char);
writer.WriteLine(json);
writer.Flush();
Debug.Log("Client World:" + json);
}
Here is my:
public class ServerClient
{
public TcpClient tcp;
public int accountId;
public StreamReader streamReader;
public int connectionId;
public ServerClient(TcpClient clientSocket)
{
tcp = clientSocket;
}
}
Here is my OnConnection function:
private void OnConnection(IAsyncResult ar)
{
connectionIncrementor++;
TcpListener listener = (TcpListener)ar.AsyncState;
NetworkStream s = clients[clients.Count - 1].tcp.GetStream();
clients.Add(new ServerClient(listener.EndAcceptTcpClient(ar)));
clients[clients.Count - 1].connectionId = connectionIncrementor;
clients[clients.Count - 1].streamReader = new StreamReader(s, true);
StartListening();
//Send a message to everyone, say someone has connected!
Dictionary<string, string> SendDataBroadcast = new Dictionary<string, string>();
SendDataBroadcast.Add("connectionId", clients[clients.Count - 1].connectionId.ToString());
Broadcast("001", SendDataBroadcast, clients, clients[clients.Count - 1].connectionId);
Console.WriteLine(clients[clients.Count - 1].connectionId + " has connected.");
}
Normally everything works fine. However if i try to send more request per 1 second the problem occurs. The message received is not full and complete. It just receives a portion of the message sent.
From Debug.Log("Client World:" + json); i can see that the message is full and complete but on the server i see that it is not.
This is not happening if i send less requests.
So for that reason i think i should create a MemoryStream and puts a message there and read it after. However i'm really not sure how i can do that. Can you help ?
The whole code is not very good, but I'll concentrate on your specific problem. It's most likely related to data buffering by StreamReader. StreamReader has buffer size (which you can pass to constructor) which defaults to 1024 bytes. When you call ReadLine - it's perfectly possible for stream reader to read more than one line from the underlying stream. In your case - you have while loop in which you enumerate connected clients and in every iteration of the loop you create new StreamReader and read one line from it. When message rate is low - all looks fine, because between your loop iterations only one line arrives. Now suppose client quickly sent 2 json messages, each of which is 800 bytes, and they both arrived into your socket. Now you call StreamReader.ReadLine. Because buffer size is 1024 - it will read 1024 bytes from socket (NetworkStream) and return first 800 to you (as a line). You process that line and discard StreamReader going to the next iteration of your while loop. By doing that you also discard part of the message (224 bytes of the next message), because they were already read from the socket into StreamReader buffer. I think from that it should be clear how to solve that problem - don't create new StreamReader every time but create one per client (for example store as a member of ServerClient) and use that.
The client looks more suspicious to me than the server.
StreamWriter is not thread-safe. Are you calling it in a thread-safe manner when using ClientWorldServer.Send? Lock up or queue your calls to ClientWorldServer.Send using a lock or BlockingCollection or some other synchronisation primitive. There is also a thread-safe wrapper of streamwriter you might be able to use.
The documentation for DataReader's DetachBuffer and DetachStream is very vague. It just says `Detaches a buffer that was previously attached to the reader'.
In short
When should reader.DetachBuffer(); be used?
Background
Reading
An example read method for a SerialDevice could look something like this:
using (var reader = new DataReader(inputStream))
{
var bytesReceived = await reader.LoadAsync(EXPECTED_RESPONSE_LENGTH);
var receivedBuffer = new byte[bytesReceived];
reader.ReadBytes(receivedBuffer);
reader.DetachStream();
return receivedBuffer;
}
This code works and seems to be stable, but since I write and read multiple times a second on an embedded device I want to avoid creating the receivedBuffer buffer each time. I modified my method to be something like the code below.
byte[] _receivedBuffer = new byte[EXPECTED_RESPONSE_LENGTH];
private async Task<byte[]> ReadOnceAsync(IInputStream inputStream)
{
using (var reader = new DataReader(inputStream))
{
reader.InputStreamOptions = InputStreamOptions.Partial;
uint bytesReceived = await reader.LoadAsync(EXPECTED_RESPONSE_LENGTH);
var isExpectedLength = (bytesReceived == EXPECTED_RESPONSE_LENGTH);
if (isExpectedLength)
{
reader.ReadBytes(_receivedBuffer);
}
reader.DetachStream();
return isExpectedLength ? _receivedBuffer: null;
}
}
This code crashes my application, sometimes with Access Violation message, within minutes of starting or within seconds if the connected device stops responding.
After I added reader.DetachBuffer(); the code is stable again, but I still don't know if DetachBuffer should be called always, sometimes or not at all.
Writing
My write method does not call writer.DetachStream() but I don't know if it should or not. The code is:
using (var writer = new DataWriter(outputStream))
{
writer.WriteBytes(toSend);
var bytesWritten = await writer.StoreAsync();
//Should writer.DetachBuffer(); be called?
writer.DetachStream();
return bytesWritten;
}
i think i'm missing something about how HttpWebRequest works via streaming when uploading large files.
basicly, i found out that i receive timeout exception when sending large files to the server, so a post suggested to do it via Async and handle the timeout myself.
The thing is, that after debugging, i found out that "GetRequestStreamAsync" method, and writing to it does nothing at the server side, the server is called only when doing GetResponseAsync
so my question is:
- code marked as //1 - it writes the file to the request stream, but i don't see that the memory is increasing, or the server even getting any request - where does the streaming go to?
This is basicly my code:
HttpWebRequest request = RESTUtils.InitializeRequest(...);
request.AllowWriteStreamBuffering = false;
request.ContentLength = i_InputStream.Length;
request.Timeout = 5000;
using (Stream requestStream = request.GetRequestStreamWithTimeout())
{
if (requestStream != null) //1
{
// We will write the stream to the request
byte[] buffer = new byte[UPLOAD_FILE_BUFFER_SIZE];
int read = i_InputStream.Read(buffer, 0, buffer.Length);
while (read > 0)
{
requestStream.Write(buffer, 0, read);
read = i_InputStream.Read(buffer, 0, buffer.Length);
}
}
}
using (var response = request.GetResponseWithTimeout(-1))
{
using (var responseStream = response.GetResponseStream())
{
}
}
public static class WebRequestExtensions
{
public static Stream GetRequestStreamWithTimeout(
this WebRequest request,
int? millisecondsTimeout = null)
{
return AsyncToSyncWithTimeout(
request.BeginGetRequestStream,
request.EndGetRequestStream,
millisecondsTimeout ?? request.Timeout);
}
public static WebResponse GetResponseWithTimeout(
this HttpWebRequest request,
int? millisecondsTimeout = null)
{
return AsyncToSyncWithTimeout(
request.BeginGetResponse,
request.EndGetResponse,
millisecondsTimeout ?? request.Timeout);
}
private static T AsyncToSyncWithTimeout<T>(
Func<AsyncCallback, object, IAsyncResult> begin,
Func<IAsyncResult, T> end,
int millisecondsTimeout)
{
var iar = begin(null, null);
if (!iar.AsyncWaitHandle.WaitOne(millisecondsTimeout))
{
var ex = new TimeoutException();
throw new WebException(ex.Message, ex, WebExceptionStatus.Timeout, null);
}
return end(iar);
}
}
Thanks!
== Edit 9/9/15 ==
Something even weirder happens, i'm attaching breakpoint right after GetResponseAsync, then i see that the server receives the call.
after that, i'm closing the process of the client -> the server is uploading the file successfully.
this happens also if i do "Abort".
anyone knows why?
Instead of using the old-style begin/end async pattern, you should consider switching to async/await which would greatly simplify your code.
You would then set the Timeout property against the request to a large value to accommodate your waiting time; then instead of using the callback-based async code, you could just do:
var request = SomeMethodToCreateRequest();
request.Timeout = int.MaxValue; // (don't do this)
var response = await request.GetResponse();
The timeout should be respected internally, and you get to simplify your code.
Following this tutorial http://msdn.microsoft.com/en-us/library/hh221581.aspx I created an HttpWebRequest.
Producing this code for the Callback function:
private void ReadCallback(IAsyncResult result)
{
HttpWebRequest request = result.AsyncState as HttpWebRequest;
if (request != null)
{
try
{
WebResponse response = request.EndGetResponse(result);
using (StreamReader streamReader1 = new StreamReader(response.GetResponseStream()))
{
string resultString = streamReader1.ReadToEnd();
}
}
catch (WebException e)
{
return;
}
}
}
Now I got some data in the resultString, but I can't return it the normal way because of the call being async (as one can read here: AsyncCallBack - Does it have to be static / Does it have to return void?).
I can create global variables and safe the resultString global to access it from everywhere, but I don't think that this is the proper way to do something like this. The MSDN just writes the results to the console (http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest(v=vs.95).aspx), not really the thing I want to.
Is there a "best practice" or something for proceeding with results form async calls (for using them in other methods that are called later on?
Sorry if the title is not clear or correct, dont know what title should i put. Please correct if wrong.
I have this code to download images from IP camera and it can download the images.The problem is how can i do the images downloading process at the same time for all cameras if i have two or more cameras?
private void GetImage()
{
string IP1 = "example.IPcam1.com:81/snapshot.cgi;
string IP2 = "example.IPcam2.com:81/snapshot.cgi;
.
.
.
string IPn = "example.IPcamn.com:81/snapshot.cgi";
for (int i = 0; i < 10; i++)
{
string ImagePath = Server.MapPath("~\\Videos\\liveRecording2\\") + string.Format("{0}", i, i + 1) + ".jpeg";
string sourceURL = ip;
WebRequest req = (WebRequest)WebRequest.Create(sourceURL);
req.Credentials = new NetworkCredential("user", "password");
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
Bitmap bmp = (Bitmap)Bitmap.FromStream(stream);
bmp.Save(ImagePath);
}
}
You should not run long-running code like that from an ASP.NET application. They are meant to simply respond to requests.
You should place this code in a service (Windows Services are easy), and control the service through a WCF service running inside of it.
You're also going to get into trouble because you don't have your WebResponse and Stream in using blocks.
There are several methods that will depend on how you want to report feedback to the user. It all comes down to multi-threading.
Here is one example, using the ThreadPool. Note that this is missing a bunch of error checking throughout... It is here as an example of how to use the ThreadPool, not as a robust application:
private Dictionary<String, String> _cameras = new Dictionary<String, String> {
{ "http://example.IPcam1.com:81/snapshot.cgi", "/some/path/for/image1.jpg" },
{ "http://example.IPcam2.com:81/snapshot.cgi", "/some/other/path/image2.jpg" },
};
public void DoImageDownload() {
int finished = 0;
foreach (KeyValuePair<String, String> pair in _cameras) {
ThreadPool.QueueUserWorkItem(delegate {
BeginDownload(pair.Key, pair.Value);
finished++;
});
}
while (finished < _cameras.Count) {
Thread.Sleep(1000); // sleep 1 second
}
}
private void BeginDownload(String src, String dest) {
WebRequest req = (WebRequest) WebRequest.Create(src);
req.Credentials = new NetworkCredential("username", "password");
WebResponse resp = req.GetResponse();
Stream input = resp.GetResponseStream();
using (Stream output = File.Create(dest)) {
input.CopyTo(output);
}
}
This example simply takes the work you are doing in the for loop and off-loads it to the thread pool for processing. The DoImageDownload method will return very quickly, as it is not doing much actual work.
Depending on your use case, you may need a mechanism to wait for the images to finish downloading from the caller of DoImageDownload. A common approach would be the use of event callbacks at the end of BeginDownload to notify when the download is complete. I have put a simple while loop here that will wait until the images finish... Of course, this needs error checking in case images are missing or the delegate never returns.
Be sure to add your error checking throughout... Hopefully this gives you a place to start.