We Keep Coding

Firestore real-time-updates in c#

I'm trying to subscribe to real-time updates with Cloud Firestore in c# using Google.Cloud.Firestore.V1Beta1. I'm using the following code, which receives updates for a short time, until the stream is closed. Has anyone got FirestoreClient.Listen to work?
// Create client
FirestoreClient firestoreClient = FirestoreClient.Create();
// Initialize streaming call, retrieving the stream object
FirestoreClient.ListenStream duplexStream = firestoreClient.Listen();
// Create task to do something with responses from server
Task responseHandlerTask = Task.Run(async () =>
{
IAsyncEnumerator<ListenResponse> responseStream = duplexStream.ResponseStream;
while (await responseStream.MoveNext())
{
ListenResponse response = responseStream.Current;
Console.WriteLine(response);
}
});
// Send requests to the server
var citiesPath = string.Format("projects/{0}/databases/{1}/documents/cities/CJThcwCipOtIEAm2tEMY", projectId, databaseId);
// Initialize a request
var dt = new DocumentsTarget { };
dt.Documents.Add(citiesPath);
ListenRequest request = new ListenRequest
{
Database = new DatabaseRootName(projectId, databaseId).ToString(),
AddTarget = new Target
{
Documents = dt
}
};
// Stream a request to the server
await duplexStream.WriteAsync(request);
// Await the response handler.
// This will complete once all server responses have been processed.
Console.WriteLine("Awaiting responseHandlerTask");
await responseHandlerTask;
Edit 1:
I've tried setting the expiration explicitly to never expire, but still no luck, I get 5 minutes in then receive a RST_STREAM.
//Setup no expiration for the listen
CallSettings listenSettings = CallSettings.FromCallTiming(CallTiming.FromExpiration(Expiration.None));
// Initialize streaming call, retrieving the stream object
FirestoreClient.ListenStream duplexStream = firestoreClient.Listen(listenSettings);
Edit 2:
It seems like a bit of a kludge, but I found it works to keep track of the last resetToken, catch the exception, then restart the request with the request token. I've updated the code that makes the original request to take an optional resumeToken.
ListenRequest request = new ListenRequest
{
Database = new DatabaseRootName(projectId, databaseId).ToString(),
AddTarget = new Target
{
Documents = dt
}
};
if (resumeToken != null)
{
Console.WriteLine(string.Format("Resuming a listen with token {0}", resumeToken.ToBase64()));
request.AddTarget.ResumeToken = resumeToken;
}
// Stream a request to the server
await duplexStream.WriteAsync(request);
It's not perfect, but I think it's the way Google implemented it in Node.js. It does result in an API call every 5 minutes, so there is some expense to it. Maybe that's the why it works this way?
Thanks

Until Jon finishes the official support, you can use something I put together if you need it right away. https://github.com/cleversolutions/FirebaseDotNetRamblings/blob/master/FirebaseDocumentListener.cs Its an extension method you can drop into your project and use like this:
//Create our database connection
FirestoreDb db = FirestoreDb.Create(projectId);
//Create a query
CollectionReference collection = db.Collection("cities");
Query qref = collection.Where("Capital", QueryOperator.Equal, true);
//Listen to realtime updates
FirebaseDocumentListener listener = qref.AddSnapshotListener();
//Listen to document changes
listener.DocumentChanged += (obj, e) =>
{
var city = e.DocumentSnapshot.Deserialize<City>();
Console.WriteLine(string.Format("City {0} Changed/Added with pop {1}", city.Name, city.Population));
};

SerialDevice: DataReader's DetachBuffer - documentation

The documentation for DataReader's DetachBuffer and DetachStream is very vague. It just says `Detaches a buffer that was previously attached to the reader'.
In short
When should reader.DetachBuffer(); be used?
Background
Reading
An example read method for a SerialDevice could look something like this:
using (var reader = new DataReader(inputStream))
{
var bytesReceived = await reader.LoadAsync(EXPECTED_RESPONSE_LENGTH);
var receivedBuffer = new byte[bytesReceived];
reader.ReadBytes(receivedBuffer);
reader.DetachStream();
return receivedBuffer;
}
This code works and seems to be stable, but since I write and read multiple times a second on an embedded device I want to avoid creating the receivedBuffer buffer each time. I modified my method to be something like the code below.
byte[] _receivedBuffer = new byte[EXPECTED_RESPONSE_LENGTH];
private async Task<byte[]> ReadOnceAsync(IInputStream inputStream)
{
using (var reader = new DataReader(inputStream))
{
reader.InputStreamOptions = InputStreamOptions.Partial;
uint bytesReceived = await reader.LoadAsync(EXPECTED_RESPONSE_LENGTH);
var isExpectedLength = (bytesReceived == EXPECTED_RESPONSE_LENGTH);
if (isExpectedLength)
{
reader.ReadBytes(_receivedBuffer);
}
reader.DetachStream();
return isExpectedLength ? _receivedBuffer: null;
}
}
This code crashes my application, sometimes with Access Violation message, within minutes of starting or within seconds if the connected device stops responding.
After I added reader.DetachBuffer(); the code is stable again, but I still don't know if DetachBuffer should be called always, sometimes or not at all.
Writing
My write method does not call writer.DetachStream() but I don't know if it should or not. The code is:
using (var writer = new DataWriter(outputStream))
{
writer.WriteBytes(toSend);
var bytesWritten = await writer.StoreAsync();
//Should writer.DetachBuffer(); be called?
writer.DetachStream();
return bytesWritten;
}

Why does a bound SUB receive only one message from a connecting PUB?

I'm making examples for my ZeroMQ CLR namespace, however I have a problem with PUB/SUB.
Why do I get only the first message? Sometimes I get no message, if I debug through the client (on PubSub_Client(arg);) I get some messages.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Security.Cryptography;
using ZeroMQ;
namespace ZeroMQ.Test
{
static partial class Program
{
static string PubSub_FrontendAddress = "tcp://127.0.0.1:2772";
public static void Main(string[] args)
{
if (args == null || args.Length < 1)
{
// say here were some arguments...
args = new string[] { "World" };
}
// Setup the ZContext
context = ZContext.Create();
CancellationTokenSource cancellor0 = null;
{
// Create the "Server" cancellor and threads
cancellor0 = new CancellationTokenSource();
var serverThread = new Thread(PubSub_Server);
serverThread.Start(cancellor0.Token);
serverThread.Join(64);
}
{
Thread.Sleep(1000);
Console.WriteLine("Starting...");
// foreach arg we are the Client, asking the Server
foreach (string arg in args)
{
PubSub_Client(arg);
// Thread.Sleep(1000);
}
Console.WriteLine("Ended...");
}
if (cancellor0 != null)
{
// Cancel the Server
cancellor0.Cancel();
}
// we could have done here context.Terminate()
}
static void PubSub_Server(object cancelluS)
{
var cancellus = (CancellationToken)cancelluS;
using (var socket = ZSocket.Create(context, ZSocketType.SUB))
{
socket.Bind(PubSub_FrontendAddress);
socket.SubscribeAll();
/* var poller = ZPollItem.Create(socket, (ZSocket _socket, out ZMessage message, out ZError _error) =>
{
while (null == (message = _socket.ReceiveMessage(/* ZSocketFlags.DontWait, * out _error)))
{
if (_error == ZError.EAGAIN)
{
_error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(_error);
}
return true;
}); /**/
while (!cancellus.IsCancellationRequested)
{
ZError error;
ZMessage request;
/* if (!poller.TryPollIn(out request, out error, TimeSpan.FromMilliseconds(512)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
if (null == (request = socket.ReceiveMessage(ZSocketFlags.DontWait, out error)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
foreach (ZFrame frame in request)
{
string strg = frame.ReadString();
Console.WriteLine("{0} said hello!", strg);
}
}
socket.Unbind(PubSub_FrontendAddress);
}
}
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
using (var crypto = new RNGCryptoServiceProvider())
{
var identity = new byte[8];
crypto.GetBytes(identity);
socket.Identity = identity;
}
socket.Connect(PubSub_FrontendAddress);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
}
}

I'm having trouble with your design which seems just wrong:
A single subscriber and multiple publishers is an odd choice. I trust you have a good reason for it, but you should have said what that is. When sending messages from multiple clients to a single server, it is normal to use DEALER/ROUTER sockets instead. PUB/SUB is intended for a small set of publishers to a large number of subscribers.
A client that connects, sends one message, then immediately disconnects, is another very unusual use case that I hope is just an example:
For one thing, you are open to linger problems whereby the message will get dropped on the disconnect it is isn't sent within the linger timeout. [I don't know what the default linger is for your language binding, so that may or may not be an issue, but you should at least check to ensure that it isn't.]
For another, as you've already found, there are issues around the time it takes to connect to a socket, which may lead to PUB messages getting dropped if they are sent before the socket has properly connected.
If you insist on using PUB/SUB in this manner, you will need an out of band protocol to synchronise the PUB and SUB threads before the pub messages are sent. There are examples of how to do this reliable pub/sub in the zeromq guide. This will involve a second set of sockets in the same threads to send the synchronisation messages; DEALER sockets don't drop messages which is why they are suitable for that purpose...
But, DEALER/ROUTER sockets would appear to be a better choice than PUB/SUB unless there is some design requirement that hasn't been disclosed.

Well... There was a comment by Martin Sustrik: "The problem is that connecting is asynchronous and takes certain amount of time."
Now there is Thread.Sleep(64) - and it works...:
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
socket.Connect(PubSub_FrontendAddress);
Thread.Sleep(64);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
Do you know any better way to get the connection established?

C# all pipe instances are busy

The following code creates a new thread acting first as a named pipe client for sending parameters and then as a server for retrieving results. After that it executes a function in another AppDomain acting as a named pipe server and after that as a client to send the results back.
public OrderPrice DoAction()
{
Task<OrderPrice> t = Task<OrderPrice>.Factory.StartNew(NamedPipeClient, parameters);
if (domain == null)
{
domain = AppDomain.CreateDomain(DOMAINNAME);
}
domain.DoCallBack(AppDomainCallback);
return t.Result;
}
static OrderPrice NamedPipeClient(object parameters) {
OrderPrice price = null;
using (NamedPipeClientStream stream = new NamedPipeClientStream(PIPE_TO)) {
stream.Connect();
SerializeToStream(stream, parameters);
}
using (NamedPipeServerStream stream = new NamedPipeServerStream(PIPE_BACK)) {
stream.WaitForConnection();
price = (OrderPrice)DeserializeFromStream(stream);
}
return price;
}
void AppDomainCallback() {
OrderPrice price = null;
using (NamedPipeServerStream stream = new NamedPipeServerStream(PIPE_TO)) {
stream.WaitForConnection();
List<object> parameters = (List<object>)DeserializeFromStream(stream);
if (mi != null)
price = (OrderPrice)mi.Invoke(action, parameters.ToArray());
}
using (NamedPipeClientStream stream = new NamedPipeClientStream(PIPE_BACK)) {
stream.Connect();
SerializeToStream(stream, price);
}
}
The code is called once per second on average and it worked fine for 7+ hours. But at some point "system.io.ioexception all pipe instances are busy" is thrown and they wont reconnect anymore after that. Browsing here it seems like it could be because of not properly disposing the pipe objects, but I guess thats all good since they are inside using statements.
Does anyone have any clue what could be wrong here? The code is in .NET 4.0 running on windows server 2008.

Sounds like it should be a mutex instead of a simple lock
Lock, mutex, semaphore... what's the difference?
as far as the occasional halting, it could be starvation or a deadlock.
This is good reading material for abstracts on what may be happening
http://en.wikipedia.org/wiki/Dining_philosophers_problem

Named pipes app freezing after sending a request to NamedPipeServerStream

I've written some code that uses named pipes to send a string from one application to another. It works fine once, but when the client tries to send the to the server application for a second time it freezes when it tries to connect to the client.
The server code is this:
static void StartServer()
{
Task.Factory.StartNew(() =>
{
var server = new NamedPipeServerStream("MyPipe");
server.WaitForConnection();
StreamReader reader = new StreamReader(server);
while (true)
{
var line = reader.ReadLine();
if (line != null)
{
System.Windows.Forms.MessageBox.Show("Data: : " + line);
}
}
});
}
The client code is:
private void Test()
{
using (var client = new NamedPipeClientStream("MyPipe"))
{
client.Connect();
StreamWriter writer = new StreamWriter(client);
writer.WriteLine("INCOMING:1234567");
writer.Flush();
client.Dispose();
}
}
Tracking the code through, I can see that loop in the server code is continuously checking for any lines being read in but not finding any. The client is hanging on the client.Connect() call when the Test() method is called for a second time. No exceptions are raised.
Can anyone see where I'm going wrong?

Your server stays connected to the first pipe instance used to send the first string.
However, your client is disposing its pipe instance after each test, so the second test creates and tries to connect on a new pipe instance, but the server is no longer listening, so the Connect call blocks waiting for a server.
You need to either:
make your server multithreaded, so that it continues listening while servicing instances already connected; or
Refactor your client so that it connects once, then reuses that connected instance.

Following on from what #Chris Dickson answered, I solved my problem with the following code:
Task.Factory.StartNew(() =>
{
var server = new NamedPipeServerStream("MyPipe", PipeDirection.InOut, 1, PipeTransmissionMode.Message, PipeOptions.Asynchronous);
StreamReader reader = new StreamReader(server);
Boolean connectedOrWaiting = false;
while (true)
{
if (!connectedOrWaiting)
{
server.BeginWaitForConnection((a) => { server.EndWaitForConnection(a); }, null);
connectedOrWaiting = true;
}
if (server.IsConnected)
{
var line = reader.ReadLine();
if (line != null)
System.Windows.Forms.MessageBox.Show("Data: : " + line);
server.Disconnect();
connectedOrWaiting = false;
}
}
});