I got a c# grpc service which can get a request to get a setting in a form of a SINGLE json string which can be very large and I don't want to send it all at once. However, the string can not be processed before it arrives completely.
I know there's an option to use the stream keyword in the protobuf and read the string as it arrives, instead of sending it as a whole.
I have only found examples of receiving several items from a stream (with asyncServerStreamingCall.ResponseStream.MoveNext()), However I don't know how to stream a single string.
My questions are:
Is streaming a correct approach for this situation where you need to stream only a single object (string in this case), and CANT process it before it arrives completely?
How do I stream and receive a string like that (from both server and client sides)?
Streaming in gRPC is, as you mentioned, talking about sending multiple discreet operations on a single open channel. Whether unary or streaming, individual messages in gRPC must be received and decided in their entirety - there isn't a concept of an open byte-stream. Instead, you would need to send multiple messages that each contain some fragment of the payload, presumably just in a bytes field, and send the large payload split over some number of such messages. The receiver would need to combine them, perhaps appending to a file, or similar, if it will be inconveniently large for in-memory storage.
Related
I'm writing a class library to communicate with a PLC by using TCP. The communication is based on sending a data string which is terminated by a CRLF and next waiting for an acknowledge string (also terminated by a CRLF) to confirm the data is received (yes I know this is also included in the TCPIP protocol, but this is another discussion).
Currently I'm facing two major problems:
I'm setting the TcpClient.SendTimeout property, however it looks like when the data is send (by TcpClient.Client.Send), the sender does not wait for the receiver the data to be read. Why?
Because of the sender is not waiting, an acknowledge string and immediately the next data string can be send. So, the receiver is getting two packages. Is there a way to read the buffer only till the first CRLF (acknowledge) and leave the next data string in the buffer for the next TcpClient.Client.Read command.?
Thanks in advance,
Mark
TCP is a streaming protocol. There are no packets that you can program against. The receiver must be able to decode the data no matter in what chunks it arrives. Assume one byte chunks, for example.
Here, it seems the receiver can just read until it finds a CRLF. StreamReader can do that.
the sender does not wait for the receiver the data to be read
TCP is asynchronous. When your Send completes the receiver hasn't necessarily processed the data. This is impossible to ensure at the TCP stack level. The receiving app might have called Receive and gotten the data but it might not have processed it. The TCP stack can't know.
You must design your protocol so that this information is not needed.
I just read one byte till
That can work but it is very CPU intensive and inefficient.
I have a node TCP server working and waiting for data and for every socket I have
socket.on("data", function () {
});
Now, as far as I understand, this will get invoked whenever there's any data received. That means that if I send a large string, it will get segmented into multiple packets and each of those will invoke the event separately. Therefore I could concatenate the data until the "end" event is invoked. According to the Node documentation this happens when the FIN packet is sent.
I have to admit I don't know much about networking but this FIN packet, do I have to send it manually when sending data from my C# app or will thise code
var stream = client.GetStream();
using (var writer = new StreamWriter(stream)) writer.Write(request);
send it automatically when it manages to send the whole request string?
Secondly, how does it work from the other end? How do I send a "batch" of data from Node to my C# client so that it knows that the whole "batch" should be considered one thing, despite it being in multiple packets?
Also, is there an equivalent of the "end" even in .NET? Currently, I'm blocking until the stream's DataAvailable is true but that will trigger on the first packet, right? It won't wait for the whole thing.
I'd appreciate if someone could shed some light on this for me.
The TCP FIN packet will be sent when you call writer.Close() in C#, which will trigger the end event in Node as you said.
Without seeing how your C# reading code looks I can't give specifics, but C# will not fire an event when Node closes the connection. It will no longer be stream.CanRead, and if you had a current stream.Read call blocking, it will throw an exception.
TCP provides a stream of bytes, and nothing more. If you are planning to send several messages back and forth from Node and C#, it is up to you to send your messages in such a way that they can be separated. For instance, you could prefix each message with the length, so that you read one byte, and then read that many bytes after it for the message. If your messages are always text, you could encode it as JSON and separate messages with newlines.
I am trying to bind Sound and Image Sequence Data through ArrayList in order to get it synchronized and serializing it through Binary formatter to be send over Network stream.
The Server end threw an exception:
THE STREAM CAN NOT SUPPORT SEEK OPERATION.
What should I have to do in order to sync Objects to be sent over a single Network stream Instance
TCP is stream based and not message based (as UDP is). That means that there is no telling when a message starts or ends. TCP only guarantees that all bytes are received and in the correct order. It does not guarantee that everything sent with one Send() will be received with one Receive().
Hence you need to specify some kind of message identification mechanism. In this case, a header is the way to go as Jon suggested.
However, you need to understand that the entire header might not be received at once. And that two messages might arrive at once. So you need to parse the received buffer before sending anything to the BinaryFormatter for deserialization.
I would split each object you want to send out into a separate "message" where a message consists of (say) 4 bytes indicating the body length, and then the body itself.
When you want to send a serialized object, you serialize to a byte array, write out the length, then write out the data.
At the server side, you read the length, read that much data into a byte array, then deserialize from that message. The incoming stream is only used to read messages, not objects.
I have a TCP connection , which client is PHP and server is C#
this socket connection transfers a image to the socket server , but
randomly some times the transfer get corrupted [image hash is different]
PHP Client
$file = file_get_contents('img.bmp');
socket_write($socket,$file.$cordinates); it sends //image + sme other data
$recv = socket_read ($socket, 500, PHP_BINARY_READ) // read the server response
This stream always transfer a Bitmap image + some data .
C#
this.DataSocket = this.Listner.Accept();
int filelength = this.DataSocket.Receive(this.buffer, this.buffer.Length, SocketFlags.None)
i investigated that in a fresh-browser [newly opened ] this never failed. but when i using this created service several times frequently in the same browser this intended to fail.
when i check with a different browser or new instance of the browser it never failed in first few attempts.
i thought it was some problem with caching but i disable caching using headers but same problem exists
You can't simply expect to write an entire file to the socket at once, nor can you expect to read the file from the socket in one operation. The socket read and write APIs for just about any network programming API from BSD sockets to WinSock to .NET network classes are all going to transmit or receive data up to the desired byte count.
If you look at the documentation for PHP socket_write for example:
Returns the number of bytes successfully written to the socket or FALSE on failure. The error code can be retrieved with socket_last_error(). This code may be passed to socket_strerror() to get a textual explanation of the error.
Note:
It is perfectly valid for socket_write() to return zero which means no bytes have been written. Be sure to use the == operator to check for FALSE in case of an error.
You will typically want to choose a block size like 4096 or 16384 and loop transmitting or receiving that block size until you get the desired number of bytes transmitted or received. Your code will have to check the return value of the send or receive function you're calling and adjust your file pointer accordingly. If transmit returns 0, that could just mean the send buffer is full (not fatal) so continue sending (might want a Sleep(0) delay). If receive returns 0, this usually means the other side has cleanly closed the connection.
One of the most critical flaws in your simple network code usage is that you're not sending the size of the file before you send the file data, so there's no way for the receiver to know how much to read before sending their response. For a simple operation like this, I'd suggest just sending a binary 32bit integer (4 bytes). This would be part of the schema for your operation. So the receiver would first read 4 bytes and from that know how many more bytes need to be read (one buffer size at a time). The receiver keeps reading until they have that many bytes.
I hope this helps. It would be great if socket code were as simple as the usage you attempted, but unfortunately it isn't. You have to select a buffer size, and then keep reading or writing buffers of that size until you get what you want, and you have to convey to the other side how much data you plan on sending.
That you think caching has anything to do with the problem implies that either there is a lot of functionality outside of the code you've published which is affecting the result or that you are a very long way from understanding the problem.
Without knowing the structure of bmp files, my first concern would be how you separate the file from the additional info sent. A few things you could try...
If '$cordinates' (sic) is a fixed size, then put this at the front of the message, not the back
Log the size sent from PHP and the size received.
base64 encode the binary file before sending it (and decode at the receiving end)
Non of above solutions didn't work for me , but i found out that create a new instance every time after a one request will solve the problem. but i don't think its a reliable way.
i tried the client using ASP.NET but same results. i think its not a problem with client PHP its surely a problem of the socket server
I have a C# server that accepts multiple clients, and multiple messages from each client.
1- In order to start reading from each client i need to pass a buffer (bytes), but the problem is I don't know how much data is the client is going to send. So is there a way to know how much data a client is going to send so that i can start reading for the correct amount of data?
2- Is it oK if i use only 1 byte array to read from all clients? or do i need to create a byte array for reading from each client?
Unless your protocol dictates how much data will be sent, no. Typically you read one buffer's-worth and then potentially read more. It will really depend on the protocol though. If the client can only send one message on each connection, you'll typically keep reading until the next call returns 0 bytes. Otherwise either the messages have delimiters or a length-prefix.
Absolutely not - assuming you're going to be reading from multiple clients concurrently (why else would you use asynchronous communications?) you'd end up with the different clients' data all being written over each other. Create a new byte array for each client. Depending on exactly what you do with the data you may be able to reuse the same byte array for the next read for the same client - and you could reuse the byte array for later clients, if you really wanted... but don't read from multiple clients at the same time into the same buffer.
So is there a way to know how much data a client is going to send so that i can start reading for the correct amount of data?
Any protocol ought to have some mechanism for a client to indicate when it is done sending data, either as a "length" value that is sent before the actual data, or as a special terminating sequence that is sent after the data.
Is it oK if i use only 1 byte array to read from all clients? or do i need to create a byte array for reading from each client?
Depends on how your program works. If you'll have multiple simultaneous clients, obviously you can't have just a single buffer because they'll end up overwriting each other. If it's one client after the other, but only one at a time, there's no problem in having just one buffer.