TCP socket communication for image streaming and Texture2D error - c#

I want to ask for some ideas about why this error occurs.
First, let me show you the symptom of my project <- .gif file is uploaded in this link.
I implemented this project with Python TCP server and Unity client.
As you watch this .gif file, Unity displays the streaming depth image from the Python server.
Although Unity seems to show the image stream in real-time, it also shows 'query mark' frequently.
I heard that this query mark is displayed whenever unity Texture2D is failed to receive data, but even I added a flag to check loading image is true or false, it always true.
Furthermore, When you see the console of Unity, it also shows there are some lost-bytes comparing to the python console (black console is for the python).
I already tested this TCP server works well with python client(that code is also uploaded above link). So, I assumed it is Unity's problem, but I don't know in detail and want to know how to fix this.
This is my Python TCP server:
mport pyrealsense2 as d435
import socketserver
import socket
import cv2
import numpy as np
from queue import Queue
from _thread import *
# _Set queue
enclosure_queue = Queue()
# _Configures of depth and color streams
pipeline = d435.pipeline()
config = d435.config()
config.enable_stream(d435.stream.depth, 640, 480, d435.format.z16, 30)
config.enable_stream(d435.stream.color, 640, 480, d435.format.bgr8, 30)
# _ D435 process
def D435(queue):
print("D435 processing", end="\n ")
pipeline.start(config) # _Start streaming
try:
while True:
# _Wait for a coherent pair of frames: depth and color
frames = pipeline.wait_for_frames()
depth_frame, color_frame = (frames.get_depth_frame(), frames.get_color_frame())
if not (depth_frame and color_frame):
print("Missing frame...", end="\n")
continue
# _Convert <pyrealsense2 frame> to <ndarray>
depth_image = np.asanyarray(depth_frame.get_data()) # convert any array to <ndarray>
color_image = np.asanyarray(color_frame.get_data())
# _Apply colormap on depth image
# (image must be converted to 8-bit per pixel first)
depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.05), cv2.COLORMAP_BONE)
#depth_colormap = cv2.bitwise_not(depth_colormap ) # reverse image
#print("Depth map shape = ", depth_colormap.shape)
# _Encoding
target_frame = depth_colormap
#print(target_frame.shape)
encode_param = [int(cv2.IMWRITE_JPEG_QUALITY),95] # 0 ~ 100 quality
#encode_param = [cv2.IMWRITE_PNG_COMPRESSION,0] # 0 ~ 9 Compressiong rate
#encode_param = [int(cv2.IMWRITE_WEBP_QUALITY),95] # 0 ~ 100 quality
result, imgencode = cv2.imencode('.jpg', target_frame, encode_param) # Encode numpy into '.jpg'
data = np.array(imgencode)
stringData = data.tostring() # Convert numpy to string
print("byte Length: ", len(stringData))
queue.put(stringData) # Put the encode in the queue stack
# __ Image show
images = np.hstack((color_image, depth_colormap)) # stack both images horizontally
cv2.namedWindow('RealSense', cv2.WINDOW_AUTOSIZE)
cv2.imshow('RealSense', images)
cv2.waitKey(1)
finally:
cv2.destroyAllWindows()
# _Stop streaming
pipeline.stop()
class MyTCPHandler(socketserver.BaseRequestHandler):
queue = enclosure_queue
stringData = str()
def handle(self):
# 'self.request' is the TCP socket connected to the client
print("A client connected by: ", self.client_address[0], ":", self.client_address[1] )
while True:
try:
# _server <- client
self.data = self.request.recv(1024).strip() # 1024 byte for header
if not self.data:
print("The client disconnected by: ", self.client_address[0], ":", self.client_address[1] )
break
# _Get data from Queue stack
MyTCPHandler.stringData = MyTCPHandler.queue.get()
# _server -> client
#print(str(len(MyTCPHandler.stringData)).ljust(16).encode()) # <str>.ljust(16) and encode <str> to <bytearray>
self.request.sendall(str(len(MyTCPHandler.stringData)).ljust(16).encode())
self.request.sendall(MyTCPHandler.stringData)
#self.request.sendall(len(MyTCPHandler.stringData).to_bytes(1024, byteorder= "big"))
#self.request.sendall(MyTCPHandler.stringData)
except ConnectionResetError as e:
print("The client disconnected by: ", self.client_address[0], ":", self.client_address[1] )
break
if __name__ == "__main__":
# _Webcam process is loaded onto subthread
start_new_thread(D435, (enclosure_queue,))
# _Server on
HOST, PORT = socket.gethostname(), 8080
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
print("****** Server started ****** ", end="\n \n")
try:
server.serve_forever()
except KeyboardInterrupt as e:
print("****** Server closed ****** ", end="\n \n" )
and Unity script:
Client_unity.cs
using System;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Text;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
namespace imageStream_client
{
public class Client_unity : MonoBehaviour
{
IPHostEntry ipHostInfo;
IPAddress ipAddress; // IPv4
TcpClient client_sock;
const int PORT = 8080;
NetworkStream stream;
// Start is called before the first frame update
void Start()
{
Debug.Log("*****Unity frame started *****");
ipHostInfo = Dns.GetHostEntry(Dns.GetHostName());
ipAddress = ipHostInfo.AddressList[1];
client_sock = new TcpClient(ipAddress.ToString(), PORT);
Debug.Log("***** Client Connected to the server *****");
stream = client_sock.GetStream();
}
// Update is called once per frame
void Update()
{
//Debug.Log("**** Buffer streaming ****");
// _Client -> Server
string message = "1";
byte[] buff = Encoding.ASCII.GetBytes(message);
stream.Write(buff, 0, buff.Length); // spend the byte stream into the Stream
// _Client <- Server
byte[] recvBuf = new byte[client_sock.ReceiveBufferSize]; // total receiveBuffer size
int readBytes = stream.Read(recvBuf, 0, recvBuf.Length);
//Debug.Log($"total receiveBuffer length: {recvBuf.Length}");
Debug.Log($"Real-read byte length: {readBytes}");
// _Set display image
byte[] image = new byte[readBytes];
Buffer.BlockCopy(recvBuf, 0, image, 0, readBytes);
Viewer.instance.SetImageToDisplay(image);
//Viewer.instance.SetImageToDisplay();
}
}
}
Viewer.cs
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
namespace imageStream_client
{
public class Viewer : MonoBehaviour
{
public RawImage rawimage;
public AspectRatioFitter fit;
#region Data
public static Viewer instance;
private byte[] imageToDisplay;
#endregion
#region Get Set
public void SetImageToDisplay(byte[] image)
{
DisplayImage(image);
//instance.imageToDisplay = image;
}
#endregion
#region API
internal static void DisplayImage(byte[] image)
{
if (image == null)
return;
if (image.Length > 0)
{
//이미지 크기 생성 이후 streaming 되는 이미지 크기에 맞게 수정 해야함
Texture2D texture = new Texture2D(640, 480);
//byte형식의 데이터를 texture2D 형식으로 읽음
bool load = texture.LoadImage(image);
if (load)
{
//이미지를 화면에 입힘(Canvas 아래 RawImage)
instance.rawimage.texture = texture as Texture;
//이미지 가로 세로 비율
instance.fit.aspectRatio = 640 / 480;
}
}
}
#endregion
// Start is called before the first frame update
void Awake()
{
instance = this;
}
// Update is called once per frame
void Update()
{
//DisplayImage(instance.imageToDisplay);
}
}
}

The problem is that you moved your read function into update, which calles every frame in Unity, so you just cant catch to receive data.
The solution is:
void Start()
{
Debug.Log("*****Unity frame started *****");
ipHostInfo = Dns.GetHostEntry(Dns.GetHostName());
ipAddress = ipHostInfo.AddressList[1];
client_sock = new TcpClient(ipAddress.ToString(), PORT);
Debug.Log("***** Client Connected to the server *****");
stream = client_sock.GetStream();
StartReceiveFunction();
}
void StartReceiveFunction()
{
// Can be Application.IsRunning or whatever bool you need to receive data
while (true)
{
if(stream.CanRead && client_sock.ReceiveBufferSize > 0){
bytes = new byte[client_sock.ReceiveBufferSize];
stream.Read(bytes, 0, client_sock.ReceiveBufferSize);
Viewer.instance.SetImageToDisplay(image);
}
}

Okay, I found the problem and fixed it.
The problem was the wrong implementation of the TCP server.
I changed the python server code like below:
class MyTCPHandler(socketserver.BaseRequestHandler):
queue = enclosure_queue
stringData = str()
def handle(self):
# 'self.request' is the TCP socket connected to the client
print("A client connected by: ", self.client_address[0], ":", self.client_address[1] )
while True:
try:
# _server <- client
self.data = self.request.recv(1024).strip() # 1024 byte for header
#print("Received from client: ", self.data)
if not self.data:
print("The client disconnected by: ", self.client_address[0], ":", self.client_address[1] )
break
# _Get data from Queue stack
MyTCPHandler.stringData = MyTCPHandler.queue.get()
# _server -> client
#print(str(len(MyTCPHandler.stringData)).ljust(16).encode()) # <str>.ljust(16) and encode <str> to <bytearray>
###self.request.sendall(str(len(MyTCPHandler.stringData)).ljust(16).encode()) # <- Make this line ignored when you connect with C# client.
self.request.sendall(MyTCPHandler.stringData)
#self.request.sendall(len(MyTCPHandler.stringData).to_bytes(1024, byteorder= "big"))
#self.request.sendall(MyTCPHandler.stringData)
except ConnectionResetError as e:
print("The client disconnected by: ", self.client_address[0], ":", self.client_address[1] )
break
I had tested this server code with a python TCP client.
When the client connected to the server and spending-receiving data, it should know the size of the buffer to transmitted data by the server.
So, self.request.sendall(str(len(MyTCPHandler.stringData)).ljust(16).encode()) is needed.
However, TCPclient class in C# is sort of different to python one. it doesn't need to receive the explicit buffer size of transmitted data from the server (maybe that was processed automatically inside). So, the above-mentioned python code line can be ignored.
I reuploaded the new version here.

Related

Named Pipes IPC: Python server, C# Client

Title sums it up. There are plenty of examples around with a c# server and python client communicating back and forth.
I'd like to understand how I can instead create a python server and c# client for some interprocess communication.
I managed to find a solution. To begin, I'd first like to clarify some confusing terminology and obscure naming conventions used in dotnet core.
It appears that the NamedPipeServerStream and NamedPipeClientStream don't actually operate on named pipes but instead on unix domain sockets. This means that we must use sockets to communicate between processes rather than FIFO files.
Another frustration I find with dotnet core is that when creating a socket or connecting to one, the NamedPipeServerStream and NamedPipeClientStream classes will add "CoreFxPipe_" to the beginning of the socket name. See related question.
Python Server
#!/usr/bin/python3
import socket
import os
import struct
SOCK_PATH = "/tmp/CoreFxPipe_mySocket"
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock:
try:
os.remove(SOCK_PATH)
except OSError:
pass
sock.bind(SOCK_PATH)
sock.listen()
conn, addr = sock.accept()
with conn:
try:
while True:
amount_expected = struct.unpack('I', conn.recv(4))[0]
print("amount_expected :", amount_expected)
message = conn.recv(amount_expected)
print("Received message : ", message.decode())
# Send data
message_rev = message[::-1].decode()
print("Sent message (reversed) : ", message_rev)
conn.sendall(
struct.pack(
'I',
len(message_rev)
)
+ message_rev.encode('utf-8')
)
except (struct.error, KeyboardInterrupt) as e:
print(e)
finally:
print('closing socket')
C# Client
using System;
using System.IO;
using System.IO.Pipes;
using System.Text;
class PipeClient
{
static void Main(string[] args)
{
using (NamedPipeClientStream pipeClient =
new NamedPipeClientStream(".", "mySocket", PipeDirection.InOut))
{
// Connect to the pipe or wait until the pipe is available.
Console.WriteLine("Attempting to connect to pipe...");
pipeClient.Connect();
try
{
// Read user input and send that to the client process.
using (BinaryWriter _bw = new BinaryWriter(pipeClient))
using (BinaryReader _br = new BinaryReader(pipeClient))
{
while (true)
{
//sw.AutoFlush = true;
Console.Write("Enter text: ");
var str = Console.ReadLine();
var buf = Encoding.ASCII.GetBytes(str); // Get ASCII byte array
_bw.Write((uint)buf.Length); // Write string length
_bw.Write(buf); // Write string
Console.WriteLine("Wrote: \"{0}\"", str);
Console.WriteLine("Let's hear from the server now..");
var len = _br.ReadUInt32();
var temp = new string(_br.ReadChars((int)len));
Console.WriteLine("Received from client: {0}", temp);
}
}
}
// Catch the IOException that is raised if the pipe is broken
// or disconnected.
catch (IOException e)
{
Console.WriteLine("ERROR: {0}", e.Message);
}
}
Console.Write("Press Enter to continue...");
}
}
Sources:
https://abgoswam.wordpress.com/2017/07/13/named-pipes-c-python-net-core/
https://realpython.com/python-sockets/
https://unix.stackexchange.com/questions/75904/are-fifo-pipe-unix-domain-socket-the-same-thing-in-linux-kernel

C# socket hangs when receive in loop - python as socket server

I'm familiar with C#, and know some python. Recent days I'm learning the book Programming Python, 4th Edition and have run the very basic socket samples: echo-server.py and echo-client.py
They work well on my Windows, python 3.x.
python server:
from socket import *
myHost = 'localhost'
myPort = 50007
sockobj = socket(AF_INET, SOCK_STREAM)
sockobj.bind((myHost, myPort))
sockobj.listen(5)
while True:
connection, address = sockobj.accept()
print('Server connected by', address)
while True:
data = connection.recv(1024)
if not data: break
connection.send(b'Echo=>' + data)
connection.close()
Now I want to learn socket in C# too, so I wrote a C# .net framework 4.5 socket client, expecting to receive and show what echo-client.py does.
I got the C# demo from msdn and made some refactor to reduce code size.
public static void Main(string[] args)
{
string server = "localhost";
int port = 50007;
string request = "GET / HTTP/1.1\r\nHost: " + server +
"\r\nConnection: Close\r\n\r\n";
Byte[] sent = Encoding.ASCII.GetBytes(request);
Byte[] recv = new Byte[256];
IPHostEntry hostEntry = Dns.GetHostEntry(server);
IPEndPoint ipe = new IPEndPoint(hostEntry.AddressList[1], port);
Socket s =
new Socket(ipe.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
s.Connect(ipe);
s.Send(sent, sent.Length, 0);
int bytes = 0;
string page = "recived:\r\n";
//do
{
bytes = s.Receive(recv, recv.Length, 0);
page = page + Encoding.ASCII.GetString(recv, 0, bytes);
}
//while (bytes > 0);
Console.WriteLine(page);
Console.WriteLine("result");
Console.ReadKey();
}
My test steps:
If I set up a web site using local IIS, such as
http://localhost:801, then above code can show the homepage html
contents, this means my C# code is working.
Run echo-server.py, and change C# code's port to 50007, then run,
nothing output in console, and application does not exit, if I place a break point within the loop, I can see the loop has only run once. The python server did output some log saying C# is connecting.
Comment do while loop(as commented in code), this time the output is exactly same as echo-client.py(expected).
So I'm wondering what's wrong when I'm using do while loop?

Serial Port Request-Response communication using c#

I want to do serial port communication with a machine which uses RS232-USB ports.
I am using serial port class. I am very new to the concept. In my first Machine interfacing I only had to do the serialport.readLine( to get the readings from the machine and there was no need to send ACK /NAK). but for the new machine interface the document says following things:
The following is an example of the RA500 communication:
Computer :<05h 31h 0dh>
RA500 :1st line of information
Computer :<06h 0dh>
RA500 :2nd line of information
Computer :<06h 0dh>
RA500 :”EOF”<0dh>
What i understood from this is i have to write to comport before reading from it. this is what i am doing in my code:
ACK = "06 0d"; NAK = "15 0d"; str = "05 31 0d";
while (count <= 5)
{
rx = ComPortDataReceived(str);
if (!string.IsNullOrEmpty(rx))
{
str = ACK;
returnReading += rx;
}
else if (string.IsNullOrEmpty(rx)) str = NAK;
count++;
}
private string ComPortDataReceived(string str)
{
string Rx = string.Empty;
string exceptionMessage = string.Empty;
try
{
byte[] bytes = str.Split(' ').Select(s => Convert.ToByte(s, 16)).ToArray();
comPort.Write(bytes, 0, bytes.Length);
Rx = comPort.ReadExisting();
PEHRsLibrary.writeTextFile(DateTime.Now + " RxString :" + Rx);
return Rx;
}
catch(Exception e){}
when i use this code i am receiving empty strings as responce. but if i use comPort.ReadExisting() only without using comPort.Write i am receving a string with all the readings but the probblem is it only gives one line of information and dosnt give 2nd or 3rd line readings.
when i try using comPort.ReadLine() after Write() I am getting Timeout exception.
i dont know what i am doing wrong in this case. I am writing ACk after receving 1st line but not receving 2nd line. Next thing i am gonna try is read() as byte and then convert it to string instead of using ReadExisting(). Any other way i can do this?

C# - Communicate with WinUSB device with LIBUSB

I am trying to communicate with a Nokia Lumia phone(RM-917), over USB using LIBUSING and C#. LIBUSB is able to see the device's information(pid,vid,etc). However, I am not able to successfully write to ANY endpoint, even sending the exact command as the Windows Device Recovery Tool.
According to WinUSB, the write endpoint is EP07, however, this endpoint just times out. I have tried every other endpoint, and all of these fail.
`
public void initDevice()
{
if(this.lumiaDevice == null)
{
throw new Exception("LumiaPhoneManager does not have a selected device");
}
UsbDeviceFinder MyUsbFinder = new UsbDeviceFinder(0x0421, 0x0661);
MyUsbDevice = UsbDevice.OpenUsbDevice(MyUsbFinder);
IUsbDevice wholeUsbDevice = MyUsbDevice as IUsbDevice;
if (!ReferenceEquals(wholeUsbDevice, null))
{
// This is a "whole" USB device. Before it can be used,
// the desired configuration and interface must be selected.
// Select config #1
wholeUsbDevice.SetConfiguration(1);
// Claim interface #0.
wholeUsbDevice.ClaimInterface(1);
}
if (this.writer == null)
{
writer = MyUsbDevice.OpenEndpointWriter(WriteEndpointID.Ep07);
}
}
public void readPCode()
{
currentID++;
var _x = new jsonPkt();
ErrorCode ec = ErrorCode.None;
int bytesWritten;
_x.id = this.currentID + 1;
_x.method = "ReadProductCode";
string value = #"{""jsonrpc"":""<JSONRPC>"",""id"":<ID>,""method"":""<METHOD>"",""params"":null}";
value = value.Replace("<JSONRPC>", "2.0");
value = value.Replace("<ID>", currentID.ToString());
value = value.Replace("<METHOD>", _x.method.ToString());
ec = writer.Write(Encoding.Default.GetBytes(value), 8000, out bytesWritten);
currentID++;
if (ec != ErrorCode.None) throw new Exception(UsbDevice.LastErrorString);
byte[] readBuffer = new byte[1024];
while (ec == ErrorCode.None)
{
int bytesRead;
// If the device hasn't sent data in the last 100 milliseconds,
// a timeout error (ec = IoTimedOut) will occur.
ec = reader.Read(readBuffer, 100, out bytesRead);
// if (bytesRead == 0) throw new Exception("No more bytes!");
// Write that output to the console.
this.rtb.Text += Encoding.Default.GetString(readBuffer, 0, bytesRead).ToString() + "\n";
}
}
Found the solution
Debugged the OEM software and found the program was using a different path to the USB device. After that I was getting access denied errors, which was solved by moving the project to a different drive. For reasons unknown, when the program runs on c drive, the CreateFile function fails with access denied
Its possible that to activate write, you need to send some class specific control request first. You mentioned that windows device recovery tool is able to write.
You can install USB packet sniffer software in your windows PC and then use the device manager to write some data to the device. Packet sniffer tool will be able to capture all the packets sent to the device.
This way you can see the exact device requests which are required to enable write operation.
Analyzer software - http://www.usblyzer.com/
Please try this way to resolve your problem.
PS- I am assuming you do not have a hardware USB packet analyzer like Lecroy advisor or Beagle. Software packet sniffer should be fine since the host is a PC.

Send CTCP Messages

I'm currently working on a project for an introductory class on C#. The idea is to make a basic irc client that connects to a single channel on a single irc server. Most things have been easy to implement, however, I've hit a brick wall when attempting to send CTCP messages.
The format is supposed to be: PRIVMSG target : /U+0001ACTION message/U+0001
I am able to identify these messages as they come in, using the Unicode control character, but whenever I try to send my own "ACTION message" is received.
Here's the code I'm using to send the message:
private TcpClient irc; //declarations of properties of irc class
private NetworkStream stream;
private string _inputLine;
StreamReader reader;
StreamWriter writer;
public void Connect() //connects to the irc server in _server on port _port
{
irc = new TcpClient(_server, _port);
stream = irc.GetStream();
reader = new StreamReader(stream);
writer = new StreamWriter(stream);
Send("NICK " + Nick);
Send("USER " + Nick + " 0 * :" + Nick);
Listen();
OnConnect();
}
private void button1_Click(object sender, EventArgs e) //method is from a windows form
{
if (txtMessage.Text.StartsWith("/me"))//sends contents of textbox to SendCTCP if it is an action message
{
test.SendCTCP("ACTION " + txtMessage.Text.Remove(0, 3));
OnAction(test.Nick, txtMessage.Text.Remove(0, 3)); //method OnAction is within the widows form and outputs the message in a textbox for messages sent/received to the channel
}
else
{
test.SendMessage(txtMessage.Text);
ChannelTest(test.Nick, txtMessage.Text, null);
}
txtMessage.Text = "";
}
public void SendCTCP(string message) //formats message with /U+0001 characters
{
char control = '\x01';
Send("PRIVMSG " + _channel + " : " + control.ToString() + message + control.ToString());
}
public void Send(string message) //sends message to server using StreamWriter writer
{
writer.WriteLine(message);
writer.Flush();
}
The Send method uses a StreamWriter to send the message to the server.
I've tried with and without the ToString(). I've tried the character itself within the Send block, rather than in control. I've tried every representation of the character I've been able to find, but I've been unable to make it work. Any help would be appreciated.
Sending the test message "/me tests" through the client results, when the server sends it to other clients in the same channel, in the message: IRCTest!IRCTest#mask PRIVMSG #Channel : ACTION tests
Which is interpreted by every client I can find as: : ACTION tests
edit: I've updated the code with, I hope, all the relevant variables and methods
Well, I feel like an idiot, but at least I figured it out. I've changed it to use '\u0001' as the character, and (wait for it. . .) I deleted the space between the : and the control character. It now functions perfectly.

Categories