Unity - Running a Neural Network - c#

I trained a model using PyTorch. In Unity, I am using a WebCamTexture to display a live video. How can I feed the webcam frames into the PyTorch model, then perform actions in Unity with the output of the model?
I found Unity ML-agents, but it doesn't seem like it would help with this situation.

You can capture cam data on each update, then run your pytorch model by feeding it the data you just captured. I haven't tried and not sure how pytorch works, but for generic python scripts you can do something like:
...
void Start()
{
...
data = new Color32[webcamTexture.width * webcamTexture.height];
...
}
...
void FixedUpdate ()
{
...
webCamTexture.GetPixels32(data); //this is faster than returning a Color32 object
...
}
...
private void runPython(string pathToPythonExecutable, string pyTorchScript, Color32[] data)
{
var startInfo = new ProcessStartInfo();
var pyTorchArgs = convertDataToYourPyTorchInputFormat (data)
startInfo.Arguments = string.Format("{0} {1}", pyTorchScript, pyTorchArgs);
startInfo.FileName = pathToPythonExecutable;
startInfo.UseShellExecute = false;
var process = Process.Start(start));
process.WaitForExit();
//do stuff in unity with the return value of process (process.ExitCode) or whatever.
}
Mind you, this may create significant overhead to create and end processes using an external executable file. There are some libraries that allow you to run python scripts inside c#. I can think of 2: IronPython (http://ironpython.net) and Python for .Net (http://pythonnet.github.io) I have never tried them though.

Related

is there a way to send data from unity c# and python program?

i have a unity project and i want to send the position of the object alongside some other data from it to a python project that will have a neural network then sending the output back to unity.
the data that i need to send from unity to python is the position of the object (x,y,z) and some bools maybe some strings, the output of the neural network will be (w,a,s,d), what i did is that i wrote the data to a text file and the python read the data from that file but this way will always produce an error because the python will need to read the file while c# is writing which will produce IO error, another way is using clipboard but this will be viable for small data but won't be good in my case.
c# code
void Update()
{
System.IO.StreamWriter saveFile = new System.IO.StreamWriter("Reading/Positions/PlanePos.txt", false);
saveFile.Write(this.transform.position);
saveFile.Close();
System.IO.StreamWriter saveFile2 = new System.IO.StreamWriter("Reading/Positions/done.txt", false);
if (this.transform.position.y> 2)
{
leftground = true;
}
if (leftground)
{
if (this.transform.position.y < 2)
{
saveFile2.Write("done");
saveFile2.Flush();
dones.text = "done";
}
else
{
saveFile2.Write("air");
saveFile2.Flush();
dones.text = "air";
}
}
else
{
saveFile2.Write("ground");
saveFile2.Flush();
dones.text = "ground";
}
saveFile2.Close();
}
Python Code:
def read_data():
# reading from file
file = open("D:/Cs/Grad/Tests/airplane test/Reading/Positions/PlanePos.txt", "r")
planepos = file.readline()
file.close()
file = open("D:/Cs/Grad/Tests/airplane test/Reading/Positions/AirportPosition.txt", "r")
airportpos = file.readline()
file.close()
# ==================================================================
# spliting and getting numbers
#plane_X, plane_Y, plane_Z = map(float, planepos.strip('() \n').split(','))
#airport_X, airport_Y, airport_Z = map(float, airportpos.strip('() \n').split(','))
planepos=planepos.strip('() \n').split(',')
airportpos=airportpos.strip('() \n').split(',')
return planepos[0], planepos[1], planepos[2], airportpos[0], airportpos[1], airportpos[2]
i except a way to send data from unity to python and vice versa like a server or anything that can do this.

what are the best practices to use python in c# application?

I have a requirement for using Python analytics into c# application.
To be precise: c# should call a python script and get the output back into c# application for further processing.
I have tried using IronPython as recommend by many.
ScriptEngine engine = Python.CreateEngine();
ScriptScope scope = engine.CreateScope();
engine.ExecuteFile(#"DemoPythonApplication\SentimentAnalysis.py", scope);
dynamic testFunction = scope.GetVariable("create_sentiment_analysis"); //calling a function from python script file
var result = testFunction(); //This function is returning a dynamic dictionary, which I can use in my c# code further
But the limitation with IronPython is, it is not providing support for many python libraries like pandas, numpy, nltk etc, These libraries are getting used in python scripts. (since we have a different team working on python I don't have control over them for using specific libraries.)
Another option i tried is to run the python process and calling the script
private static readonly string PythonLocation = #"Programs\Python\Python37\python.exe"; //Location of Python.exe
private static readonly string PythonScript = #"DemoPythonApplication\SentimentAnalysis.py"; //Location of Python Script
private static void ProcessInPython(int a, int b)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = PythonLocation;
start.Arguments = string.Format("{0} {1} {2}", PythonScript, a, b);
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
var result = reader.ReadToEnd();
Console.Write(result);
}
}
}
There are limitations of using this approach though, I am only able to get whatever is getting printed on the console as a string, and could not get the output that python functions are returning.
Also if I use the 2nd approach, I don't know how to call a specific function from the python script file.
Can someone help with the best practices of using python in c# for this kind of situation?

Capturing output from powershell script

I have been working on converting a GUI script from another language to C# in VS2017 for a customer. With help from the folks here I am 95% of the way there, but have run into a couple of snags; just not sure I am doing things in the best way. I'm including just the relevant portions of code below, please let me know if I am not providing enough:
The majority of the code is centered on the wpf form, which collects data for low level technicians to batch deploy a number of Virtual Machines into the VMware environment. This number could easily range into the dozens or even a hundred VMs at once. The information for each VM is specified in the form, then collected in a listview. Once the listview is fully populated it is exported to a csv. Up to this point everything works just fine.
I've next been working on actually launching the powershell/powerCLI script (also working) and capturing output. The log file is opened with a specific reader application the customer uses, which updates in real time, and the captured output is fed to the log. It is important for the technicians to see the output from the code line by line so they can react if there is an issue.
I started with something like this as a test:
string sPSScript = "C:\\Users\\Admin\\Desktop\\TestC#.ps1";
string logFile = "C:\\Users\\Admin\\Desktop\\My.log";
string logReader = "C:\\Users\\Admin\\Documents\\CMTrace.exe";
string standard_output;
System.Diagnostics.Process PSScript = new System.Diagnostics.Process();
PSScript.StartInfo.FileName =
Environment.GetFolderPath(Environment.SpecialFolder.SystemX86) +
"\\WindowsPowerShell\\v1.0\\powershell.exe";
PSScript.StartInfo.Arguments = "-command . '" + sPSScript + "' " +
vCenter.Text;
PSScript.StartInfo.RedirectStandardOutput = true;
PSScript.StartInfo.UseShellExecute = false;
PSScript.Start();
System.Diagnostics.Process LogFile = new System.Diagnostics.Process();
LogFile.StartInfo.FileName = logReader;
LogFile.StartInfo.Arguments = logFile;
LogFile.Start(); while ((standard_output =
PSScript.StandardOutput.ReadLine()) != null)
{
if (standard_output != "")
{
using (StreamWriter file = new StreamWriter(logFile, append: true))
{
file.WriteLine(standard_output);
}
}
}
While this writes to the log file in real time as expected, it creates 100 instances of the logReader application. I understand why, since I am declaring a new StreamWriter object through every pass, but am unsure how better to go about this.
I tried creating the file outside the loop, like this:
StreamWriter file = new StreamWriter(logFile, append: true) { };
System.Diagnostics.Process LogFile = new System.Diagnostics.Process();
LogFile.StartInfo.FileName = logReader;
LogFile.StartInfo.Arguments = logFile;
System.Diagnostics.Process PSScript = new System.Diagnostics.Process();
PSScript.StartInfo.FileName = Environment.GetFolderPath(Environment.SpecialFolder.SystemX86) + "\\WindowsPowerShell\\v1.0\\powershell.exe";
PSScript.StartInfo.Arguments = "-command . '" + sPSScript + "' " + vCenter.Text;
PSScript.StartInfo.RedirectStandardOutput = true;
PSScript.StartInfo.UseShellExecute = false;
LogFile.Start();
PSScript.Start();
System.Threading.Thread.Sleep(1500);
while ((standard_output = PSScript.StandardOutput.ReadLine()) != null)
{
if (standard_output != "")
{
file.WriteLine(standard_output);
}
}
It doesn't create multiple instances, but it also does not update the log file in real time as the previous code does. It only updates once the script runs, and then only partially. The script produces ~1000 lines of output, and I consistently see only about 840 written to the log file.
I thought about doing something like this:
FileStream logFS;
logFS = new FileStream(logFile, FileMode.Append);
but it appears the only options available to me to write to the file are expecting a byte array.
I am sure that I am missing something stupid simple in this, but would appreciate any suggestions on the easiest way to create the log file, open it in the reader, and then update it with the standard output from the powershell script.
why did the previous code writes in real time?
because you are wrapping it with using. And at the end of using block its gonna call dispose which calls .Flush to write to disk
Your second code block calls WriteLine but never called Flush so it writes to the disk whenever the buffer is full. Just add a .Flush call after WriteLine and you will have real time logging

How do I execute and return the results of a python script in c#?

How do I execute and return the results of a python script in c#?
I am trying to run a python script from my controller.
I have python.exe setup in a virtual environment folder created with the virtualenv command.
So just for testing purposes at the moment I would like to just return resulting string from my phython script:
# myscript.py
print "test"
And display that in a view in my asp.net mvc app.
I got the run_cmd function from a related stackoverflow question.
I've tried adding the -i option to force interactive mode and calling process.WaitForExit() with no luck.
namespace NpApp.Controllers
{
public class HomeController : Controller
{
public ActionResult Index(string returnUrl)
{
ViewBag.ReturnUrl = returnUrl;
ViewBag.textResult = run_cmd("-i C:/path/to/virtualenv/myscript.py", "Some Input");
return View();
}
private string run_cmd(string cmd, string args)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = #"C:/path/to/virtualenv/Scripts/python.exe";
start.CreateNoWindow = true;
start.Arguments = string.Format("{0} {1}", cmd, args);
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
//Console.Write(result);
process.WaitForExit();
return result;
}
}
}
}
}
It seems like myscript.py never even runs. But I get no errors, just a blank variable in my view.
Edit:
I had tried to simplify the above stuff because I thought it would be easier to explain and get an answer. Eventually I do need to use a package called "nameparser" and store the result of passed name argument into a database. But if I can just get the run_cmd to return a string I think I can take care of the rest of it. This is why I think the rest api and IronPython mentioned in the comments may not work for me here.
Ok, I figured out what the issue was thanks to some leads from the comments. Mainly it was the spaces in the path to the python.exe and the myscript.py. Turns out I didn't need -i or process.WaitForExit(). I just moved the python virtual environment into a path without spaces and everything started working. Also made sure that the myscript.py file was executable.
This was really helpful:
string stderr = process.StandardError.ReadToEnd();
string stdout = process.StandardOutput.ReadToEnd();
Debug.WriteLine("STDERR: " + stderr);
Debug.WriteLine("STDOUT: " + stdout);
That shows the python errors and output in the Output pane in Visual Studio.

C# Problem Reading Console Output to string

i want to launch ffmpeg from my app and retrive all console output that ffmpeg produces. Thing seems obvious, i followed many forum threads/articles like this one but i have problem, though i follow all information included there I seem to end up in dead end.
String that should contain output from ffmpeg is always empty. I've tried to see where is the problem so i made simple c# console application that only lists all execution parameters that are passed to ffmpeg, just to check if problem is caused by ffmpeg itself. In that case everything work as expected.
I also did preview console window of my app. When i launch ffmpeg i see all the output in console but the function that should recieve that output for further processing reports that string was empty. When my param-listing app is launched the only thing I see is the expected report from function that gets output.
So my question is what to do to get ffmpeg output as i intended at first place.
Thanks in advance
MTH
This is a long shot, but have you tried redirecting StandardError too?
Here is a part of my ffmpeg wrapper class, in particular showing how to collect the output and errors from ffmpeg.
I have put the Process in the GetVideoDuration() function just so you can see everything in the one place.
Setup:
My ffmpeg is on the desktop, ffPath is used to point to it.
namespace ChildTools.Tools
{
public class FFMpegWrapper
{
//path to ffmpeg (I HATE!!! MS special folders)
string ffPath = System.Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + "\\ffmpeg.exe";
//outputLines receives each line of output, only if they are not zero length
List<string> outputLines = new List<string>();
//In GetVideoDuration I only want the one line of output and in text form.
//To get the whole output just remove the filter I use (my search for 'Duration') and either return the List<>
//Or joint the strings from List<> (you could have used StringBuilder, but I find a List<> handier.
public string GetVideoDuration(FileInfo fi)
{
outputLines.Clear();
//I only use the information flag in this function
string strCommand = string.Concat(" -i \"", fi.FullName, "\"");
//Point ffPath to my ffmpeg
string ffPath = System.Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + "\\ffmpeg.exe";
Process processFfmpeg = new Process();
processFfmpeg.StartInfo.Arguments = strCommand;
processFfmpeg.StartInfo.FileName = ffPath;
//I have to say that I struggled for a while with the order that I setup the process.
//But this order below I know to work
processFfmpeg.StartInfo.UseShellExecute = false;
processFfmpeg.StartInfo.RedirectStandardOutput = true;
processFfmpeg.StartInfo.RedirectStandardError = true;
processFfmpeg.StartInfo.CreateNoWindow = true;
processFfmpeg.ErrorDataReceived += processFfmpeg_OutData;
processFfmpeg.OutputDataReceived += processFfmpeg_OutData;
processFfmpeg.EnableRaisingEvents = true;
processFfmpeg.Start();
processFfmpeg.BeginOutputReadLine();
processFfmpeg.BeginErrorReadLine();
processFfmpeg.WaitForExit();
//I filter the lines because I only want 'Duration' this time
string oStr = "";
foreach (string str in outputLines)
{
if (str.Contains("Duration"))
{
oStr = str;
}
}
//return a single string with the duration line
return oStr;
}
private void processFfmpeg_OutData(object sender, DataReceivedEventArgs e)
{
//The data we want is in e.Data, you must be careful of null strings
string strMessage = e.Data;
if outputLines != null && strMessage != null && strMessage.Length > 0)
{
outputLines.Add(string.Concat( strMessage,"\n"));
//Try a Console output here to see all of the output. Particularly
//useful when you are examining the packets and working out timeframes
//Console.WriteLine(strMessage);
}
}
}
}

Categories