Unable to run python script in c# which runs scrapy spider - c#

I followed this_link and I was able to run a dummy python file from my c# code like this...
public JsonResult FetchscrapyDataUrl(String website)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = #"C:\ProgramData\Anaconda3\python.exe";
start.Arguments = #"C:\Users\PycharmProjects\scraping_web\scrape_info\main.py";
//this is path to .py file from scrapy project
start.CreateNoWindow = false; // We don't need new window
start.UseShellExecute = false; // Do not use OS shell
//start.RedirectStandardOutput = true;// Any output, generated by application will be redirected back
start.RedirectStandardError = true; // Any error in standard output will be redirected back (for example exceptions)
Console.WriteLine("Python Starting");
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
string stderr = process.StandardError.ReadToEnd(); // Here are the exceptions from our Python script
string result = reader.ReadToEnd(); // Here is the result of StdOut(for example: print "test")
Console.Write(result);
}
}
}
Now I know that I can run scrapy spider from a single file main.py like this...
from scrapy import cmdline
cmdline.execute("scrapy crawl text".split())
When I run main.py file from cmd in windows it works fine but it does not work when I run it from C# code .Net framework. The error is ...
"Scrapy 1.4.0 - no active project\r\n\r\nUnknown command: crawl\r\n\r\nUse \"scrapy\" to see available commands\r\n"
Any Idea how to run this...Or am i missing some path setting in windows ??
Or should I run my spider from C# in some other way??

You need to set the WorkingDirectory property
start.WorkingDirectory = #"C:\Users\PycharmProjects\scraping_web\scrape_info\"
Or you need to cd to that directory to make it work

Related

How to run python script from C# [duplicate]

This sort of question has been asked before in varying degrees, but I feel it has not been answered in a concise way and so I ask it again.
I want to run a script in Python. Let's say it's this:
if __name__ == '__main__':
with open(sys.argv[1], 'r') as f:
s = f.read()
print s
Which gets a file location, reads it, then prints its contents. Not so complicated.
Okay, so how do I run this in C#?
This is what I have now:
private void run_cmd(string cmd, string args)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = cmd;
start.Arguments = args;
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
Console.Write(result);
}
}
}
When I pass the code.py location as cmd and the filename location as args it doesn't work. I was told I should pass python.exe as the cmd, and then code.py filename as the args.
I have been looking for a while now and can only find people suggesting to use IronPython or such. But there must be a way to call a Python script from C#.
Some clarification:
I need to run it from C#, I need to capture the output, and I can't use IronPython or anything else. Whatever hack you have will be fine.
P.S.: The actual Python code I'm running is much more complex than this, and it returns output which I need in C#, and the C# code will be constantly calling the Python code.
Pretend this is my code:
private void get_vals()
{
for (int i = 0; i < 100; i++)
{
run_cmd("code.py", i);
}
}
The reason it isn't working is because you have UseShellExecute = false.
If you don't use the shell, you will have to supply the complete path to the python executable as FileName, and build the Arguments string to supply both your script and the file you want to read.
Also note, that you can't RedirectStandardOutput unless UseShellExecute = false.
I'm not quite sure how the argument string should be formatted for python, but you will need something like this:
private void run_cmd(string cmd, string args)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = "my/full/path/to/python.exe";
start.Arguments = string.Format("{0} {1}", cmd, args);
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using(Process process = Process.Start(start))
{
using(StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
Console.Write(result);
}
}
}
If you're willing to use IronPython, you can execute scripts directly in C#:
using IronPython.Hosting;
using Microsoft.Scripting.Hosting;
private static void doPython()
{
ScriptEngine engine = Python.CreateEngine();
engine.ExecuteFile(#"test.py");
}
Get IronPython here.
Execute Python script from C
Create a C# project and write the following code.
using System;
using System.Diagnostics;
using System.IO;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApplication1
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
run_cmd();
}
private void run_cmd()
{
string fileName = #"C:\sample_script.py";
Process p = new Process();
p.StartInfo = new ProcessStartInfo(#"C:\Python27\python.exe", fileName)
{
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
};
p.Start();
string output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
Console.WriteLine(output);
Console.ReadLine();
}
}
}
Python sample_script
print "Python C# Test"
You will see the 'Python C# Test' in the console of C#.
I ran into the same problem and Master Morality's answer didn't do it for me. The following, which is based on the previous answer, worked:
private void run_cmd(string cmd, string args)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = cmd;//cmd is full path to python.exe
start.Arguments = args;//args is path to .py file and any cmd line args
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using(Process process = Process.Start(start))
{
using(StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
Console.Write(result);
}
}
}
As an example, cmd would be #C:/Python26/python.exe and args would be C://Python26//test.py 100 if you wanted to execute test.py with cmd line argument 100. Note that the path the .py file does not have the # symbol.
Actually its pretty easy to make integration between Csharp (VS) and Python with IronPython. It's not that much complex... As Chris Dunaway already said in answer section I started to build this inegration for my own project. N its pretty simple.
Just follow these steps N you will get your results.
step 1 : Open VS and create new empty ConsoleApp project.
step 2 : Go to tools --> NuGet Package Manager --> Package Manager Console.
step 3 : After this open this link in your browser and copy the NuGet Command.
Link: https://www.nuget.org/packages/IronPython/2.7.9
step 4 : After opening the above link copy the PM>Install-Package IronPython -Version 2.7.9
command and paste it in NuGet Console in VS.
It will install the supportive packages.
step 5 : This is my code that I have used to run a .py file stored in my Python.exe
directory.
using IronPython.Hosting;//for DLHE
using Microsoft.Scripting.Hosting;//provides scripting abilities comparable to batch files
using System;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Net.Sockets;
class Hi
{
private static void Main(string []args)
{
Process process = new Process(); //to make a process call
ScriptEngine engine = Python.CreateEngine(); //For Engine to initiate the script
engine.ExecuteFile(#"C:\Users\daulmalik\AppData\Local\Programs\Python\Python37\p1.py");//Path of my .py file that I would like to see running in console after running my .cs file from VS.//process.StandardInput.Flush();
process.StandardInput.Close();//to close
process.WaitForExit();//to hold the process i.e. cmd screen as output
}
}
step 6 : save and execute the code
Set WorkingDirectory or specify the full path of the python script in the Argument
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = "C:\\Python27\\python.exe";
//start.WorkingDirectory = #"D:\script";
start.Arguments = string.Format("D:\\script\\test.py -a {0} -b {1} ", "some param", "some other param");
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
Console.Write(result);
}
}
I am having problems with stdin/stout - when payload size exceeds several kilobytes it hangs. I need to call Python functions not only with some short arguments, but with a custom payload that could be big.
A while ago, I wrote a virtual actor library that allows to distribute task on different machines via Redis. To call Python code, I added functionality to listen for messages from Python, process them and return results back to .NET.
Here is a brief description of how it works.
It works on a single machine as well, but requires a Redis instance. Redis adds some reliability guarantees - payload is stored until a worked acknowledges completion. If a worked dies, the payload is returned to a job queue and then is reprocessed by another worker.
had same issure and this worked for me:
using IronPython.Hosting;
var engine = Python.CreateEngine();
engine.ExecuteFile("") //put the directory of the program in the quote marks

Exception "The requested operation requires elevation" when i run my .net code

"I'm trying to execute "main.py" file which is written in Python by my .net core web API but i got an exception.
I already give the permission of my Web API folder as well as my Python Code folder.
var file = Configuration.GetValue<string>("DE.PythonPath");
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = file;
start.Arguments = string.Format("{0} {1}", cmd, args);
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
string stderr = process.StandardError.ReadToEnd();
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
return result;
}
}
I expect the python code run but it gives exception "The requested operation requires elevation"
Visual studio need to be run as admin. So run the Visual studio as Run as Administrator

Activating conda environment from c# code (or what is the differences between manually opening cmd and opening it from c#?)

I want to run a gpu accelerated python script on windows using conda environment (dlwin36).
I’m trying to activate dlwin36 and execute a script:
1) activate dlwin36
2) set KERAS_BACKEND=tensorflow
3) python myscript.py
If I manually open cmd on my machine and write:"activate dlwin36"
it works.
But when I try opening a cmd from c# I get:
“activate is not recognized as an internal or external command, operable program or batch file.”
I tried using the following methods:
Command chaining:
var start = new ProcessStartInfo();
start.FileName = "cmd.exe";
start.Arguments = "/c activate dlwin36&&set KERAS_BACKEND=tensorflow&&python myscript.py";
Process.Start(start).WaitForExit();
(I’ve tested several variations of UseShellExecute, LoadUserProfile and WorkingDirectory)
Redirect standard input:
var commandsList = new List<string>();
commandsList.Add("activate dlwin36");
commandsList.Add("set KERAS_BACKEND=tensorflow");
commandsList.Add("python myscript.py");
var start = new ProcessStartInfo();
start.FileName = "cmd.exe";
start.UseShellExecute = false;
start.RedirectStandardInput = true;
var proc = Process.Start(start);
commandsList.ForEach(command => proc.StandardInput.WriteLine(command));
(I’ve tested several variations of LoadUserProfile and WorkingDirectory)
In both cases, I got the same error.
It seems that there is a difference between manually opening cmd and opening it from c#.
The key is to run activate.bat in your cmd.exe before doing anything else.
// Set working directory and create process
var workingDirectory = Path.GetFullPath("Scripts");
var process = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = "cmd.exe",
RedirectStandardInput = true,
UseShellExecute = false,
RedirectStandardOutput = true,
WorkingDirectory = workingDirectory
}
};
process.Start();
// Pass multiple commands to cmd.exe
using (var sw = process.StandardInput)
{
if (sw.BaseStream.CanWrite)
{
// Vital to activate Anaconda
sw.WriteLine("C:\\PathToAnaconda\\anaconda3\\Scripts\\activate.bat");
// Activate your environment
sw.WriteLine("activate your-environment");
// Any other commands you want to run
sw.WriteLine("set KERAS_BACKEND=tensorflow");
// run your script. You can also pass in arguments
sw.WriteLine("python YourScript.py");
}
}
// read multiple output lines
while (!process.StandardOutput.EndOfStream)
{
var line = process.StandardOutput.ReadLine();
Console.WriteLine(line);
}
You need to use the python.exe from your environment. For example:
Process proc = new Process();
proc.StartInfo.FileName = #"C:\path-to-Anaconda3\envs\tensorflow-gpu\python.exe";
or in your case:
start.Arguments = "/c activate dlwin36&&set KERAS_BACKEND=tensorflow&&\"path-to-Anaconda3\envs\tensorflow-gpu\python.exe\" myscript.py";
I spent a bit of time working on this and here's the only thing that works for me: run a batch file that will activate the conda environment and then issue the commands in python, like so. Let's call this run_script.bat:
call C:\Path-to-Anaconda\Scripts\activate.bat myenv
set KERAS_BACKEND=tensorflow
python YourScript.py
exit
(Note the use of the call keyword before we invoke the activate batch file.)
After that you can run it from C# more or less as shown above.
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = "cmd.exe";
start.Arguments = "/K c:\\path_to_batch\\run_script.bat";
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
start.RedirectStandardError = true;
start.WorkingDirectory = "c:\\path_to_batch";
string stdout, stderr;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
stdout = reader.ReadToEnd();
}
using (StreamReader reader = process.StandardError)
{
stderr = reader.ReadToEnd();
}
process.WaitForExit();
}
I am generating the batch file on the fly in C# to set the necessary parameters.
If this is gonna help anyone in the future. I found that you must run the activation from C:\ drive.

NLTK and stopwords Lookuperror in C# application

I am running a python script from a C# application. The script runs fine on command prompt/terminal but fails to execute when invoked via C# code.
It says Resource u'corpora/stopwords' not found. Please use the NLTK Downloader to obtain the resource: >>> nltk.download() even though I have all the required data/stopwords
Below is the error report from debug tab in Visual Studios.
Traceback (most recent call last):
File "C:\Users\Amey\Anaconda3\envs\dato-env\TrainingSetsUtil.py", line 20, in <module>
stopwords = set(stopwords.words('english'))
File "C:\Users\Amey\Anaconda3\envs\dato-env\lib\site-packages\nltk\corpus\util.py", line 99, in __getattr__
self.__load()
File "C:\Users\Amey\Anaconda3\envs\dato-env\lib\site-packages\nltk\corpus\util.py", line 64, in __load
except LookupError: raise e
LookupError:
**********************************************************************
Resource u'corpora/stopwords' not found. Please use the NLTK
Downloader to obtain the resource: >>> nltk.download()
Searched in:
- 'nltk_data'
**********************************************************************
Here's the invoking code.
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = #"C:\Users\Amey\Anaconda3\envs\dato-env\python.exe";
start.Arguments = #"C:\Users\Amey\Anaconda3\envs\dato-env\TrainingSetsUtil.py " + uname;
start.UseShellExecute = false;// Do not use OS shell
start.CreateNoWindow = true; // We don't need new window
start.RedirectStandardOutput = true;// Any output, generated by application will be redirected back
start.RedirectStandardError = true; // Any error in standard output will be redirected back (for example exceptions)
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
string stderr = process.StandardError.ReadToEnd(); // Here are the exceptions from our Python script
string result = reader.ReadToEnd(); // Here is the result of StdOut(for example: print "test")
Console.WriteLine(result);
Console.WriteLine(stderr);
}
}

Problem launching FFmpeg inside C#

I'm calling FFmpeg via ProcessStartInfo inside my C# application however, I can't keep getting the error;
File for preset 'lossless_slow' not
found
Here's my C# code;
var processinfo = new ProcessStartInfo();
processinfo.FileName = "FFmpeg\\bin\\ffmpeg.exe";
processinfo.Arguments = "-i C:\Temp\input.mp4 -y -acodec aac -strict experimental -ab 96k -vcodec libx264 -vpre lossless_slow -crf 22 -threads 0 C:\Temp\output.mp4"
processinfo.RedirectStandardOutput = true;
processinfo.RedirectStandardError = true;
processinfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Normal;
processinfo.UseShellExecute = false;
processinfo.LoadUserProfile = true;
processinfo.EnvironmentVariables.Add("HOME", #"C:\Users\wonea\.ffmpeg");
var reg = System.Diagnostics.Process.Start(processinfo);
string output = string.Empty;
string error = string.Empty;
using (System.IO.StreamReader myOutput = reg.StandardOutput)
{
output = myOutput.ReadToEnd();
}
using (System.IO.StreamReader myError = reg.StandardError)
{
error = myError.ReadToEnd();
}
Now I've put my presets in the folder
C:\Users\wonea\ .ffmpeg
and included this in the Windows path user variable HOME. This works fine when running FFmpeg from the command line, however fails when the commands are issued inside my C# application, why!? Thanks for any help...!
Also of note, I'm running the service as "Network Service".
In situations like this I always start up the procmon tool which can show you all the file operations of your application. You can set up a filter based on the name of the preset file and see where ffmpeg is trying to locate it.
Setting the HOME variable looks good here.
The only thing I see is that in the code you set HOME to be #"C:\Users\wonea.ffmpeg" and in the text you mention the file is at : C:\Users\wonea.ffmpeg
Is one of these a typo?

Categories