Python and .Net Integration options - c#

I want to integrate Python with C#. I found two approaches using Interprocess communication and IronPython
Interprocess communication requires Python.exe to be installed on all client machines so not a viable solution.
We started using IronPython, but it only supports 2.7 python version for now. We are using 3.7 version.
Following code we tried using IronPython:
private void BtnJsonPy_Click(object sender, EventArgs e)
{
// 1. Create Engine
var engine = Python.CreateEngine();
//2. Provide script and arguments
var script = #"C:\Users\user\source\path\repos\SamplePy\SamplePy2\SamplePy2.py"; // provide full path
var source = engine.CreateScriptSourceFromFile(script);
// dummy parameters to send Python script
int x = 3;
int y = 4;
var argv = new List<string>();
argv.Add("");
argv.Add(x.ToString());
argv.Add(y.ToString());
engine.GetSysModule().SetVariable("argv", argv);
//3. redirect output
var eIO = engine.Runtime.IO;
var errors = new MemoryStream();
eIO.SetErrorOutput(errors, Encoding.Default);
var results = new MemoryStream();
eIO.SetOutput(results, Encoding.Default);
//4. Execute script
var scope = engine.CreateScope();
var lib = new[]
{
"C:\\Users\\user\\source\\repos\\SamplePy\\CallingWinForms\\Lib",
"C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9\\lib",
"C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9",
"C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.StdLib.2.7.9"
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32\\Lib",
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\python.exe",
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32",
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\DLLs"
};
engine.SetSearchPaths(lib);
engine.ExecuteFile(script, scope);
//source.Execute(scope);
//5. Display output
string str(byte[] x1) => Encoding.Default.GetString(x1);
Console.WriteLine("Errrors");
Console.WriteLine(str(errors.ToArray()));
Console.WriteLine();
Console.WriteLine("Results");
Console.WriteLine(str(results.ToArray()));
lblAns.Text = str(results.ToArray());
}
The problem sometimes is, to do heavy Machine Learning programming we need to add "Modules". These Modules would be dependent on other modules. This increases point 4, Execute Scripts part of code, as more data path of that module has to be given here var lib = new[] and also some modules are not supported with Iron Python as well (for e.g. modules concerning OCR operations etc.)
Due to these limitations I found Pythonnet which also helps in integrating .net applications with Python. But I am new to it, so want some ideas on implementing the same, and code samples available, and is it feasible or recommended to use with Python 3.7
I checked that setting up Pythonnet is cumbersome initially, so want help or steps on how to set up the same. Also would like to know if in future Iron Python would support Python 3.X as well or not.

I am not familiar with IronPython, but I use pythonnet quite a lot for the same purpose - integrate Python with C#, so I can elaborate on that.
The advantage of using pythonnet for your purposes is having all the CPython packages available for you to use (numpy, scipy, pandas, Theano, Keras, scikit-learn etc), but avoiding the overhead of calling python.exe as separate process (pythonnet works by loading pythonXY.dll into your process).
Pay attention that pythonnet also requires to have stand-alone Python availiable, but you can use Embeddable Python package which is very light-weight and can be distributed with your application.
pythonnnet supports Python 3.7, but the published NuGet packages are only for Python 3.5. You have several choices to obtain pythonnet for Python 3.7:
Download pythonnet wheel package from PyPi and extract Python.Runtime.dll from it
Download NuGet package from pythonnet appveyor build artifacts, as advised on pythonnet installation wiki
Build from sources
Important note: pythonnet version has to match your Python version and bitness. For example, if you are using Python 3.7 32-bit, download pythonnet-2.4.0-cp37-cp37m-win32.whl. If your Python is 64-bit, download pythonnet-2.4.0-cp37-cp37m-win_amd64.whl. Your C# project platform target also has to match (x86 for 32-bit or x64 for 64-bit).
Code sample with similar functionality to what you have posted, using pythonnet (tested with Python 3.7.4 on Windows 7 and pythonnet NuGet from latest build artifacts):
private void Test()
{
// Setup all paths before initializing Python engine
string pathToPython = #"C:\Users\user\AppData\Local\Programs\Python\Python37-32";
string path = pathToPython + ";" +
Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", pathToPython, EnvironmentVariableTarget.Process);
var lib = new[]
{
#"C:\Users\user\source\path\repos\SamplePy\SamplePy2",
Path.Combine(pathToPython, "Lib"),
Path.Combine(pathToPython, "DLLs")
};
string paths = string.Join(";", lib);
Environment.SetEnvironmentVariable("PYTHONPATH", paths, EnvironmentVariableTarget.Process);
using (Py.GIL()) //Initialize the Python engine and acquire the interpreter lock
{
try
{
// import your script into the process
dynamic sampleModule = Py.Import("SamplePy");
// It is more maintainable to communicate with the script with
// function parameters and return values, than using argv
// and input/output streams.
int x = 3;
int y = 4;
dynamic results = sampleModule.sample_func(x, y);
Console.WriteLine("Results: " + results);
}
catch (PythonException error)
{
// Communicate errors with exceptions from within python script -
// this works very nice with pythonnet.
Console.WriteLine("Error occured: ", error.Message);
}
}
}
SamplePy.py:
def sample_func(x, y):
return x*y

Related

Embedding build information in local and CI builds like in Gradle

my question does not target a problem. It is more some kind of "Do you know something that...?". All my applications are built and deployed using CI/CD with Azure DevOps. I like to have all build information handy in the create binary and to read them during runtime. Those applications are mainly .NET Core 2 applications written in C#. I am using the default build system MSBuild supplied with the .NET Core SDK. The project should be buildable on Windows AND Linux.
Information I need:
GitCommitHash: string
GitCommitMessage: string
GitBranch: string
CiBuildNumber: string (only when built via CI not locally)
IsCiBuild: bool (Detecting should work by checking for env variables
which are only available in CI builds)
Current approach:
In each project in the solution there is a class BuildConfig à la
public static class BuildConfig
{
public const string BuildNumber = "#{Build.BuildNumber}#"; // Das sind die Namen der Variablen innerhalb der CI
// and the remaining information...
}
Here tokens are used, which get replaced with the corresponding values during the CI build. To achieve this an addon task is used. Sadly this only fills the values for CI builds and not for the local ones. When running locally and requesting the build information it only contains the tokens as they are not replaced during the local build.
It would be cool to either have the BuildConfig.cs generated during the build or have the values of the variables set during the local build (IntelliSense would be very cool and would prevent some "BuildConfig class could not be found" errors). The values could be set by an MSBuild task (?). That would be one (or two) possibilities to solve this. Do you have ideas/experience regarding this? I did not found that much during my internet research. I only stumbled over this question which did not really help me as I have zero experience with MSBuild tasks/customization.
Then I decided to have a look at build systems in general. Namly Fake and Cake. Cake has a Git-Addin, but I did not find anything regarding code generation/manipulation. Do you know some resources on that?
So here's the thing...
Short time ago I had to work with Android apps namly Java and the build system gradle. So I wanted to inject the build information there too during the CI build. After a short time I found a (imo) better and more elegant solution to do this. And this was modifying the build script in the following way (Scripting language used is Groovy which is based on Java):
def getGitHash = { ->
def stdout = new ByteArrayOutputStream()
exec {
commandLine 'git', 'rev-parse', '--short', 'HEAD'
standardOutput = stdout
}
return stdout.toString().trim().replace("\"", "\\\"")
}
def getGitBranch = { ->
def fromEnv = System.getenv("BUILD_SOURCEBRANCH")
if (fromEnv) {
return fromEnv.substring("refs/heads/".length()).replace("\"", "\\\"");
} else {
def stdout = new ByteArrayOutputStream()
exec {
commandLine 'git', 'rev-parse', '--abbrev-ref', 'HEAD'
standardOutput = stdout
}
return stdout.toString().trim().replace("\"", "\\\"")
}
}
def getIsCI = { ->
return System.getenv("BUILD_BUILDNUMBER") != null;
}
# And the other functions working very similar
android {
# ...
buildConfigField "String", "GitHash", "\"${getGitHash()}\""
buildConfigField "String", "GitBranch", "\"${getGitBranch()}\""
buildConfigField "String", "BuildNumber", "\"${getBuildNumber()}\""
buildConfigField "String", "GitMessage", "\"${getGitCommitMessage()}\""
buildConfigField "boolean", "IsCIBuild", "${getIsCI()}"
# ...
}
The result after the first build is the following java code:
public final class BuildConfig {
// Some other fields generated by default
// Fields from default config.
public static final String BuildNumber = "Local Build";
public static final String GitBranch = "develop";
public static final String GitHash = "6c87e82";
public static final String GitMessage = "Merge branch 'hotfix/login-failed' into 'develop'";
public static final boolean IsCIBuild = false;
}
Getting the required information is done by the build script itself without depending on the CI engine to fulfill this task. This class can be used after the first build its generated and stored in a "hidden" directory which is included in code analysis but exluded from your code in the IDE and also not pushed to the Git. But there is IntelliSense support. In C# project this would be the obj/ folder I guess. It is very easy to access the information as they are a constant and static values (so no reflection or similar required).
So here the summarized question: "Do you know something to achieve this behaviour/mechanism in a .NET environment?"
Happy to discuss some ideas/approaches... :)
It becomes much easier if at runtime you are willing to use reflection to read assembly attribute values. For example:
using System.Reflection;
var assembly = Assembly.GetExecutingAssembly();
var descriptionAttribute = (AssemblyDescriptionAttribute)assembly
.GetCustomAttributes(typeof(AssemblyDescriptionAttribute), false).FirstOrDefault();
var description = descriptionAttribute?.Description;
For most purposes the performance impact of this approach can be satisfactorily addressed by caching the values so they only need to read once.
One way to embed the desired values into assembly attributes is to use the MSBuild WriteCodeFragment task to create a class file that sets assembly attributes to the values of project and/or environment variables. You would need to ensure that you do this in a Target that executes before before compilation occurs (e.g. <Target BeforeTargets="CoreCompile" ...). You would also need to set the property <GenerateAssemblyInfo>false</GenerateAssemblyInfo> to avoid conflicting with the functionality referenced in the next option.
Alternatively, you may be able to leverage the plumbing in the dotnet SDK for including metadata in assemblies. It embeds the values of many of the same project variables documented for the NuGet Pack target. As implied above, this would require the GenerateAssemblyInfo property to be set to true.
Finally, consider whether GitVersion would meet your needs.
Good luck!

Boost.Interprocess v1.66 - get_bootstamp segfault with C#

I have problem with Boost.Interprocess (v1.66) library which I use in my C/C++ library which I use in C# through Marshalling (calling C native code from C#).
I found the problem if I was using Boost.Interprocess named_semaphore for sync between processes. (in open_or_create mode)
If I use my C/C++ lib with another native C/C++ code everything works fine (under newest Windows 10, Linux (4+ kernel) and even Mac OS X (>=10.11)).
The problem occurred under Windows - with C# I have C wrapper around C++ code. If I use Marshalling with simple own-build EXE --> Everything works! But If I use The same C# code (with the same C lib) in the third party application as a DLL plugin I got segfault from get_bootstamp in named_semaphore.
So I have third-party C# SW for which I create plugins (C# DLL). In that plugin I use my C library through marshalling. Marshalling work fine in test C# project (which just call C functions from C lib) but same code segfault in third-party SW.
C Library workflow:
Init all necessary C structures
Start desired TCP server (native C/C++ app) using Boost.Process
Wait for server (through named_semaphore) <-- segfault
Connect to the server...
C# code has same workflow.
Found the problem
The problem occured in boost::interprocess::ipcdetail::get_booststamp (which is called in named_semaphore). here:
struct windows_bootstamp
{
windows_bootstamp()
{
//Throw if bootstamp not available
if(!winapi::get_last_bootup_time(stamp)){
error_info err = system_error_code();
throw interprocess_exception(err);
}
}
//Use std::string. Even if this will be constructed in shared memory, all
//modules/dlls are from this process so internal raw pointers to heap are always valid
std::string stamp;
};
inline void get_bootstamp(std::string &s, bool add = false)
{
const windows_bootstamp &bootstamp = windows_intermodule_singleton<windows_bootstamp>::get();
if(add){
s += bootstamp.stamp;
}
else{
s = bootstamp.stamp;
}
}
If I debug to the line
const windows_bootstamp &bootstamp = windows_intermodule_singleton<windows_bootstamp>::get()
booststamp.stamp is not readable. The size is set to 31, capacity is set to some weird value (like 19452345) and the data is not readable. If i step over to
s += bootstamp.stamp;
the segfault occured!
Found the reason
I debug once more and set debug point to the windows_bootstamp constructor entry and I got no hit so the stamp is not initialized (I guess).
Confirmation
If I change get_bootstamp to
inline void get_bootstamp(std::string &s, bool add = false)
{
const windows_bootstamp &bootstamp = windows_intermodule_singleton<windows_bootstamp>::get();
std::string stamp;
winapi::get_last_bootup_time(stamp);
if(add){
s += stamp;
}
else{
s = stamp;
}
}
Recompile my lib and exe - everything works fine (without any problem).
My question is - what I am doing wrong? I read Boost.Interprocess doc really thoroughly but there are no advice/warnings about my problem (yeah there is "COM Initialization" in Interprocess doc but it not seems helpfull).
Or it's just a bug in Boost.interprocess and I may report it to Boost bug tracker?
Notice - if I start server manually (before I run C# code) It works without segfaults

Compile Brotli into a DLL .NET can reference

So I'd like to take advantage of Brotli but I am not familiar with Python and C++..
I know someone had compiled it into a Windows .exe. But how do I wrap it into a DLL or something that a .NET app can reference? I know there's IronPython, do I just bring in all the source files into an IronPython project and write a .NET adapter that calls into the Brotli API and exposes them? But actually, I'm not even sure if the Brotli API is Python or C++..
Looking at tools/bro.cc, it looks like the "entry" methods are defined in encode.c and decode.c as BrotliCompress(), BrotliDecompressBuffer(), BrotliDecompressStream() methods. So I suppose a DLL can be compiled from the C++ classes.
To avoid the need for Python, I have forked the original brotli source here https://github.com/smourier/brotli and created a Windows DLL version of it that you can use with .NET.
I've added a directory that contains a "WinBrotli" Visual Studio 2015 solution with two projects:
WinBrotli: a Windows DLL (x86 and x64) that contains original unchanged C/C++ brotli code.
Brotli: a Windows Console Application (Any Cpu) written in C# that contains P/Invoke interop code for WinBrotli.
To reuse the Winbrotli DLL, just copy WinBrotli.x64.dll and WinBrotli.x86.dll (you can find already built release versions in the WinBrotli/binaries folder) aside your .NET application, and incorporate the BrotliCompression.cs file in your C# project (or port it to VB or another language if C# is not your favorite language). The interop code will automatically pick the right DLL that correspond to the current process' bitness (X86 or X64).
Once you've done that, using it is fairly simple (input and output can be file paths or standard .NET Streams):
// compress
BrotliCompression.Compress(input, output);
// decompress
BrotliCompression.Decompress(input, output);
To create WinBrotli, here's what I've done (for others that would want to use other Visual Studio versions)
Created a standard DLL project, removed the precompiled header
Included all encoder and decoder original brotli C/C++ files (never changed anything in there, so we can update the original files when needed)
Configured the project to remove dependencies on MSVCRT (so we don't need to deploy other DLL)
Disabled the 4146 warning (otherwise we just can't compile)
Added a very standard dllmain.cpp file that does nothing special
Added a WinBrotli.cpp file that exposes brotli compression and decompression code to the outside Windows world (with a very thin adaptation layer, so it's easier to interop in .NET)
Added a WinBrotli.def file that exports 4 functions
I'll show one way to do that via calling python native library from .NET code. What you need:
You need to intall python 2.7 (hope that is obvious)
You need to compile brotli from source. Hopefully that is easy. First install Microsoft Visual C++ compiler for Python 2.7. Then clone brotli repository via git clone https://github.com/google/brotli.git and compile using python setup.py build_ext. When it's done, in build\lib.win32-2.7 directory you will find brotli.pyd file. This is python c++ module - we will need it later.
You need to either download pythonnet binaries or compile it from source. The reason we use pythonnet here, and not for example Iron Python is because Iron Python does not support native (C\C++) python modules, and that is what we need here. So, to compile from source, clone via git clone https://github.com/pythonnet/pythonnet.git then compile via python setup.py build. In result you will get Python.Runtime.dll (in build\lib.win32-2.7 directory), which is what we need.
When you have all that in place, create console project, reference Python.Runtime.dll and then:
public static void Main()
{
PythonEngine.Initialize();
var gs = PythonEngine.AcquireLock();
try {
// import brotli module
dynamic brotli = PythonEngine.ImportModule(#"brotli");
// this is a string we will compress
string original = "XXXXXXXXXXYYYYYYYYYY";
// compress and interpret as byte array. This array you can save to file for example
var compressed = (byte[]) brotli.compress(original);
// little trick to pass byte array as python string
dynamic base64Encoded = new PyString(Convert.ToBase64String(compressed));
// decompress and interpret as string
var decompressed = (string) brotli.decompress(base64Encoded.decode("base64"));
// works
Debug.Assert(decompressed == original);
}
finally {
PythonEngine.ReleaseLock(gs);
PythonEngine.Shutdown();
}
Console.ReadKey();
}
Then build that and put brotli.pyc you get above in the same directory with your .exe file. After all that manipulations you will be able to compress and decompress from .NET code, as you see above.
You may use Brotli.NET which provides full stream support.
github: https://github.com/XieJJ99/brotli.net/.
Nuget: https://www.nuget.org/packages/Brotli.NET/.
To compress a stream to brotli data:
public Byte[] Encode(Byte[] input)
{
Byte[] output = null;
using (System.IO.MemoryStream msInput = new System.IO.MemoryStream(input))
using (System.IO.MemoryStream msOutput = new System.IO.MemoryStream())
using (BrotliStream bs = new BrotliStream(msOutput, System.IO.Compression.CompressionMode.Compress))
{
bs.SetQuality(11);
bs.SetWindow(22);
msInput.CopyTo(bs);
bs.Close();
output = msOutput.ToArray();
return output;
}
}
To decompress a brotli stream:
public Byte[] Decode(Byte[] input)
{
using (System.IO.MemoryStream msInput = new System.IO.MemoryStream(input))
using (BrotliStream bs = new BrotliStream(msInput, System.IO.Compression.CompressionMode.Decompress))
using (System.IO.MemoryStream msOutput = new System.IO.MemoryStream())
{
bs.CopyTo(msOutput);
msOutput.Seek(0, System.IO.SeekOrigin.Begin);
output = msOutput.ToArray();
return output;
}
}
To support dynamic compress in web applications,add the code like this in the Global.asax.cs:
protected void Application_PostAcquireRequestState(object sender, EventArgs e)
{
var app = Context.ApplicationInstance;
String acceptEncodings = app.Request.Headers.Get("Accept-Encoding");
if (!String.IsNullOrEmpty(acceptEncodings))
{
System.IO.Stream baseStream = app.Response.Filter;
acceptEncodings = acceptEncodings.ToLower();
if (acceptEncodings.Contains("br") || acceptEncodings.Contains("brotli"))
{
app.Response.Filter = new Brotli.BrotliStream(baseStream, System.IO.Compression.CompressionMode.Compress);
app.Response.AppendHeader("Content-Encoding", "br");
}
else
if (acceptEncodings.Contains("deflate"))
{
app.Response.Filter = new System.IO.Compression.DeflateStream(baseStream, System.IO.Compression.CompressionMode.Compress);
app.Response.AppendHeader("Content-Encoding", "deflate");
}
else if (acceptEncodings.Contains("gzip"))
{
app.Response.Filter = new System.IO.Compression.GZipStream(baseStream, System.IO.Compression.CompressionMode.Compress);
app.Response.AppendHeader("Content-Encoding", "gzip");
}
}
}

How to make CUDA dll that can be used in C# application?

It would be good if you could give me a brief tutorial instead of a few words.
My CUDA application is working as I wanted. Now, the problem is how to export CUDA code to C# as I would like to make front end and everything else in C#.
From this link:
http://www.codeproject.com/Articles/9826/How-to-create-a-DLL-library-in-C-and-then-use-it-w
I know how to make a library in C language that can be imported into C# application as Win32 dll.
But my question is, how to make CUDA application dll (or some other extension) that can be shipped to C# and used from C# application?
It would be good if there is somewhere tutorial for CUDA like the one for C library to C# app(above link).
I am using Win7 64 bit, Visual Studio 2010 Ultimate, Cuda Toolikt 5.0 and NSight 2.2.012313
ManagedCUDA is perfect for this type of thing. First you need to follow the instructions in the documentation to set up your Visual Studio Project.
Here is an example of a solution:
test.cu (compiles to test.ptx)
#if !defined(__CUDACC__)
#define __CUDACC__
#include <host_config.h>
#include <device_launch_parameters.h>
#include <device_functions.h>
#include <math_functions.h>
#endif
extern "C"
{
__global__ void test(float * data)
{
float a = data[0];
float b = data[1];
float c = data[2];
data[0] = max(a, max(b, c));
}
}
and here is the C# code:
private static void Test()
{
using (CudaContext ctx = new CudaContext())
{
CudaDeviceVariable<float> d = new CudaDeviceVariable<float>(3);
CUmodule module = ctx.LoadModulePTX("test.ptx");
CudaKernel kernel = new CudaKernel("test", module, ctx)
{
GridDimensions = new dim3(1, 1),
BlockDimensions = new dim3(1, 1)
};
kernel.Run(d.DevicePointer);
}
}
This is just a proof of concept, the device memory is not even initialized and the result is not read but is enough to illustrate how to do it.
You have several options how to distribute your application. In this case i opted for compiling the .cu file into PTX and load it inside the C# project from filesystem.
You could also embed the PTX as a resource directly into your C# application.
You could also compile into a cubin and load or embed that instead of PTX.

G-WAN + Phalanger

OK i have this crazy idea, since php does not play nice with G-WAN, maybe the solution is to use phalanger to compile php code to c# mono assembly and then use it from g-wan?
Anyone has any experience with this combination and could help?
OR maybe i'm wrong and G-wan can run php?
Did someone tried PH7 ?
PH7 is a PHP engine which allow the host application to compile and execute PHP scripts in-process.
As an embedded interpreter, it allows multiple interpreter states to coexist in the same program, without any interference between them.
PH7 is threadsafe.
But in order to be thread-safe, PH7 must be compiled with the PH7_ENABLE_THREADS compile time directive defined.
Well, I did contact the people behind Phalanger (and a few other solutions) to add support for PHP. And their reply (at the time) was that Phalanger was no longer developed.
Now it has been re-emplemented as a CLR language this might give PHP a second life. While I have used the G-WAN 3.9 beta I did not yet try to play with the various languages supported by the Mono runtime.
Regarding the genuine PHP library, I wrote the code below to make it run:
// ----------------------------------------------------------------------------
// php.c: G-WAN using PHP scripts
//
// To build PHP5:
//
// CFLAGS="-O3" ./configure --enable-embed --enable-maintainer-zts --with-tsrm-pthreads --without-pear
// make clean
// make
// sudo make install
/* Installing PHP SAPI module: embed
Installing PHP CLI binary: /usr/local/bin/
Installing PHP CLI man page: /usr/local/php/man/man1/
Installing PHP CGI binary: /usr/local/bin/
Installing build environment: /usr/local/lib/php/build/
Installing header files: /usr/local/include/php/
Installing helper programs: /usr/local/bin/
program: phpize
program: php-config
Installing man pages: /usr/local/php/man/man1/
page: phpize.1
page: php-config.1
Installing PEAR environment: /usr/local/lib/php/
[PEAR] Archive_Tar - already installed: 1.3.7
[PEAR] Console_Getopt - already installed: 1.3.0
[PEAR] Structures_Graph- already installed: 1.0.4
[PEAR] XML_Util - already installed: 1.2.1
[PEAR] PEAR - already installed: 1.9.4
Wrote PEAR system config file at: /usr/local/etc/pear.conf
You may want to add: /usr/local/lib/php to your php.ini include_path
/home/pierre/Downloads/PHP/php5.4-20/build/shtool install -c ext/phar/phar.phar /usr/local/bin
ln -s -f /usr/local/bin/phar.phar /usr/local/bin/phar
Installing PDO headers: /usr/local/include/php/ext/pdo/ */
/*
enabling the 'thread safety' --enable-maintainer-zts option results in:
error: 'tsrm_ls' undeclared (first use in this function)
*/
/*
tsrm_ls
TSRM local storage - This is the actual variable name being passed around
inside the TSRMLS_* macros when ZTS is enabled. It acts as a pointer to
the start of that thread's independent data storage block.
TSRM
Thread Safe Resource Manager - This is an oft overlooked, and seldom if
ever discussed layer hiding in the /TSRM directory of the PHP source code.
By default, the TSRM layer is only enabled when compiling a SAPI which
requires it (e.g. apache2-worker). All Win32 builds have this layer
enabled enabled regardless of SAPI choice.
ZTS
Zend Thread Ssafety - Often used synonymously with the term TSRM.
Specifically, ZTS is the term used by ./configure
( --enable-experimental-zts for PHP4, --enable-maintainer-zts for PHP5),
and the name of the #define'd preprocessor token used inside the engine
to determine if the TSRM layer is being used.
TSRMLS_??
A quartet of macros designed to make the differences between ZTS and
non-ZTS mode as painless as possible. When ZTS is not enabled, all
four of these macros evaluate to nothing. When ZTS is enabled however,
they expand out to the following definitions:
TSRMLS_C tsrm_ls
TSRMLS_D void ***tsrm_ls
TSRMLS_CC , tsrm_ls
TSRMLS_DC , void ***tsrm_ls
PHP relies on global variables from resource type identifiers, to
function callback pointers, to request specific information such as
the symbol tables used to store userspace variables. Attempting to
pass these values around in the parameter stack would be more than
unruly, it'd be impossible for an application like PHP where it's
often necessary to register callbacks with external libraries which
don't support context data.
So common information, like the execution stack, the function and
class tables, and extension registries all sit up in the global
scope where they can be picked up and used at any point in the
application.
For single-threaded SAPIs like CLI, Apache1, or even Apache2-prefork,
this is perfectly fine. Request specific structures are initialized
during the RINIT/Activation phase, and reset back to their original
values during the RSHUTDOWN/Deactivation phase in preparation for
the next request. A given webserver like Apache1 can serve up multiple
pages at once because it spawns multiple processes each in their own
process space with their own independant copies of global data.
The trouble starts with threaded webservers like Apache2-worker, or IIS
where two or more threads trying to run the a request at the same time.
Each thread wants to use the global scope to store its request-specific
information, and tries to do so by writing to the same
storage space. At the least, this would result in userspace variables
declared in one script showing up in another. In practice, it leads to
quick and disasterous segfaults and completely unpredictable behavior as
memory is double freed or written with conflicting information by separate
threads.
*/
#pragma include "/usr/local/include/php"
#pragma include "/usr/local/include/php/main"
#pragma include "/usr/local/include/php/TSRM"
#pragma include "/usr/local/include/php/Zend"
#pragma link "/usr/local/lib/libphp5.so"
#include "gwan.h" // G-WAN exported functions
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/syscall.h>
#include <php/sapi/embed/php_embed.h>
#include <php/Zend/zend_stream.h>
static pid_t gettid(void) { return syscall(__NR_gettid); }
// PHP
static int ub_write(const char *str, unsigned int str_len TSRMLS_DC)
{
puts(str); // this is the stdout output of a PHP script
return 0;
}
static void log_message(char * message)
{
printf("log_message: %s\n", message);
}
static void sapi_error(int type, const char * fmt, ...) { }
static void php_set_var(char *varname, char *varval)
{
zval *var;
MAKE_STD_ZVAL(var);
ZVAL_STRING(var, varval, 1);
zend_hash_update(&EG(symbol_table), varname, strlen(varname) + 1,
&var, sizeof(zval*), NULL);
}
static char *php_get_var(char *varname)
{
zval **data = NULL;
char *ret = NULL;
if(zend_hash_find(&EG(symbol_table), varname, strlen(varname) + 1,
(void**)&data) == FAILURE)
{
printf("Name not found in $GLOBALS\n");
return "";
}
if(!data)
{
printf("Value is NULL (not possible for symbol_table?)\n");
return "";
}
ret = Z_STRVAL_PP(data);
return ret;
}
static int php_init(void)
{
static int once = 0;
if(once) return 0;
once = 1;
static char *myargv[2] = {"toto.php", NULL};
php_embed_module.log_message = log_message;
php_embed_module.sapi_error = sapi_error;
php_embed_module.ub_write = ub_write;
if(php_embed_init(1, myargv PTSRMLS_CC) == FAILURE)
{
printf("php_embed_init error\n");
return 1;
}
return 0;
}
static void php_shutdown()
{
php_embed_shutdown(TSRMLS_C);
}
static int php_exec(char *str)
{
zval ret_value;
int exit_status;
zend_first_try
{
PG(during_request_startup) = 0;
// run the specified PHP script file
// sprintf(str, "include (\"% s \ ");", scriptname);
zend_eval_string(str, &ret_value, "toto.php" TSRMLS_CC);
exit_status = Z_LVAL(ret_value);
} zend_catch
{
exit_status = EG(exit_status);
}
zend_end_try();
return exit_status;
}
__thread char reply_num[8] = {0};
__thread pid_t tid = 0;
int main(int argc, char *argv[])
{
if(!tid)
{
tid = gettid();
s_snprintf(reply_num, 8, "%u", tid);
php_init();
}
xbuf_t *reply = get_reply(argv);
//php_set_var("argv", argv[0]);
php_set_var(reply_num, "");
char fmt[] = //"print(\"from php [$test]\n\");\n"
"$reply%s = \"Hello World (PHP)\";\n";
char php[sizeof(fmt) + sizeof(reply_num) + 2];
s_snprintf(php, sizeof(php), fmt, reply_num);
php_exec(php);
xbuf_cat(reply, php_get_var(reply_num));
return 200;
}
If anybody can make this code work with more than one worker thread without crashing the PHP runtime, then PHP will be added to G-WAN.
Here is what G-WAN produces with one single worker thread:
-----------------------------------------------------
weighttp -n 100000 -c 100 -t 1 -k "http://127.0.0.1:8080/?php.c"
finished in 0 sec, 592 millisec, **168744 req/s**, 48283 kbyte/s
requests: 100000 total/started/done/succeeded, 0 failed/errored
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 29299985 bytes total, 27599985 bytes http,
1700000 bytes data
-----------------------------------------------------
That would be great to resolve this PHP threading issue. Thanks for helping anyone!

Categories