We Keep Coding

IronPython: Error when launching .py script from c#

I'm running into a similar problem as this guy: IronPython : Microsoft.Scripting.SyntaxErrorException: 'unexpected token '=''
Unfortunately, there we no answers on that thread.
This is my code:
var engine = Python.CreateEngine();
var scope = engine.CreateScope();
try
{
engine.ExecuteFile(String.Concat(Directory.GetParent(Environment.CurrentDirectory).Parent.Parent.FullName, "\\Client.py"), scope);
}
catch(Exception e)
{
Console.WriteLine(e.Message);
}
And then here's the .py (that doesn't do anything yet, really):
#imports
import os
import tempfile
#Test
print("Here we go.")
The abomination to get the full path for my python file was an attempt to check if it got the path wrong or couldn't find the file, wasn't the case but I left it there. Debugger shows that the path is correct. However, it always fails on engine.ExecuteFile(...). and catches an exception that, according to the debugger is null. I got this error:
Microsoft.Scripting.SyntaxErrorException
and then goofed around with settings, changing Tools > Options > Debugging > General > "Enable just my Code" from checked to unchecked which lead to me not getting the SyntaxErrorException anymore but instead it's now this, but it still fails at the same line, with an exception that is still null:
IronPython.Runtime.Exceptions.ImportException in Microsoft.Dynamic.dll
At this point I don't know if I made a step in the right direction or went one back. Can anyone help with this?
EDIT: I need to correct this. There currently is an exception that states: "No module named os" instead of being just null which makes sense considering the exception type.

I moved the Lib folder to my project folder and had the search path extended by it like this:
String projectPath = Directory.GetParent(Environment.CurrentDirectory).Parent.Parent.FullName;
var engine = Python.CreateEngine();
var libs = new[]
{
String.Concat(projectPath, "\\Lib")
};
var pySP = engine.GetSearchPaths();
foreach (String resource in libs)
{
pySP.Add(resource);
}
following this thread: IronPython: No module named json.
Then I undid the changes I made to the debugging settings, and lastly added NuGet packages. Maybe there is now redundance with the added search path and the newly added packages but I am not willing to test my luck and undo any of the changes. Here is my .csproj file, for anyone who might need it when in the same position:
<Project Sdk="Microsoft.NET.Sdk.WindowsDesktop">
<PropertyGroup>
<OutputType>WinExe</OutputType>
<TargetFramework>netcoreapp3.1</TargetFramework>
<UseWPF>true</UseWPF>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="IronPython" Version="2.7.10" />
<PackageReference Include="IronPython.Interpreter" Version="2.7.4" />
<PackageReference Include="IronPython.StdLib" Version="2.7.10" />
</ItemGroup>
</Project>

MSBuild auto increment build version differently for release/debug

Context:
I use MSBuild to build my projects. Currently I use a date of release version number that, unfortunately, lacks clarity when multiple releases occur in the same day. In Directory.Build.props:
<PropertyGroup>
<Version>
$([System.DateTime]::Now.Year).
$([System.DateTime]::Now.Month).
$([System.DateTime]::Now.Day).
$([System.Convert]::ToUInt16(
$([MSBuild]::Divide(
$([System.DateTime]::Now.TimeOfDay.TotalSeconds),
1.32
))
))
</Version>
</PropertyGroup>
Goal:
Create a versioning scheme that looks something like this:
3/23/20:
Release Build: 2020.3.23.0
3/24/20:
Debug Build: 2020.3.24.0
Debug Build: 2020.3.24.1
Debug Build: 2020.3.24.2
Release Build: 2020.3.24.0
Debug Build: 2020.3.24.3
Release Build: 2020.3.24.1
Essentially: the first three numbers are year/month/day, because date of release is frequently important. Then use auto incrementing version numbers for releases within the same day. Incrementing on debug is useful so I can confirm the correct version of software is being loaded and run, but I don't want confusingly high numbers on release builds. I may play around with some additional indicator for debug builds, but I should be able to figure that out on my own.
Question:
How can I auto increment builds within the same day, having a separate version for debug and release? Ideally solutions that don't add additional dependencies are preferred, but if there is no way without, then it is acceptable.

MSBuild auto increment build version differently for release/debug
In general, MSBuild did not have a function to see the version number of the obvious incremental build but only used the timestamp of the system build determines the build order as you used before.
In fact, if you create a custom property in msbuild to record the version number of the incremental build, it still needs to use an entity to store the record, and if it is not used, the parameter is reinitialized for each build (the msbuild attribute can only be identified in msbuild).
So the ideal way it that use textfile as an intermediate. You can follow my solution:
Solution
1) create a custom msbuild task which does increment the value of the record property.
--a) Create a class library project called MyCustomTask then Right-click on the project-->Add Reference-->reference Microsoft.Build.Framework dll and Microsoft.Build.Utilities.v4.0 dll.
--b) add these into CustomTask.cs(this is the name of the task which will be used in xxx.csproj file).
public class CustomTask : Task
{
private int _number;
[Required]
public int number //input taskitem
{
get { return _number; }
set { _number = value; }
}
private int _lastnumber;
[Output]
public int LastNumber //output value
{
get { return _lastnumber; }
set { _lastnumber = value; }
}
public override bool Execute() // Execution logic
{
LastNumber = number + 1;
return true;
}
}
--c) Then build the project and remember to store its MyCustomTask dll.
2) Aim to your main project and then create two txt files called Debug.txt,Release.txt and give each of them an initial value of 0.
3) add these into your Directory.Build.props file:
<Project>
<UsingTask TaskName="CustomTask" AssemblyFile="xxxxxx\MyCustomTask\MyCustomTask\MyCustomTask\bin\Debug\MyCustomTask.dll(the local path of the dll)"> </UsingTask>
<PropertyGroup>
<Record></Record>
</PropertyGroup>
<Target Name="WriteToFile1" BeforeTargets="PrepareForBuild">
<PropertyGroup>
<Record Condition="'$(Configuration)'=='Debug' and !Exists('$(TargetPath)')">
0
</Record>
<Record Condition="'$(Configuration)'=='Release'and !Exists('$(TargetPath)')">
0
</Record>
</PropertyGroup>
<ItemGroup Condition="'$(Configuration)'=='Debug'">
<MyTextFile Include="Debug.txt">
<Number>$(Record)</Number>
</MyTextFile>
</ItemGroup>
<ItemGroup Condition="'$(Configuration)'=='Release'">
<MyTextFile Include="Release.txt">
<Number>$(Record)</Number>
</MyTextFile>
</ItemGroup>
<WriteLinesToFile
File="#(MyTextFile)"
Lines="$(Record)"
Overwrite="true"
Encoding="Unicode" Condition="'$(Configuration)'=='Debug'"/>
<WriteLinesToFile
File="#(MyTextFile)"
Lines="$(Record)"
Overwrite="true"
Encoding="Unicode" Condition="'$(Configuration)'=='Release'"/>
<PropertyGroup>
<Version>
$([System.DateTime]::Now.Year).
$([System.DateTime]::Now.Month).
$([System.DateTime]::Now.Day).
$(Record)
</Version>
</PropertyGroup>
</Target>
<Target Name="ReadLineFromFile" BeforeTargets="WriteToFile1">
<ReadLinesFromFile File="Debug.txt" Condition="'$(Configuration)'=='Debug'">
<Output TaskParameter="Lines" PropertyName="Record"/>
</ReadLinesFromFile>
<ReadLinesFromFile File="Release.txt" Condition="'$(Configuration)'=='Release'">
<Output TaskParameter="Lines" PropertyName="Record"/>
</ReadLinesFromFile>
<CustomTask number="$(Record)">
<Output TaskParameter="LastNumber" PropertyName="Record"/>
</CustomTask>
</Target>
</Project>
4) When you execute a task which depends on Build to show the property Version, it will work well as you hope.
Note that it will work for incremental build and if you click Rebuild(which execute Clean and then Build), it will set the version number to zero and start the rethrow.
Overall, this is an ideal solution which I try to realize it.

Retrieve assembly version for use in Jenkins build

Tools
MSBuild v14
Visual Studio 2013
Jenkins v2.111 running on Windows Server 2012
Git (bare repo on local file server)
Windows Batch
My goal
Build a c# Visual Studio project using MSBuild that pulls back the major and minor version numbers from the projects AssemblyInfo.cs for use during the build. The build would produce something like 1.2.$BUILD_NUMBER resulting in something like 1.2.121, 1.2.122, 1.2.123 and so on. Once the user opts to 'release' the build, a clickonce deployment with correct version in the folder name is copied to its target destination and a tag applied to the Git repository.
Pipeline example
Below is a 'work in progress' of what I've got up to. Any suggestions to improve are welcome. For those that are wondering why I'm coping the codebase out to a temporary folder. I'm using a multi-branch job in Jenkins and the folders that are auto-generated are extremely long! This gave me errors along the lines that my file name, project name or both are too long (because the entire path is above the 255 or so character length). So the only way to get around this was to copy out contents so the build and publish would work.
pipeline {
agent none
stages {
stage ('Checkout'){
agent any
steps
{
checkout scm
}
}
stage ('Nuget Restore'){
agent any
steps
{
bat 'nuget restore "%WORKSPACE%\\src\\Test\\MyTestSolution.sln"'
}
}
stage('Build Debug') {
agent any
steps
{
bat "xcopy %WORKSPACE%\\src\\* /ey d:\\temp\\"
bat "\"${tool 'MSBuild'}\" d:\\temp\\Test\\MyTestSolution.sln /p:Configuration=Debug /target:publish /property:PublishUrl=d:\\temp\\ /p:OutputPath=d:\\temp\\build\\ /p:GenerateBootstrapperSdkPath=\"C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v8.1A\\Bootstrapper\" /p:VersionAssembly=1.0.$BUILD_NUMBER /p:ApplicationVersion=1.0.$BUILD_NUMBER"
}
}
stage('Deploy to Dev'){
agent none
steps {
script {
env.DEPLOY_TO_DEV = input message: 'Deploy to dev?',
parameters: [choice(name: 'Deploy to dev staging area?', choices: 'no\nyes', description: 'Choose "yes" if you want to deploy this build')]
}
}
}
stage ('Deploying to Dev')
{
agent any
when {
environment name: 'DEPLOY_TO_DEV', value: 'yes'
}
steps {
echo 'Deploying debug build...'
}
}
stage ('Git tagging')
{
agent any
steps
{
bat 'd:\\BuildTargets\\TagGit.bat %WORKSPACE% master v1.0.%BUILD_NUMBER%.0(DEV) "DEV: Build deployed."'
}
}
}
}
At the moment I've hard coded the major and minor version in the above script. I want to pull these values out of the AssemblyInfo.cs so that developers can control it from there without editing the Jenkinsfile. Any suggestions/best practice to achieve this?
Because I'm doing a clickonce deployment for a winforms app I've had to use MSBuild's VersionAssembly and ApplicationVersion switches to pass in the version. This seems to help with correctly labelling folders when MSBuild publishes the files. Have I have missed something in my setup which would negate these switches and make life simpler?
The last action in my pipeline is to trigger a .bat file to add a tag back into the master branch of the repository. This is another reason that I need to make the major and minor version accessible to the pipeline script.
MSBuild target for editing AssemblyInfo.cs
This code was taken from here: http://www.lionhack.com/2014/02/13/msbuild-override-assembly-version/
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<CompileDependsOn>
CommonBuildDefineModifiedAssemblyVersion;
$(CompileDependsOn);
</CompileDependsOn>
</PropertyGroup>
<Target Name="CommonBuildDefineModifiedAssemblyVersion" Condition="'$(VersionAssembly)' != ''">
<!-- Find AssemblyInfo.cs or AssemblyInfo.vb in the "Compile" Items. Remove it from "Compile" Items because we will use a modified version instead. -->
<ItemGroup>
<OriginalAssemblyInfo Include="#(Compile)" Condition="%(Filename) == 'AssemblyInfo' And (%(Extension) == '.vb' Or %(Extension) == '.cs')" />
<Compile Remove="**/AssemblyInfo.vb" />
<Compile Remove="**/AssemblyInfo.cs" />
</ItemGroup>
<!-- Copy the original AssemblyInfo.cs/.vb to obj\ folder, i.e. $(IntermediateOutputPath). The copied filepath is saved into #(ModifiedAssemblyInfo) Item. -->
<Copy SourceFiles="#(OriginalAssemblyInfo)"
DestinationFiles="#(OriginalAssemblyInfo->'$(IntermediateOutputPath)%(Identity)')">
<Output TaskParameter="DestinationFiles" ItemName="ModifiedAssemblyInfo"/>
</Copy>
<!-- Replace the version bit (in AssemblyVersion and AssemblyFileVersion attributes) using regular expression. Use the defined property: $(VersionAssembly). -->
<Message Text="Setting AssemblyVersion to $(VersionAssembly)" />
<RegexUpdateFile Files="#(ModifiedAssemblyInfo)"
Regex="Version\("(\d+)\.(\d+)(\.(\d+)\.(\d+)|\.*)"\)"
ReplacementText="Version("$(VersionAssembly)")"
/>
<!-- Include the modified AssemblyInfo.cs/.vb file in "Compile" items (instead of the original). -->
<ItemGroup>
<Compile Include="#(ModifiedAssemblyInfo)" />
</ItemGroup>
</Target>
<UsingTask TaskName="RegexUpdateFile" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
<ParameterGroup>
<Files ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true" />
<Regex ParameterType="System.String" Required="true" />
<ReplacementText ParameterType="System.String" Required="true" />
</ParameterGroup>
<Task>
<Reference Include="System.Core" />
<Using Namespace="System" />
<Using Namespace="System.IO" />
<Using Namespace="System.Text.RegularExpressions" />
<Using Namespace="Microsoft.Build.Framework" />
<Using Namespace="Microsoft.Build.Utilities" />
<Code Type="Fragment" Language="cs">
<![CDATA[
try {
var rx = new System.Text.RegularExpressions.Regex(this.Regex);
for (int i = 0; i < Files.Length; ++i)
{
var path = Files[i].GetMetadata("FullPath");
if (!File.Exists(path)) continue;
var txt = File.ReadAllText(path);
txt = rx.Replace(txt, this.ReplacementText);
File.WriteAllText(path, txt);
}
return true;
}
catch (Exception ex) {
Log.LogErrorFromException(ex);
return false;
}
]]>
</Code>
</Task>
</UsingTask>
</Project>
Git tagging
This bat file is kicked off and passed values used to create and push a tag to the defined repository.
echo off
set gitPath=%1
set gitBranchName=%2
set gitTag=%3
set gitMessage=%4
#echo on
#echo Adding tag to %gitBranchName% branch.
#echo Working at path %gitPath%
#echo Tagging with %gitTag%
#echo Using commit message: %gitMessage%
d:
cd %gitPath%
git checkout %gitBranchName%
git pull
git tag -a %gitTag% -m %gitMessage%
git push origin %gitBranchName% %gitTag%
If there are any other gold nuggests that would help streamline or improve this overall workflow, would welcome those too!

I recently had the same problem which i solved by creating a Windows Script.
for /f delims^=^"^ tokens^=2 %%i in ('findstr "AssemblyFileVersion" %1\\AssemblyFile.cs') DO SET VERSION=%%i
This script extracts the version number from the AssemblyInfo.cs and put it inside an variable so it can be used later to tag the commit (in the same step though) :
CALL FindAssemblyVersion .\Properties
git tag %VERSION%
git push http://%gitCredentials%#url:port/repo.git %VERSION%

Not exactly from the assembly file but a very handy workaround to get the file version from the DLL while working with Jenkins, and using batch (or powershell) command:
Goto the directory where your DLL exists [CD Foo/Bar ]
FOR /F "USEBACKQ" %F IN (`powershell -NoLogo -NoProfile -Command (Get-Item "myApi.dll").VersionInfo.FileVersion`) DO (SET fileVersion=%F )
echo File version: %fileVersion%

MSBuild custom task depending on another project

I have an odd solution where I need one of the projects to "compile" files in another one.
The compiler (showing here a minimal example) is as follows (MSBuild custom task):
public class MyCompileTask : Task
{
[Required]
public ITaskItem[] InputFiles { get; set; }
[Output]
public ITaskItem[] OutputFiles { get; set; }
public override bool Execute()
{
var generatedFileNames = new List<string>();
foreach (var inputFile in this.InputFiles)
{
var inputFileName = inputFile.ItemSpec;
var outputFileName = Path.ChangeExtension(inputFileName, ".res.txt");
var source = File.ReadAllText(inputFileName);
var compiled = source.ToUpper();
File.WriteAllText(outputFileName, compiled + "\n\n" + DateTime.Now);
generatedFileNames.Add(outputFileName);
}
this.OutputFiles = generatedFileNames.Select(name => new TaskItem(name)).ToArray();
return true;
}
}
As you see, it only uppercases the content of the input files.
This was project A - the "compiler" library.
Project B, for now the main application, has a file "lorem.txt" that needs to be "compiled" into "lorem.res.txt" and put as an EmbeddedResource in B.exe/B.dll.
In B.csproj I added the following:
<PropertyGroup>
<CoreCompileDependsOn>$(CoreCompileDependsOn);InvokeMyCompile</CoreCompileDependsOn>
</PropertyGroup>
<UsingTask TaskName="MyCompiler.MyCompileTask" AssemblyFile="$(MSBuildProjectDirectory)\..\MyCompiler\bin\$(Configuration)\MyCompiler.dll" />
<Target Name="MyCompile" Inputs="lorem.txt" Outputs="lorem.res.txt">
<MyCompileTask InputFiles="lorem.txt">
<Output TaskParameter="OutputFiles" PropertyName="OutputFiles" />
</MyCompileTask>
</Target>
<Target Name="InvokeMyCompile" Inputs="lorem.txt" Outputs="lorem.res.txt">
<Exec Command=""$(MSBuildBinPath)\MSBuild.exe" /t:MyCompile "$(ProjectDir)$(ProjectFileName)"" />
</Target>
(The 2 layers of targets and an explicit msbuild.exe invocation is a workaround to another problem. In fact, much of this example is stolen from that Q.)
The most important part works, i.e. when I change lorem.txt and build, lorem.res.txt gets regenerated.
However:
When lorem.res.txt is physically deleted, a build does nothing (says it's up-to-date) until I actually refresh the project in VS. So, MSBuild does not "know" that lorem.res.txt is actually required to build the project.
More importantly, when I change anything in project A, project B recompiles but does not re-run the compilation lorem.txt -> lorem.res.txt. So MSBuild does not "know" that the transformation is dependent on another project.
How can I declare these dependencies in the csproj file?
Bonus question: how to mark the output file (lorem.res.txt) as a generated EmbeddedResource so I don't have to track it in VS but it's still put into the assembly?

•When lorem.res.txt is physically deleted, a build does nothing (says it's up-to-date) until I actually refresh the project in VS. So, MSBuild does not "know" that lorem.res.txt is actually required to build the project.
I create a demo and reproduce your issue on my side, you could use msbuild command line to avoid it.
•More importantly, when I change anything in project A, project B recompiles but does not re-run the compilation lorem.txt -> lorem.res.txt. So MSBuild does not "know" that the transformation is dependent on another project.
Because the custom task reference the DLL file, when change anything in project A, you need to rebuild project to generate newer DLL file.
Bonus question: how to mark the output file (lorem.res.txt) as a generated EmbeddedResource so I don't have to track it in VS but it's still put into the assembly?
You can add custom ItemGroup in BeforeBuild target to achieve it.
<Target Name="BeforeBuild" DependsOnTargets="MyCompile">
<ItemGroup>
<Content Include="lorem.res.txt">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Target>

How to transform files before adding them to an assembly?

I would like to do the following :
(project is a User Control library for WPF)
add a bunch of .FX (shader source code) files to the project as resources (Build action)
transform each to a .PS file (compiled shader) by invoking FXC.EXE utility
use the resulting file in place of the inputted file
I have been looking to write a CustomTool, unfortunately the tool is never seen by Visual Studio as it's mentioned in the article. In the article it is said that sometimes it is not seen but in my case it translates to every time.
I also looked at MSBuild Transforms but I'm not really sure if it would be appropriate for the task.
The goal of this is to include shader files source code and transform them at build time instead of manually building them from command line and dropping them every time to the project.
Do you know how one can achieve this ? Any methods are welcome
EDIT
Answer thanks to #Luaan :
public class CompileEffectTask : Task
{
public string[] Files { get; set; }
public override bool Execute()
{
if (Files != null)
{
foreach (string file in Files)
{
if (file != null)
{
Log.LogMessage(MessageImportance.High, file);
string s = #"C:\Program Files (x86)\Windows Kits\8.1\bin\x86\fxc.exe";
string changeExtension = Path.ChangeExtension(file, "ps");
string arguments = string.Format("/T ps_3_0 /Fo \"{0}\"" + " " + "\"{1}\"", changeExtension,
file);
Log.LogMessage(MessageImportance.High, arguments);
var process = new Process
{
StartInfo = new ProcessStartInfo(s, arguments)
};
process.Start();
process.WaitForExit();
}
}
}
return true;
}
}
And the MSBuild part :
<UsingTask TaskName="CompileEffectTask" AssemblyFile="D:\HLSLCompiler.dll" />
<PropertyGroup>
<BuildDependsOn>
MyCustomTarget1;
$(BuildDependsOn);
</BuildDependsOn>
</PropertyGroup>
<Target Name="MyCustomTarget1">
<Message Text="CompileEffectTask started" Importance="high" />
<Message Text="Compiling FX files ..." Importance="high" />
<CompileEffectTask Files="#(CompileEffectTask)"/>
<Message Text="Adding resulting .PS files as resources ..." Importance="high" />
<ItemGroup>
<Resource Include="**\*.ps" />
</ItemGroup>
</Target>
<Target Name="AfterBuild">
<CreateItem Include="**\*.ps">
<Output TaskParameter="Include" ItemName="DeleteAfterBuild" />
</CreateItem>
<Delete Files="#(DeleteAfterBuild)" />
</Target>
(still needs some cleaning but it works :D)

Custom tools do work, in fact, but they're rather tricky to setup - they're COM extensions to Visual Studio. However, the better solution for your case would be a custom build target or a pre-build event anyway - custom tools (code generators) are better suited for generating code (text) rather than binary files.
So, the pre-build event is the simple one. It's just some script that's run before the project starts building. You can find it in project properties. The simplest way would be to have all your .fx files in one directory, and in the pre-build event, you'd just call fxc.exe on each of them.
Now, build targets are cooler. They allow you to add your own build actions to files, among other things. So you'd just select CompileEffect in Build action of your files, and magic happens.
The target file can be quite simple:
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<AvailableItemName Include="CompileEffect"></AvailableItemName>
</ItemGroup>
</Project>
Or you can just put the ItemGroup part inside of your project file directly (otherwise you'd want to include this target file).
Next, you want to set the task as part of your build:
<PropertyGroup>
<BuildDependsOn>
MyCompileTarget;
$(BuildDependsOn);
</BuildDependsOn>
</PropertyGroup>
This basically says "run my build target first, and after that whatever you'd want".
Now, for the building:
<Target Name="MyCompileTarget">
<CompileEffectTask
ProjectDirectory="$(ProjectDir)"
Files="#(CompileEffect)"
RootNamespace="$(RootNamespace)">
</CompileEffectTaskTask>
</Target>
How does Visual Studio know what CompileEffectTask is?
<UsingTask TaskName="MyAssembly.CompileEffectTask"
AssemblyFile="C:\MyAssembly.dll"/>
And then you just need to implement the compiler task itself.
Now, if you only want to call an executable or a batch script, you don't even need that custom task, because there's a lot of built-in tasks in MSBuild (and even more in MSBuild Community Tasks). Exec task should work:
<Target Name="MyCompileTarget">
<Exec Command="fxc.exe #(CompileEffect)" />
</Target>
You might have to write a for cycle there, I'm not entirely sure. There's a lot of things you can do to customize project builds, http://msdn.microsoft.com/en-us/library/0k6kkbsd.aspx (especially the Task refecence part) is a rather good start.