I'm doing some experiments with Microsoft Dynamics CRM. You interact with it through web services and I have added a Web Reference to my project. The web service interface is very rich, and the generated "Reference.cs" is some 90k loc.
I'm using the web reference in a console application. I often change something, recompile and run. Compilation is fast, but newing up the web service reference is very slow, taking some 15-20 seconds:
CrmService service = new CrmService();
Profiling reveals that all time is spent in the SoapHttpClientProtocol constructor.
The culprit is apparently the fact that the XML serialization code (not included in the 90k loc mentioned above) is generated at run time, before being JIT'ed. This happens during the constructor call. The wait is rather frustrating when playing around and trying things out.
I've tried various combinations of sgen.exe, ngen and XGenPlus (which takes several hours and generates 500MB of additional code) but to no avail. I've considered implementing a Windows service that have few CrmService instances ready to dish out when needed but that seems excessive.
Any ideas?
The following is ripped from this thread on the VMWare forums:
Hi folks,
We've found that sgen.exe does work. It'just that there is a couple of additional steps beyond pre-generating the serializer dll's that we missed in this thread. Here is the detailed instruction
PROBLEM
When using the VIM 2.0 SDK from .NET requires long time to instantiate the VimService class. (The VimService class is the proxy class generated by running 'wsdl.exe vim.wsdl vimService.wsdl')
In other words, the following line of code:
_service = new VimService();
Could take about 50 seconds to execute.
CAUSE
Apparently, the .NET XmlSerializer uses the System.Xml.Serialization.* attributes annotating the proxy classes to generate serialization code in run time. When the proxy classes are many and large, as is the code in VimService.cs, the generation of the serialization code can take a long time.
SOLUTION
This is a known problem with how the Microsoft .NET serializer works.
Here are some references that MSDN provides about solving this problem:
http://msdn2.microsoft.com/en-us/library/bk3w6240.aspx
http://msdn2.microsoft.com/en-us/library/system.xml.serialization.xmlserializerassemblyattribute.aspx
Unfortunately, none of the above references describe the complete solution to the problem. Instead they focus on how to pre-generate the XML serialization code.
The complete fix involves the following steps:
Create an assembly (a DLL) with the pre-generated XML serializer code
Remove all references to System.Xml.Serialization.* attributes from the proxy code (i.e. from the VimService.cs file)
Annotate the main proxy class with the XmlSerializerAssemblyAttribute to point it to where the XML serializer assembly is.
Skipping step 2 leads to only 20% improvement in the instantiation time for the VimService class. Skipping either step 1 or 3 leads to incorrect code. With all three steps 98% improvement is achieved.
Here are step-by-step instructions:
Before you begin, makes sure you are using .NET verison 2.0 tools. This solution will not work with version 1.1 of .NET because the sgen tool and the XmlSerializationAssemblyAttribute are only available in version 2.0 of .NET
Generate the VimService.cs file from the WSDL, using wsdl.exe:
wsdl.exe vim.wsdl vimService.wsdl
This will output the VimService.cs file in the current directory
Compile VimService.cs into a library
csc /t:library /out:VimService.dll VimService.cs
Use the sgen tool to pre-generate and compile the XML serializers:
sgen /p VimService.dll
This will output the VimService.XmlSerializers.dll in the current directory
Go back to the VimService.cs file and remove all System.Xml.Serialization.* attributes. Because the code code is large, the best way to achieve that is by using some regular expression substitution tool. Be careful as you do this because not all attributes appear on a line by themselves. Some are in-lined as part of a method declaration.
If you find this step difficult, here is a simplified way of doing it:
Assuming you are writing C#, do a global replace on the following string:
[System.Xml.Serialization.XmlIncludeAttribute
and replace it with:
// [System.Xml.Serialization.XmlIncludeAttribute
This will get rid of the Xml.Serialization attributes that are the biggest culprits for the slowdown by commenting them out. If you are using some other .NET language, just modify the replaced string to be prefix-commented according to the syntax of that language. This simplified approach will get you most of the speedup that you can get. Removing the rest of the Xml.Serialization attributes only achieves an extra 0.2 sec speedup.
Add the following attribute to the VimService class in VimService.cs:
[System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = "VimService.XmlSerializers")]
You should end up with something like this:
// ... Some code here ...
[System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = "VimService.XmlSerializers")]
public partial class VimService : System.Web.Services.Protocols.SoapHttpClientProtocol {
// ... More code here
Regenerate VimSerice.dll library by
csc /t:library /out:VimService.dll VimService.cs
Now, from your application, you can add a reference to VimSerice.dll library.
Run your application and verify that VimService object instanciation time is reduced.
ADDITIONAL NOTES
The sgen tool is a bit of a black box and its behavior varies depending on what you have in your Machine.config file. For example, by default it is supposed to ouptut optimized non-debug code, but that is not always the case. To get some visibility into the tool, use the /k flag in step 3, which will cause it to keep all its temporary generated files, including the source files and command line option files it generated.
Even after the above fix the time it takes to instantiate the VimService class for the first time is not instantaneous (1.5 sec). Based on empirical observation, it appears that the majority of the remaining time is due to processing the SoapDocumentMethodAttribute attributes. At this point it is unclear how this time can be reduced. The pre-generated XmlSerializer assembly does not account for the SOAP-related attributes, so these attributes need to remain in the code. The good news is that only the first instantiation of the VimService class for that app takes long. So if the extra 1.5 seconds are a problem, one could try to do a dummy instantiation of this class at the beginning of the application as a means to improve user experience of login time.
You might wish to look into the Sgen.exe tool that comes with .NET. There's also a handy little thing in Visual Studio's C# project properties "Build" page, at the very bottom, called "Build serialization assembly" that automatically runs Sgen for you.
I believe that this is not an SGEN issue. I have looked at the constructor code, and I see that it is doing a lot of reflection (based on the XmlIncludeAttribute on the class). It reflects on all of them, and can take a really long time.
There is a pre-generated XmlSerializer assembly that comes with CRM. Check to see whether you have SdkTypeProxy.XmlSerializers.dll and SdkProxy.XmlSerializers.dll in the GAC.
If you don't then that means that when you create the CrmService, .net will generate the XmlSerializer assembly which can take some time.
Hope this helps
I came across this thread when trying to find out why my initial SoapHttpClientProtocol calls were taking so long.
I found that setting the Proxy to null/Empty stopped the Proxy AutoDetect from occurring - This was taking up to 7 seconds on the initial call:
this.Proxy = GlobalProxySelection.GetEmptyWebProxy();
I have used above detailed answer as guide, and went a few steps forward, making a script to automate process. Script is made out of two files :
generateproxy.bat :
REM if your path for wsdl, csc or sgen is missing, please add it here (it varies from machine to machine)
set PATH=%PATH%;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools;C:\Program Files (x86)\MSBuild\14.0\Bin
wsdl http://localhost:57237/VIM_WS.asmx?wsdl REM create source code out of WSDL
PowerShell.exe -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'" REM proces source code (remove annotations, add other annotation, put class into namespace)
csc /t:library /out:references\VIM_Service.dll VIM_WS.cs REM compile source into dll
sgen /p references\VIM_Service.dll /force REM generate serializtion dll
generateproxy.ps1
(Get-Content VIM.cs) |
ForEach-Object {
$_ -replace "(?<attr>\[global::System.Xml.Serialization.[^\]]*\])", "/*${attr}*/" `
-replace "public partial class VIM", "[System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = ""VIM_Service.XmlSerializers"")] `npublic partial class VIM" `
-replace "using System;", "namespace Classes.WS_VIM { `n`nusing System;"
} |
Set-Content VIM.cs
Add-Content VIM.cs "`n}"
I have added those two files to client project, and in the pre-build event I have added lines
cd..\..
generateproxy
So, before every build, proxy classes are regenerated, and developer has (almost) no need to think about it. While building, WS must be up and running, and its URL must be in bat file. As a result of prebuild, two dll files will regenerate in client project's subfolder references.
After first execution of scripts, you should add reference to new dll.
Related
I stumbled into a cooperation project, where the other part references an interface library of mine and deploys a self compiled MEF Plugin for our tool. I know which methods those guys are using and I want to monitor our library during the the build process, if the method signatures have been changed (just to make sure, noone checked in stuff, which should lead to another interface version and impairs the plugins loadability).
Actually, I have a console project in mind, where the signatures are somehow hardcoded and checked via reflection - but maybe there is a more elegant or simple way.
Any hint would be great.
Thanks in advance!
Roslyn 2.3 introduces a feature for generating reference assemblies. That is an assembly containing only public types and members. When used together with the "deterministic" feature (=> reproducible builds), the generated reference assembly remains binary identical as long as no changes to the public interface is made (implementation changes and private/internal members don't matter).
So you can add this to your csproj:
<PropertyGroup>
<Deterministic>true</Deterministic>
<ProduceReferenceAssembly>true</ProduceReferenceAssembly>
</PropertyGroup>
Until VS 2017 15.5 comes out, I suggest adding <CompileUsingReferenceAssemblies>false</CompileUsingReferenceAssemblies> to all consuming projects because the IDE (e.g. "go to definition") has some problems with this feature unless you are using the "new project system" that is used for .NET Core and .NET Standard projects. (The idea would be that projects referencing the project are only rebuilt if the public interface changes - this speeds up incremental build for large solutions when only implementations change).
These changes will create a ref folder in your output. You can then check if the checksum of the assmbly in there matches a known cheksum on each build.
I ended up creating a small console application with a try catch block, using the same interface dll and the same objects as the project partner does - compiled with the last released interface library. During execution it falls into the catch branch if the signatures got invalid (discovered by the normal .NET processes) - then the exitcode is raised with -1.
Doing all this in the post build processes, cathcing the exit code as discribed this article and breaking build automatically.
Not very happy with that solution, but got it working ... Further ideas still wanted :-)
When I recompile my project (asp.net, c#) with aspnet_compiler the rebuilt binaries change (when compared to the previous build) even if no code changes have been made.
This, I understand, is due to the build generating a new Module Version ID (guid) each time it builds (to distinguish between builds), another similar question talks about this: Can i specify the module version id (MVID) when building a .net assembly?
The above linked question seems to suggest there is no way to rebuild a project and have the binaries match a previous build of the same unchanged code.. ok, fine, I understand - but why are all the binaries being rebuilt at all?
I would think, according to the documentation ( http://msdn.microsoft.com/en-us/library/ms229863(v=vs.80).aspx ), that unless -c is specified as an argument the aspnet_compiler should only rebuild those binaries that actually need to be (due to changed code). Am I misunderstanding or maybe missing something?
The aspnet_compiler arguments I'm using:
aspnet_compiler -f -u -fixednames -nologo -v / -p .\myproject\ .\mybuild\
Note that this issue occurs only with a WebSite project, not a Web Application project (they are compiled differently).
Also this issue occurs even if you create a WebSite project and page with no functionality, and never open it or change it in anyway between builds.
Decompiling the binaries that are produced shows no differences. Comparing the binaries of two "identical" builds shows small differences in the same part of the binaries each time - which I believe is probably related to the random build guid. I've found no way of avoiding this change between builds.
Check out this excellent answer by Eric Lippert on how does the C# compiler makes multi passes to compile the source code. There can me many reasons why your build was not identical to the previous one, although the functionality is same.
Compilers replace special language features such as using block with with IL equivalents
The compilers does many optimizations on your code, each iteration may produce slightly different output.
Compilers have to create materialized names for anonymous method names and they are different each time you compile
And many more reasons you could easily figure it out using a dis-assembler
Check out these dis-assemblers and decompile your library or executable to gain better understanding.
http://ilspy.net/ , http://www.telerik.com/products/decompiler.aspx
I've found in many cases using the aspnet_compiler especially in situations where my projects have references to other project in the same solution results in full rebuilds that are often hard to explain. (though the few times I've investigated there were "changes" even if they don't truly effect anything such as changes to whitespace, comments, etc)
I've also had problems with a number of plugins in visual studio that have done everything from manipulate tabulation and other white space, the actual project file, etc. While these changes have no noticeable change to us humans, the compiler takes one look and goes "I see a change! REBUILD ALL THE THINGS!!!"
Not sure my answer is any help, but I would disable your plugins, run the compiler, then run the compiler again and see what happens...
I've just come across a pretty strange problem with VS2010 and Script#, which most of the time I am able to re-create.
In my simple scenario I have 2 projects in my solution; a standard Asp.Net MVC2 Web Application, and a Script# jQuery Class Library. I created a static class (attributed with [Imported]) with a static method on it, the intention being that I can map this class in code to an external Javascript library, as described in the documentation.
However, it seems that whenever I decorate such a class with [IgnoreNamespace] to achieve this goal, the project stops successfully compiling but doesn't give me any feedback as to why it's failing (no errors in the error window, for example). It's not easy to get rid of either, as Visual Studio seems to get into a permanent state of not build failure; removing the classes and project files doesn't solve it, nor restarting visual studio. The only way I can get VS to build the project successfully is to delete the project entirely, create a new one then add the files back in, which is annoying to say the least.
With a verbose build output setting, I get the following:
Target "AfterCompile" in file "C:\Program Files (x86)\ScriptSharp\v1.0\ScriptSharp.targets" from project "e:\project\local\ScriptSharpDemo\Scripts\Scripts.csproj" (target "Compile" depends on it):
Task "ScriptCompilerTask"
Done executing task "ScriptCompilerTask" -- FAILED.
Done building target "AfterCompile" in project "Scripts.csproj" -- FAILED.
.. which doesn't tell me whole lot.
There have been a couple of times where I have managed to create this type of class and then successfully build, but mostly I can reproduce this problem pretty reliably.
At this point I'm inclined to think that the bug lies with Script#, but would just like to have that confirmed, and to find a possible work around if there is one.
Just in case anyone is having a similar issue, I've found the cause of the problem.
When adding a class using this method, or copying in a file from another project for use within Script#, this causes a reference to System.dll to be added to the project. This (understandably) causes the project to stop compiling without error.
It would be nice to have a warning about this or for Script# to somehow detect when this situation occurs and/or create a new template for when I use 'Add class' or import a file, but it is just a convenience issue and at least now I can painlessly get my project compiling again just by removing this reference.
When trying to make my project build again, I came across the following, possible solutions:
The "Home\HomePage.cs" and "Shared\Utility.cs" must not be deleted and remain where they wre initially created
The "Home\HomePage.cs" and "Shared\Utility.cs" must be the last entries in the "*.csproj"-file. After them, no "Compile" tag should follow
Problematic calls to "Script.Literal" might cause silent fails - especially be careful when having parameters (like Script.Literal("{0}.doFoo()", variable))
The same seems to be true for "String.Format" when the format parameters are invalid
Namespaces and folders seem to cause many problems, putting all classes into the same namespace and all classes into the same folder might help
I tried all of the suggestions that have been given here, but continued to see the issue. Eventually, I determined that the cause in my situation was that I had added an [IntrinsicProperty] attribute to one of my properties. Removing it solved the issue. Don't ask me why this was causing a problem, but I thought I would share this solution in case others run into it.
Recently I've been working with MSTest, and I noticed that the testframework generates accessor classes dynamically at compile time. How can one do this?
There's an xml file in a VS2010 C# project. I'd like to make an enum out of certain data in this xml file. Can this be done? And if so, how?
I'd recommend T4 templates myself. Very easy to use and specifically designed to allow you to generate code during the build. http://msdn.microsoft.com/en-us/library/bb126445.aspx
Method A) Read the xml file, parse it, generate C# code from it, write out the C# code to a temp file, compile that code; delete the temp file.
Method B) Read the xml file, parse it. Generate IL code directly from it using method in the System.Reflection.Emit namespace, or those in the System.CodeDom namespace.
MSTest achieves this in a couple of different ways. In short they essentially do the following IIRC
Hook into the build system
At the start of the build they generate their acessor's into hidden files in the project
After the build completes they remove their files
You can achieve a similar effect via the same process. However hooking into the build system is a bit complicated. A much simpler approach is to build a custom tool / code generator and hook. This allows you to process a file at build time and spit out a corresponding code file to include in the build.
There are several examples on the web on how to achieve this. Here are a couple
http://www.raboof.com/Projects/VsCodeGeneratorShim/
http://www.ramymostafa.com/?p=204
The System.CodeDom Namespace is one option you have.
It allows you to automatically generate a class using C# Code and compile it as well.
You can maybe call this code as a postbuild during your build of your project.
This example shows how to create a class using this namespace
I do some stuff with XMLSerializer class. Like a most of beginners I face with performance issue on app start. I read a lot of blogs, articles and finally use SGEN tool. Now performance looks to be ok but few things are still not clear for me.
1) Should I use SGEN syntax like this:
SGen.exe /assembly:MyAssembly /type:MyRootXmlType
or it's enough to just (I use this syntax currently, I have only one Serializable class in my assembly):
SGen.exe /assembly:MyAssembly
When /type parameter can be useful in practice?
2) I read on MSDN ( http://msdn.microsoft.com/en-us/library/ee704594.aspx )
This command generates the serializer
assembly
MyAssembly.XmlSerializers.dll, which
should be added as a reference to the
current project or to the GAC.
but on all other blogs and articles (for example on SO) I read that is enough to just put MyAssembly.XmlSerializers.dll into folder where MyAssembly exists.
So what is right?
3) It's enough to just generate .XmlSerilizers.dll and deploy it together with MyAssembly or should I add something to my code yet?
I don't think so but I would like to be sure that I will not have to change my code and "magic" happens automatically.
If you don't use the /type argument for sgen.exe then it will generate de/serialization code for all public types in the assembly. Note that the [Serializable] attribute is not used in XML serialization. I doubt you'd want this, use /type to keep the generated assembly small.
Adding a reference is not necessary, Xml serialization always tries an Assembly.Load() on the .XmlSerializers.dll assembly anyway. Plus, you'll never reference the generated XmlSerializationWriterXxx and XmlSerializationReaderXxx classes directly in your code. It does have one advantage, the build system will automatically copy the assembly when you include the project in a solution.
Installing it in the GAC is only worth considering when different apps serialize and deserialize the XML file. You can provide other apps with the .XmlSerializers.dll assembly by copying the assembly by hand as well. Which is a bit error prone, use your own judgment here. Check the previous paragraph for a way to automate the copy.