Cannot use Spark.Net UDFs and HDInsight cluster - c#

I have tried to run a simple application in prod env containing the code from https://github.com/dotnet/spark/blob/master/examples/Microsoft.Spark.CSharp.Examples/Sql/Batch/Basic.cs
The applications runs fine and emits output to stdout until it this code crashes when it hits the first UDF.
Thanks for any insights you can share on this.
Env.
Code is packaged using
dotnet publish -c Release -f netcoreapp3.1 -r ubuntu.16.04-x64
HDInsight cluster HDI 4.0, Spark 2.4
-- Server is setup using the guidelines in https://learn.microsoft.com/en-us/dotnet/spark/tutorials/hdinsight-deployment
spark-submit --master yarn --conf spark.yarn.appMasterEnv.DOTNET_ASSEMBLY_SEARCH_PATHS="./app/publish.zip" --archives wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/publish.zip#mySparkApp --class org.apache.spark.deploy.dotnet.DotnetRunner wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/microsoft-spark-2.4.x-0.12.1.jar wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/publish.zip mySparkApp
(and all sorts of variations on this in desperation, --deploy-mode cluster, various paths, etc, etc. nothing works)
stdout:
...
+---+-----+
|age| name|
+---+-----+
| 22|Ricky|
| 36| Jeff|
| 62|Geddy|
+---+-----+
[2020-10-28T09:15:10.1478641Z] [wn0-hdinsi] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'showString' failed for class '41' when called with 3 arguments ([Index=1, Type=Int32, Value=20], [Index=2, Type=Int32, Value=20], [Index=3, Type=Boolean, Value=False], )
[2020-10-28T09:15:10.1480587Z] [wn0-hdinsi] [Error] [JvmBridge] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 16.0 failed 4 times, most recent failure: Lost task 0.3 in stage 16.0 (TID 210, wn0-hdinsi.xwccrqijnmqujdjghwrza0nzbb.fx.internal.cloudapp.net, executor 2): org.apache.spark.api.python.PythonException: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Spark.Utils.UdfSerDe.<>c.b__10_0(TypeData td) in //src/csharp/Microsoft.Spark/Utils/UdfSerDe.cs:line 262
at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
at Microsoft.Spark.Utils.UdfSerDe.DeserializeType(TypeData typeData) in //src/csharp/Microsoft.Spark/Utils/UdfSerDe.cs:line 258
at Microsoft.Spark.Utils.UdfSerDe.Deserialize(UdfData udfData) in //src/csharp/Microsoft.Spark/Utils/UdfSerDe.cs:line 160
at Microsoft.Spark.Utils.CommandSerDe.DeserializeUdfs[T](UdfWrapperData data, Int32& nodeIndex, Int32& udfIndex) in //src/csharp/Microsoft.Spark/Utils/CommandSerDe.cs:line 333
at Microsoft.Spark.Utils.CommandSerDe.Deserialize[T](Stream stream, SerializedMode& serializerMode, SerializedMode& deserializerMode, String& runMode) in /_/src/csharp/Microsoft.Spark/Utils/CommandSerDe.cs:line 306
at Microsoft.Spark.Worker.Processor.CommandProcessor.ReadSqlCommands(PythonEvalType evalType, Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 188
at Microsoft.Spark.Worker.Processor.CommandProcessor.ReadSqlCommands(PythonEvalType evalType, Stream stream, Version version) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 98
at Microsoft.Spark.Worker.Processor.CommandProcessor.Process(Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 43
at Microsoft.Spark.Worker.Processor.PayloadProcessor.Process(Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\PayloadProcessor.cs:line 82
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 143
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:456)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:410)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
------cut--------
My problem did indeed turn out to be an issue with paths. For anyone else having the same problem, I got this to work by having the dll with the UDF (can be the same dll as the general spark app) has to be listed in "--files". So essentially you need a zip file with assemblies AND then have links to the dlls directly. There is probably a smarter way, but that did it for me (when running on cluster mode):
spark-submit --deploy-mode cluster --master yarn --files wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/mySparkApp.dll --class org.apache.spark.deploy.dotnet.DotnetRunner wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/microsoft-spark-2.4.x-0.12.1.jar wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/publish.zip mySparkApp

The error is because the dll with your code can’t be found.
Two things, firstly in yarn mode . at the beginning of the DOTNET_ASSEMBLY_SEARCH_PATHS causes the users home directory to be prepended to the path so it isn’t currentdirectory/app/publish.zip so if that is different then it will be looking in the wrong place.
Secondly make sure the publish.zip doesn’t contain folders and the dll with the udf is at the top level of the zip.
Instead of putting the zip inside the app folder I would just use the current folder and don’t worry about DOTNET_ASSEMBLY_SEARCH_PATHS
For a walkthrough make sure you follow:
https://learn.microsoft.com/en-us/dotnet/spark/tutorials/hdinsight-deployment

Related

C# Windows Service - Getting Random Windows Error Reporting logs even when service not installed

I'm currently working on a C# Windows Service.
I'm logging various things to the Windows Event Log, and using the Event Viewer to check the results.
As happens during development, things don't work, and every now and then Service would break and the Windows Error Reporting would log lots of entries like
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0
Problem signature:
P1: MyServiceName.exe
P2: 1.0.0.0
P3: 5b9fcf54
P4: MyServiceName
P5: 1.0.0.0
P6: 5b9fcf54
P7: 280
P8: 16e
P9: System.NullReferenceException
P10:
Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERF8E3.tmp.WERInternalMetadata.xml
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_MyServiceName_...
Analysis symbol:
Rechecking for solution: 0
Report Id: 7c7efe65-6759-4a75-8581-72bc12182800
Report Status: 100
Hashed bucket:
Cab Guid: 0
All fair enough, or so I thought.
The problem is that I'm getting these reports at random times as well.
It might log them after five minutes, or half an hour, or anywhere in between.
I started off thinking something was broken, and trying to find the bug, but then it logged a load of errors while the service was uninstalled. Not just not running, completely uninstalled.
I have now tried all of the following, and I am STILL getting these random Windows Error Reporting logs, which is making it impossible to tell actual problems from this random junk:-
1) Uninstall the Service
It's not showing up in the Services App list.
If I start a Command Prompt in Administrator mode and type:
C:\WINDOWS\system32>sc queryex MyServiceName
it returns
[SC] EnumQueryServicesStatus:OpenService FAILED 1060:
The specified service does not exist as an installed service.
If I look through Processes and Services in Task Manager, nothing shows up
The service is uninstalled!
2) Rebooting
Just in case something was still in memory
3) Deleting every reference to MyServiceName in the Windows Registry
In case these was a dodgy registry key still kicking about
4) Rebooting
If all else fails...
5) Deleting the .EXE file
So there's no way it can be loaded and run
6) Rebooting yet again
Because why not!
And still, within a few minutes of rebooting, there they are...
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0
Problem signature:
P1: MyServiceName.exe
P2: 1.0.0.0
P3: 5b9fcf54
P4: MyServiceName
P5: 1.0.0.0
P6: 5b9fcf54
P7: 280
P8: 16e
P9: System.NullReferenceException
P10:
Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERF8E3.tmp.WERInternalMetadata.xml
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_MyServiceName_...
Analysis symbol:
Rechecking for solution: 0
Report Id: 7c7efe65-6759-4a75-8581-72bc12182800
Report Status: 100
Hashed bucket:
Cab Guid: 0
And a couple of dozen others just like it.
I haven't installed or run the Service since I started trying to get rid of these random errors, and as I mentioned I've even deleted the .EXE files so it can't be installed or run.
Anyone know why I'm still getting this random junk, and more importantly how to stop it?
Edit: JuanR asked for the AppCrash file, so here it is:-
Version=1
EventType=CLR20r3
EventTime=131817392609639254
ReportType=2
Consent=1
UploadTime=131817585683279486
ReportStatus=100
ReportIdentifier=428e2b66-f3ae-461f-8221-df0633ba6dad
IntegratorReportIdentifier=e2ac36e7-3506-403f-9efc-cd22cfac94da
Wow64Host=34404
Wow64Guest=332
NsAppName=MyServiceName.exe
OriginalFilename=MyServiceName.exe
AppSessionGuid=000015b4-0000-0007-71e5-c234384fd401
TargetAppId=W:00064dae5f701edaa06ce44c0466d2ceb81300000000!0000a6504bbe6f18e0042ad1f80d12f5a7c97896d572!MyServiceName.exe
TargetAppVer=2018//09//18:10:11:10!0!MyServiceName.exe
BootId=4294967295
ServiceSplit=13
TargetAsId=2065
IsFatal=1
Response.type=4
Sig[0].Name=Problem Signature 01
Sig[0].Value=MyServiceName.exe
Sig[1].Name=Problem Signature 02
Sig[1].Value=1.0.0.0
Sig[2].Name=Problem Signature 03
Sig[2].Value=5ba0cf3e
Sig[3].Name=Problem Signature 04
Sig[3].Value=System
Sig[4].Name=Problem Signature 05
Sig[4].Value=4.7.3151.0
Sig[5].Name=Problem Signature 06
Sig[5].Value=5b44403a
Sig[6].Name=Problem Signature 07
Sig[6].Value=2da3
Sig[7].Name=Problem Signature 08
Sig[7].Value=11f
Sig[8].Name=Problem Signature 09
Sig[8].Value=System.Security.Security
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=10.0.17134.2.0.0.256.48
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=2057
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=2beb
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=2beba6fb4680d73a8c78ca7c24ccdb46
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=b1f0
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=b1f0b380dbcd74b72a4df4e63607c2ae
UI[2]=C:\TFSOnline\Tools\MyServiceName\MyServiceName\bin\Debug\MyServiceName.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=MyServiceName stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\TFSOnline\Tools\MyServiceName\MyServiceName\bin\Debug\MyServiceName.exe
LoadedModule[1]=C:\WINDOWS\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\WINDOWS\SYSTEM32\MSCOREE.DLL
LoadedModule[3]=C:\WINDOWS\System32\KERNEL32.dll
LoadedModule[4]=C:\WINDOWS\System32\KERNELBASE.dll
LoadedModule[5]=C:\WINDOWS\System32\ADVAPI32.dll
LoadedModule[6]=C:\WINDOWS\System32\msvcrt.dll
LoadedModule[7]=C:\WINDOWS\System32\sechost.dll
LoadedModule[8]=C:\WINDOWS\System32\RPCRT4.dll
LoadedModule[9]=C:\WINDOWS\System32\SspiCli.dll
LoadedModule[10]=C:\WINDOWS\System32\CRYPTBASE.dll
LoadedModule[11]=C:\WINDOWS\System32\bcryptPrimitives.dll
LoadedModule[12]=C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll
LoadedModule[13]=C:\WINDOWS\System32\SHLWAPI.dll
LoadedModule[14]=C:\WINDOWS\System32\combase.dll
LoadedModule[15]=C:\WINDOWS\System32\ucrtbase.dll
LoadedModule[16]=C:\WINDOWS\System32\GDI32.dll
LoadedModule[17]=C:\WINDOWS\System32\gdi32full.dll
LoadedModule[18]=C:\WINDOWS\System32\msvcp_win.dll
LoadedModule[19]=C:\WINDOWS\System32\USER32.dll
LoadedModule[20]=C:\WINDOWS\System32\win32u.dll
LoadedModule[21]=C:\WINDOWS\System32\kernel.appcore.dll
LoadedModule[22]=C:\WINDOWS\SYSTEM32\VERSION.dll
LoadedModule[23]=C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
LoadedModule[24]=C:\WINDOWS\SYSTEM32\MSVCR120_CLR0400.dll
LoadedModule[25]=C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\mscorlib\399032397425364b053c532bbbeacc09\mscorlib.ni.dll
LoadedModule[26]=C:\WINDOWS\System32\ole32.dll
LoadedModule[27]=C:\Windows\Microsoft.NET\Framework\v4.0.30319\clrjit.dll
LoadedModule[28]=C:\WINDOWS\System32\OLEAUT32.dll
LoadedModule[29]=C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System\6e52f5ddc8a0027c55a2c15df97d50a9\System.ni.dll
LoadedModule[30]=C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Core\2d2bc5d43039ac23595b27676dcfcd3b\System.Core.ni.dll
LoadedModule[31]=C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Configuration\ce7b3ccf1b67903e135f62bd847db8dc\System.Configuration.ni.dll
LoadedModule[32]=C:\WINDOWS\System32\shell32.dll
LoadedModule[33]=C:\WINDOWS\System32\cfgmgr32.dll
LoadedModule[34]=C:\WINDOWS\System32\shcore.dll
LoadedModule[35]=C:\WINDOWS\System32\windows.storage.dll
LoadedModule[36]=C:\WINDOWS\System32\profapi.dll
LoadedModule[37]=C:\WINDOWS\System32\powrprof.dll
LoadedModule[38]=C:\WINDOWS\System32\FLTLIB.DLL
LoadedModule[39]=C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Xml\536177f34c4c0eeb95bcccd76ca90847\System.Xml.ni.dll
LoadedModule[40]=C:\WINDOWS\SYSTEM32\bcrypt.dll
LoadedModule[41]=C:\WINDOWS\SYSTEM32\CRYPTSP.dll
LoadedModule[42]=C:\WINDOWS\system32\rsaenh.dll
LoadedModule[43]=C:\WINDOWS\SYSTEM32\iphlpapi.dll
LoadedModule[44]=C:\WINDOWS\SYSTEM32\DNSAPI.dll
LoadedModule[45]=C:\WINDOWS\System32\WS2_32.dll
LoadedModule[46]=C:\WINDOWS\System32\NSI.dll
LoadedModule[47]=C:\WINDOWS\SYSTEM32\dhcpcsvc6.DLL
LoadedModule[48]=C:\WINDOWS\SYSTEM32\dhcpcsvc.DLL
LoadedModule[49]=C:\WINDOWS\SYSTEM32\WINNSI.DLL
LoadedModule[50]=C:\WINDOWS\SYSTEM32\activeds.dll
LoadedModule[51]=C:\WINDOWS\SYSTEM32\adsldpc.dll
LoadedModule[52]=C:\WINDOWS\System32\WLDAP32.dll
LoadedModule[53]=C:\WINDOWS\System32\clbcatq.dll
LoadedModule[54]=C:\WINDOWS\system32\adsldp.dll
LoadedModule[55]=C:\WINDOWS\SYSTEM32\sxs.dll
LoadedModule[56]=C:\WINDOWS\SYSTEM32\wkscli.dll
LoadedModule[57]=C:\WINDOWS\SYSTEM32\cscapi.dll
LoadedModule[58]=C:\WINDOWS\SYSTEM32\netutils.dll
LoadedModule[59]=C:\WINDOWS\SYSTEM32\logoncli.dll
LoadedModule[60]=C:\WINDOWS\system32\mswsock.dll
LoadedModule[61]=C:\Windows\System32\rasadhlp.dll
LoadedModule[62]=C:\WINDOWS\System32\fwpuclnt.dll
LoadedModule[63]=C:\WINDOWS\SYSTEM32\DSPARSE.dll
LoadedModule[64]=C:\WINDOWS\System32\msv1_0.DLL
LoadedModule[65]=C:\WINDOWS\SYSTEM32\NtlmShared.dll
LoadedModule[66]=C:\WINDOWS\SYSTEM32\cryptdll.dll
LoadedModule[67]=C:\Windows\Microsoft.NET\Framework\v4.0.30319\diasymreader.dll
LoadedModule[68]=C:\WINDOWS\System32\psapi.dll
LoadedModule[69]=C:\WINDOWS\SYSTEM32\rasapi32.dll
LoadedModule[70]=C:\WINDOWS\SYSTEM32\rasman.dll
LoadedModule[71]=C:\WINDOWS\SYSTEM32\rtutils.dll
LoadedModule[72]=C:\WINDOWS\SYSTEM32\winhttp.dll
LoadedModule[73]=C:\WINDOWS\SYSTEM32\secur32.dll
LoadedModule[74]=C:\WINDOWS\System32\schannel.dll
LoadedModule[75]=C:\WINDOWS\System32\CRYPT32.dll
LoadedModule[76]=C:\WINDOWS\System32\MSASN1.dll
LoadedModule[77]=C:\WINDOWS\SYSTEM32\mskeyprotect.dll
LoadedModule[78]=C:\WINDOWS\SYSTEM32\ncrypt.dll
LoadedModule[79]=C:\WINDOWS\SYSTEM32\NTASN1.dll
LoadedModule[80]=C:\WINDOWS\system32\ncryptsslp.dll
LoadedModule[81]=C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
LoadedModule[82]=C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.Transactions\v4.0_4.0.0.0__b77a5c561934e089\System.Transactions.dll
LoadedModule[83]=C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.Data.OracleClient\v4.0_4.0.0.0__b77a5c561934e089\System.Data.OracleClient.dll
LoadedModule[84]=C:\WINDOWS\SYSTEM32\urlmon.dll
LoadedModule[85]=C:\WINDOWS\SYSTEM32\iertutil.dll
LoadedModule[86]=C:\WINDOWS\SYSTEM32\PROPSYS.dll
LoadedModule[87]=C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Xml.Linq\6e4d9ba028653154945437d7674d20a3\System.Xml.Linq.ni.dll
LoadedModule[88]=C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.EnterpriseServices\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.EnterpriseServices.Wrapper.dll
LoadedModule[89]=C:\WINDOWS\system32\security.dll
LoadedModule[90]=C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Runteb92aa12#\5470ed48a2649b4c1fd9e883daa502b9\System.Runtime.Serialization.ni.dll
OsInfo[0].Key=vermaj
OsInfo[0].Value=10
OsInfo[1].Key=vermin
OsInfo[1].Value=0
OsInfo[2].Key=verbld
OsInfo[2].Value=17134
OsInfo[3].Key=ubr
OsInfo[3].Value=286
OsInfo[4].Key=versp
OsInfo[4].Value=0
OsInfo[5].Key=arch
OsInfo[5].Value=9
OsInfo[6].Key=lcid
OsInfo[6].Value=2057
OsInfo[7].Key=geoid
OsInfo[7].Value=242
OsInfo[8].Key=sku
OsInfo[8].Value=48
OsInfo[9].Key=domain
OsInfo[9].Value=1
OsInfo[10].Key=prodsuite
OsInfo[10].Value=256
OsInfo[11].Key=ntprodtype
OsInfo[11].Value=1
OsInfo[12].Key=platid
OsInfo[12].Value=10
OsInfo[13].Key=sr
OsInfo[13].Value=0
OsInfo[14].Key=tmsi
OsInfo[14].Value=48160
OsInfo[15].Key=osinsty
OsInfo[15].Value=3
OsInfo[16].Key=iever
OsInfo[16].Value=11.285.17134.0-11.0.85
OsInfo[17].Key=portos
OsInfo[17].Value=0
OsInfo[18].Key=ram
OsInfo[18].Value=8144
OsInfo[19].Key=svolsz
OsInfo[19].Value=445
OsInfo[20].Key=wimbt
OsInfo[20].Value=0
OsInfo[21].Key=blddt
OsInfo[21].Value=180410
OsInfo[22].Key=bldtm
OsInfo[22].Value=1804
OsInfo[23].Key=bldbrch
OsInfo[23].Value=rs4_release
OsInfo[24].Key=bldchk
OsInfo[24].Value=0
OsInfo[25].Key=wpvermaj
OsInfo[25].Value=0
OsInfo[26].Key=wpvermin
OsInfo[26].Value=0
OsInfo[27].Key=wpbuildmaj
OsInfo[27].Value=0
OsInfo[28].Key=wpbuildmin
OsInfo[28].Value=0
OsInfo[29].Key=osver
OsInfo[29].Value=10.0.17134.286.amd64fre.rs4_release.180410-1804
OsInfo[30].Key=buildflightid
OsInfo[30].Value=39b802d6-2dc5-4161-973b-28cf09eb3ffb
OsInfo[31].Key=edition
OsInfo[31].Value=Professional
OsInfo[32].Key=ring
OsInfo[33].Key=expid
OsInfo[34].Key=containerid
OsInfo[35].Key=containertype
OsInfo[36].Key=edu
OsInfo[36].Value=0
File[0].CabName=WERInternalMetadata.xml
File[0].Path=WERCF72.tmp.WERInternalMetadata.xml
File[0].Flags=327682
File[0].Type=5
File[0].Original.Path=\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERCF72.tmp.WERInternalMetadata.xml
FriendlyEventName=Stopped working
ConsentKey=CLR20r3
AppName=MyServiceName
AppPath=C:\TFSOnline\Tools\MyServiceName\MyServiceName\bin\Debug\MyServiceName.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=9EE7A7FA2D9AD07D426D34AC6F6F0ACA
MetadataHash=1785215478
The path is mentions throughout is
C:\TFSOnline\Tools\MyServiceName\MyServiceName\bin\Debug\MyServiceName.exe
This is one of the EXEs I deleted. I also deleted the one in obj/Debug.
I've scanned the entire drive, using several methods, and no other instances of MyServiceName.exe exist, so it's not been copied anywhere.
And the one thing I realised I never mentioned: it's a Windows 10 machine
Edit #2:
None of the things the Service does, for example adding Windows Event logs, adding database table rows or creating files, are happening at the moment, so I'm reasonably sure the service isn't running.
I had all that working before starting to look into this issue, and haven't changed any of it since, so if it was running I'd expect to still see them.
To check, I've just re-installed the service as it was before, and I'm getting all the entries I expect.
Edit #3:
Tried deleting it with the SC command:-
C:\WINDOWS\system32>sc delete MyServiceName.exe
[SC] OpenService FAILED 1060:
The specified service does not exist as an installed service.
Solved it!
Nothing to do with the Service at all, just Windows being crap as always.
Turns out that every time Windows was logging a message, it was also churning out a load of old messages that had got stuck in its queue, and logging them as new messages.
I ran Disk Cleanup, and told it to clear out all the old error messages, and it's been running for three days now with no errors at all.

Why do I need any Saga storage?

I'm struggling with an error message I'm getting from the NServiceBus.Host during startup and not sure where I'm going wrong. I initially encountered the problem within a large project but I seem to be able to reproduce this from scratch.
In VS2012, start with a new C# Class Library project, set for .NET 4.6.2. Then add NuGet packages - NServiceBus (6.0.0), NServiceBus.Host (7.0.1) and NServiceBus.NHibernate (7.2.0).
Then edit the EndpointConfiguration class to be as follows:
using NServiceBus.Features;
using NServiceBus.Persistence;
namespace NSB6_Pure
{
using NServiceBus;
public class EndpointConfig : IConfigureThisEndpoint
{
public void Customize(EndpointConfiguration endpointConfiguration)
{
//TODO: NServiceBus provides multiple durable storage options, including SQL Server, RavenDB, and Azure Storage Persistence.
// Refer to the documentation for more details on specific options.
endpointConfiguration.UsePersistence<NHibernatePersistence, StorageType.Timeouts>();
endpointConfiguration.DisableFeature<MessageDrivenSubscriptions>();
endpointConfiguration.DisableFeature<Sagas>();
// NServiceBus will move messages that fail repeatedly to a separate "error" queue. We recommend
// that you start with a shared error queue for all your endpoints for easy integration with ServiceControl.
endpointConfiguration.SendFailedMessagesTo("error");
// NServiceBus will store a copy of each successfully process message in a separate "audit" queue. We recommend
// that you start with a shared audit queue for all your endpoints for easy integration with ServiceControl.
endpointConfiguration.AuditProcessedMessagesTo("audit");
}
}
}
Then compile (in this case a debug build) and, from a command prompt, try to run NServiceBus.Host.exe in the bin\Debug directory. In my case, I get this output, most of it in Red:
2017-04-28 12:46:11.876 INFO DefaultFactory Logging to 'C:\blah\bin\Debug\' with level Info
2017-04-28 12:46:13.446 FATAL NServiceBus.LicenseManager Your license has expire
d! You can renew it at https://particular.net/licensing.
2017-04-28 12:46:13.561 ERROR NServiceBus.GenericHost Exception when starting en
dpoint.
System.InvalidOperationException: In order to use NServiceBus with NHibernate yo
u need to provide at least one connection string. You can do it via (in order of
precedence):
* specifying 'NServiceBus/Persistence/NHibernate/Saga' connection string for th
e Saga persister
* specifying 'NServiceBus/Persistence' connection string that applies to all pe
rsisters
* specifying 'NServiceBus/Persistence/connection.connection_string' or 'NServic
eBus/Persistence/connection.connection_string_name' value in AppSettings or your
NHibernate configuration file.
For most scenarios the 'NServiceBus/Persistence' connection string is the best o
ption.
at NServiceBus.Persistence.NHibernate.NHibernateConfigurationBuilder.Validate
ConfigurationViaConfigFile(Configuration configuration, String configPrefix) in
C:\BuildAgent\work\5135de308b2f3016\src\NServiceBus.NHibernate\Internal\NHiberna
teConfigurationBuilder.cs:line 130
at NServiceBus.Features.NHibernateStorageSession.Setup(FeatureConfigurationCo
ntext context) in C:\BuildAgent\work\5135de308b2f3016\src\NServiceBus.NHibernate
\SynchronizedStorage\NHibernateStorageSession.cs:line 45
at NServiceBus.Features.FeatureActivator.ActivateFeature(FeatureInfo featureI
nfo, List`1 featuresToActivate, IConfigureComponents container, PipelineSettings
pipelineSettings) in C:\Build\src\NServiceBus.Core\Features\FeatureActivator.cs
:line 194
at NServiceBus.Features.FeatureActivator.SetupFeatures(IConfigureComponents c
ontainer, PipelineSettings pipelineSettings) in C:\Build\src\NServiceBus.Core\Fe
atures\FeatureActivator.cs:line 57
at NServiceBus.InitializableEndpoint.<Initialize>d__1.MoveNext() in C:\Build\
src\NServiceBus.Core\InitializableEndpoint.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
at NServiceBus.GenericHost.<Start>d__1.MoveNext() in C:\BuildAgent\work\fc89e
968acb99302\src\NServiceBus.Hosting.Windows\GenericHost.cs:line 48
2017-04-28 12:46:13.582 ERROR NServiceBus.Hosting.Windows.WindowsHost Start fail
ure
System.InvalidOperationException: In order to use NServiceBus with NHibernate yo
u need to provide at least one connection string. You can do it via (in order of
precedence):
* specifying 'NServiceBus/Persistence/NHibernate/Saga' connection string for th
e Saga persister
* specifying 'NServiceBus/Persistence' connection string that applies to all pe
rsisters
* specifying 'NServiceBus/Persistence/connection.connection_string' or 'NServic
eBus/Persistence/connection.connection_string_name' value in AppSettings or your
NHibernate configuration file.
For most scenarios the 'NServiceBus/Persistence' connection string is the best o
ption.
at NServiceBus.Persistence.NHibernate.NHibernateConfigurationBuilder.Validate
ConfigurationViaConfigFile(Configuration configuration, String configPrefix) in
C:\BuildAgent\work\5135de308b2f3016\src\NServiceBus.NHibernate\Internal\NHiberna
teConfigurationBuilder.cs:line 130
at NServiceBus.Features.NHibernateStorageSession.Setup(FeatureConfigurationCo
ntext context) in C:\BuildAgent\work\5135de308b2f3016\src\NServiceBus.NHibernate
\SynchronizedStorage\NHibernateStorageSession.cs:line 45
at NServiceBus.Features.FeatureActivator.ActivateFeature(FeatureInfo featureI
nfo, List`1 featuresToActivate, IConfigureComponents container, PipelineSettings
pipelineSettings) in C:\Build\src\NServiceBus.Core\Features\FeatureActivator.cs
:line 194
at NServiceBus.Features.FeatureActivator.SetupFeatures(IConfigureComponents c
ontainer, PipelineSettings pipelineSettings) in C:\Build\src\NServiceBus.Core\Fe
atures\FeatureActivator.cs:line 57
at NServiceBus.InitializableEndpoint.<Initialize>d__1.MoveNext() in C:\Build\
src\NServiceBus.Core\InitializableEndpoint.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
at NServiceBus.GenericHost.<Start>d__1.MoveNext() in C:\BuildAgent\work\fc89e
968acb99302\src\NServiceBus.Hosting.Windows\GenericHost.cs:line 54
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
at NServiceBus.Hosting.Windows.WindowsHost.Start() in C:\BuildAgent\work\fc89
e968acb99302\src\NServiceBus.Hosting.Windows\WindowsHost.cs:line 33
And the bit I'm focussing on is here:
In order to use NServiceBus with NHibernate you need to provide at least one connection string. You can do it via (in order of precedence):
specifying 'NServiceBus/Persistence/NHibernate/Saga' connection string for the Saga persister
I've checked the NServiceBus code - this isn't a generic message. If the issue was the lack of config for Timeouts1, I'm sure it would be talking about it rather than Saga, since that part of the error message is parameterized.
But I don't want to use Sagas in this particular project. I'd have thought DisableFeature<Sagas> would be sufficient to indicate to NServiceBus that I don't want to use sagas here, but apparently not.
So why is NServiceBus trying to configure Saga storage and, more importantly, how do I stop it from doing so?
1Of course, I am expecting such an error to occur since I haven't done anything such as even adding an app.config to the class library yet. So even once the Saga issue is resolved, I'm expecting this code to still produce errors. Just hopefully ones I can work out how to fix for myself.
(The rationale behind my wanting to avoid configuring any storage is that the real work to do is to upgrade some existing endpoints that have dedicated databases for each feature (Timeouts/Subscriptions/Sagas) and although I've been recommending for a while that we ought to merge these databases, and could thus just configure a NServiceBus/Persistence database for each endpoint, that's not been approved. But I'm loath to add another database for some of these endpoints when they only use timeouts at present)
I'm Dennis van der Stelt, a developer at Particular Software, makers of NServiceBus. I replied on your support case already, but wanted to update this question as well, so that others can find this as well. Evk mentioned most of it already, I just wanted to mention the GH issue and the complete code.
The following code configuration makes sure it doesn't complain about the saga connectionstring.
endpointConfiguration.DisableFeature<NHibernateStorageSession>();
endpointConfiguration.DisableFeature<MessageDrivenSubscriptions>();
endpointConfiguration.DisableFeature<Sagas>();
endpointConfiguration.UsePersistence<NHibernatePersistence, StorageType.Timeouts>();
Then just provide a connectionstring for timeouts
<add name="NServiceBus/Persistence/NHibernate/Timeout" connectionString="server=.\sqlexpress;database=nservicebus; Trusted_Connection=True;" />
That will make it work properly and as expected.
The reason why disabling the NHibernateStorageSession is necessary is because it is a feature inside the persister that is enabled. More information can be found in a GitHub issue I've created based on this case, which can be found here.
I must admit I have not much knowledge about NServiceBus, I just feel obliged to try to help people with high reputation when they ask questions :) So I don't have an explanation why it works like this, only where it fails. In stack trace you can see NHibernateStorageSession.Setup, and NHibernateStorageSession is one of the features. So it fails when trying to activate that feature. Setup starts like this:
protected override void Setup(FeatureConfigurationContext context)
{
NHibernateConfiguration config = new NHibernateConfigurationBuilder(context.Settings, "Saga", new string[1]
{
"StorageConfiguration"
}).Build();
// the rest
}
Where second parameter is connection string suffix. In this case it is hardcoded to "Saga", and as you see - there is no check that Sagas feature is enabled or not. NHibernateConfigurationBuilder then tries to find a connection string for saga (or any more general connection string) and fails to do so with the exception message you observe.
So easy solution is to disable that feature via
endpointConfiguration.DisableFeature<NHibernateStorageSessio‌​n>();
However you should take care of course and ensure functionality you need is not dependent on that feature. It's not entirely clear for me what this feature is for, and I didn't found any documentation for it (though of course its name tells us something about its purpose).
You can do it via (in order of precedence):
It just means the first point would take precedence over the other, if you need to set Saga, but otherwise just do one of the other points:
specifying 'NServiceBus/Persistence' connection string that applies to all persisters
specifying 'NServiceBus/Persistence/connection.connection_string' or 'NServiceBus/Persistence/connection.connection_string_name' value in AppSettings or your NHibernate configuration file.
For most scenarios the 'NServiceBus/Persistence' connection string is the best option.
So you are not required to setup Saga, but it looks like you will not avoid setting a connection string somewhere.

Not able to see the final result after the Reduce function got executed using Windows Azure Storage in MapReduce

I am using c#.net for writing the map and reduce function.I have basically followed the example being given here
Final command
Hadoop jar hadoop-streaming.jar -files "hdfs:///example/apps/map.exe,hdfs:///example/apps/reduce.exe" -input "/example/apps/data.csv" -output "/example/apps/output.txt" -mapper "map.exe" -reducer "reduce.exe"
The Job ran successfully
Now from the Interactive JS mode, if I write
js> #cat /example/apps/output.txt
cat: File does not exist: /example/apps/output.txt
Where as :
js> #ls /example/apps/output.txt
Found 3 items
-rw-r--r-- 3 xxxx supergroup 0 2013-02-22 10:23 /example/apps/output.txt/_SUCCESS
drwxr-xr-x - xxxx supergroup 0 2013-02-22 10:22 /example/apps/output.txt/_logs
-rw-r--r-- 3 xxxx supergroup 0 2013-02-22 10:23 /example/apps/output.txt/part-00000
What is the mistake I am making and how can I see the output?
The -output flag specifies an output folder, not a file. Since there can be multiple reducers, each one will produce a file in this folder.
In this case, you have one reducer, and it produced one file: part-00000. If there were more reducers, they would be named part-00001, part-00002, etc.
The command cat /example/apps/output.txt/part-00000 will display your output. In the future, don't name your output folders something.txt, as that will just confuse you and others :)

Gems with .NET Applications - How do I set up the Executables so they run without error?

I have a gem, roundhouse, which is an application compiled with .NET (C#). Runs on Windows and it should run in a 32 bit process.
To set up my gemspec, I set:
Gem::Specification.new do |s|
s.platform = 'mswin32'
s.name = 'roundhouse'
s.version = version
s.files = Dir['lib/**/*'] + Dir['bin/**/*']
s.bindir = 'bin'
s.executables << 'rh.exe'
When I install the gem, I should be able to type rh.exe from the command line at any path and it should run correctly.
In practice, I'm not seeing this work correctly. This is what I'm getting back:
Window has this for the header: 16 bit MS-DOS Subsystem
C:\WINDOWS\system32\cmd.exe - rh.exe
The NTVDM CPU has encountered an illegal instruction.
CS:xxxx IP:xxxx OP:xx xx xx xx xx Choose 'Close' to terminate the application.
Here is a picture of the issue (link to TwitPic): Error
If I go to the directory where the item was installed, I can run it and it works great. It's just something in the registration of the command to run from anywhere.
I did quite a bit of searching before asking and came up with nothing. It could be that I don't know what I should be searching for. So let me ask the question, is there a way to register an executable with gems for windows executable applications (built with .NET) and have them register properly with the command line? If so, how is that done?
UPDATE:
I found that gems creates a shim in the C:\Ruby\bin directory that points back to the other file. So there is a rh.exe file that is really just a text file. This is its contents:
#!C:/Ruby/bin/ruby.exe
#
# This file was generated by RubyGems.
#
# The application 'roundhouse' is installed as part of a gem, and
# this file is here to facilitate running it.
#
require 'rubygems'
version = ">= 0"
if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
version = $1
ARGV.shift
end
gem 'roundhouse', version
load Gem.bin_path('roundhouse', 'rh.exe', version)
if you're distributing it with the file "rh.exe"
then you'll want to create a file
bin/rh
s.executables << 'bin/rh'
then when it's installed gems will create an "rh.bat" file which runs ruby "bin/rh" essentially (as you've seen).
So within bin/rh put something like
result = system(File.dirname(__FILE__) + "/rh.exe" ARGV.join(' '))
exit 1 unless result
result = system(File.dirname(__FILE__) + "/rh.exe " + ARGV.join(' '))
exit 1 unless result
So the endresult should maybe look like? note the space after 'rh.exe'

Retrieving product information from an unmanaged executing application in C#/.NET

In C#, it is possible to retrieve assembly related information like product name, version etc using reflection:
string productName = Assembly.GetCallingAssembly().GetName().Name;
string versionString = Assembly.GetCallingAssembly().GetName().Version.ToString();
How do I do the equivalent if the executing assembly is written in unmanaged C++ (say)? Is it even possible? Assume that I have a .NET dll which is being invoked in unmanaged code via a COM interface.
edit:
To make things absolutely clear, this is my scenario:
I have an executable written in
unmanaged C++
I have a dll written
in C#/.NET
The dll is invoked by the
executable via a COM interface
Within the .NET dll I want to be
able to retrieve information like
the product name and version of the
calling executable.
Possible?
Walking the stack is not necessary to find out what process you are in. You simply make a single Win32 API call:
HMODULE hEXE = GetModuleHandle(NULL);
According to the documentation for this call:
If this parameter is NULL, GetModuleHandle returns a handle to the file used to create the calling process (.exe file).
You can turn this module handle into a filename with GetModuleFileName(), another standard Win32 API. File name in hand, you can then call GetFileVersionInfo() to retrieve the VS_VERSIONINFO structure for that file. The information you want is in there.
Now since you are in .NET you could use P/Invoke signatures for GetModuleHandle(), GetModuleFileName(). For GetFileVersionInfo() you can use System.Diagnostics.FileVersionInfo.
But actually the easiest way to do it is probably to stick with the System.Diagnostics namespace, everything you need is there. Call System.Diagnostics.Process.GetCurrentProcess() to return a Process object for the process you are running in. Then you can retrieve a ProcessModule from the MainModule property. ProcessModule has a property called FileVersionInfo. The information you want is there.
you could use the following code in VB.Net to retrieve extended document properties:
Sub Main()
Dim arrHeaders(41)
Dim shell As New Shell32.Shell
Dim objFolder As Shell32.Folder
objFolder = shell.NameSpace("C:\tmp\")
For i = 0 To 40
arrHeaders(i) = objFolder.GetDetailsOf(objFolder.Items, i)
Next
For Each strFileName In objfolder.Items
For i = 0 To 40
Console.WriteLine(i & vbTab & arrHeaders(i) & ": " & objFolder.GetDetailsOf(strFileName, i))
Next
Next
End Sub
Add a COM reference to Microsoft Shell Controls and Automation to your project to compile.
The output of the above program will be a list of the meta data assigned to all files in C:\tmp such as
0 Name: dpvoice.dll
1 Size: 208 KB
2 Type: Application Extension
3 Date Modified: 14.04.2008 04:41
4 Date Created: 14.04.2008 04:41
5 Date Accessed: 01.12.2008 09:56
6 Attributes: A
7 Status: Online
8 Owner: Administrators
9 Author:
10 Title:
11 Subject:
12 Category:
13 Pages:
14 Comments:
15 Copyright:
16 Artist:
17 Album Title:
18 Year:
19 Track Number:
20 Genre:
21 Duration:
22 Bit Rate:
23 Protected:
24 Camera Model:
25 Date Picture Taken:
26 Dimensions:
27 :
28 :
29 Episode Name:
30 Program Description:
31 :
32 Audio sample size:
33 Audio sample rate:
34 Channels:
35 Company: Microsoft Corporation
36 Description: Microsoft DirectPlay Voice
37 File Version: 5.3.2600.5512
38 Product Name: Microsoftr Windowsr Operating System
39 Product Version: 5.03.2600.5512
40 Keywords:
Let's assume you're after an EXE/DLL's PE header data that #divo's calls return e.g. Company, Product etc... These btw. are derived from calling Win32 Version Info API's - details up on MSDN:
http://msdn.microsoft.com/en-us/library/ms646981.aspx
The next challenge you face is enumerating the callstack to discover your caller's module context. I've not tried - but if you examine your own callstack, I doubt you'll see the unmanaged caller's frames marshalled into there. Suspect it stops at transitional frame injected before switching into the CCW. Also since it's COM, conceivably the caller could call from out of process - your caller would be a proxy process.
If that fails - you'd need the debugging API's to unwind the external stack - that introduces other constraints:
elevated security permissions required to traverse the stack
potential performance impact unwinding the stack.
On a call-by-call basis either of these could make the debugger approach impractical.
Update
Some research indicates there are plenty of bugs and gotchas for reading the stack above the CCW transitional frame even in the debugger. e.g.
http://support.microsoft.com/kb/317221
Mixed Unmanaged/Managed symbol resolution is pretty ugly - some thoughts here on how to do it... DaveBr's blog on debugging is pretty awesome too.
http://bytes.com/groups/net-vc/280340-stackwalk-callstack-symbol-resolve-managed-unmanaged-code-dbghelp-etc
http://blogs.msdn.com/davbr/archive/2005/10/06/478006.aspx
There is plenty of fodder on the steps taken marshalling calls between unmanaged/managed clients - e.g.
http://msdn.microsoft.com/en-us/library/ms973872.aspx

Categories