We Keep Coding

Cannot use Spark.Net UDFs and HDInsight cluster

I have tried to run a simple application in prod env containing the code from https://github.com/dotnet/spark/blob/master/examples/Microsoft.Spark.CSharp.Examples/Sql/Batch/Basic.cs
The applications runs fine and emits output to stdout until it this code crashes when it hits the first UDF.
Thanks for any insights you can share on this.
Env.
Code is packaged using
dotnet publish -c Release -f netcoreapp3.1 -r ubuntu.16.04-x64
HDInsight cluster HDI 4.0, Spark 2.4
-- Server is setup using the guidelines in https://learn.microsoft.com/en-us/dotnet/spark/tutorials/hdinsight-deployment
spark-submit --master yarn --conf spark.yarn.appMasterEnv.DOTNET_ASSEMBLY_SEARCH_PATHS="./app/publish.zip" --archives wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/publish.zip#mySparkApp --class org.apache.spark.deploy.dotnet.DotnetRunner wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/microsoft-spark-2.4.x-0.12.1.jar wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/publish.zip mySparkApp
(and all sorts of variations on this in desperation, --deploy-mode cluster, various paths, etc, etc. nothing works)
stdout:
...
+---+-----+
|age| name|
+---+-----+
| 22|Ricky|
| 36| Jeff|
| 62|Geddy|
+---+-----+
[2020-10-28T09:15:10.1478641Z] [wn0-hdinsi] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'showString' failed for class '41' when called with 3 arguments ([Index=1, Type=Int32, Value=20], [Index=2, Type=Int32, Value=20], [Index=3, Type=Boolean, Value=False], )
[2020-10-28T09:15:10.1480587Z] [wn0-hdinsi] [Error] [JvmBridge] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 16.0 failed 4 times, most recent failure: Lost task 0.3 in stage 16.0 (TID 210, wn0-hdinsi.xwccrqijnmqujdjghwrza0nzbb.fx.internal.cloudapp.net, executor 2): org.apache.spark.api.python.PythonException: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Spark.Utils.UdfSerDe.<>c.b__10_0(TypeData td) in //src/csharp/Microsoft.Spark/Utils/UdfSerDe.cs:line 262
at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
at Microsoft.Spark.Utils.UdfSerDe.DeserializeType(TypeData typeData) in //src/csharp/Microsoft.Spark/Utils/UdfSerDe.cs:line 258
at Microsoft.Spark.Utils.UdfSerDe.Deserialize(UdfData udfData) in //src/csharp/Microsoft.Spark/Utils/UdfSerDe.cs:line 160
at Microsoft.Spark.Utils.CommandSerDe.DeserializeUdfs[T](UdfWrapperData data, Int32& nodeIndex, Int32& udfIndex) in //src/csharp/Microsoft.Spark/Utils/CommandSerDe.cs:line 333
at Microsoft.Spark.Utils.CommandSerDe.Deserialize[T](Stream stream, SerializedMode& serializerMode, SerializedMode& deserializerMode, String& runMode) in /_/src/csharp/Microsoft.Spark/Utils/CommandSerDe.cs:line 306
at Microsoft.Spark.Worker.Processor.CommandProcessor.ReadSqlCommands(PythonEvalType evalType, Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 188
at Microsoft.Spark.Worker.Processor.CommandProcessor.ReadSqlCommands(PythonEvalType evalType, Stream stream, Version version) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 98
at Microsoft.Spark.Worker.Processor.CommandProcessor.Process(Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 43
at Microsoft.Spark.Worker.Processor.PayloadProcessor.Process(Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\PayloadProcessor.cs:line 82
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 143
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:456)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:410)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
------cut--------
My problem did indeed turn out to be an issue with paths. For anyone else having the same problem, I got this to work by having the dll with the UDF (can be the same dll as the general spark app) has to be listed in "--files". So essentially you need a zip file with assemblies AND then have links to the dlls directly. There is probably a smarter way, but that did it for me (when running on cluster mode):
spark-submit --deploy-mode cluster --master yarn --files wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/mySparkApp.dll --class org.apache.spark.deploy.dotnet.DotnetRunner wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/microsoft-spark-2.4.x-0.12.1.jar wasbs://xxx#yyy.blob.core.windows.net/SparkJobs/publish.zip mySparkApp

The error is because the dll with your code can’t be found.
Two things, firstly in yarn mode . at the beginning of the DOTNET_ASSEMBLY_SEARCH_PATHS causes the users home directory to be prepended to the path so it isn’t currentdirectory/app/publish.zip so if that is different then it will be looking in the wrong place.
Secondly make sure the publish.zip doesn’t contain folders and the dll with the udf is at the top level of the zip.
Instead of putting the zip inside the app folder I would just use the current folder and don’t worry about DOTNET_ASSEMBLY_SEARCH_PATHS
For a walkthrough make sure you follow:
https://learn.microsoft.com/en-us/dotnet/spark/tutorials/hdinsight-deployment

Why do I get an Xml Validation error when running my application on one server, but not on my other server's systems?

I've deployed my ASP.Net C# application to three other systems without problem. I haven't changed the application code, or even the installer. I have also gone through and made sure that my IIS 10 settings and connections are all set up as close to the other environments as possible. One of the systems/environments that the application runs successfully on is running Windows Server 2008 alongside IIS 7, while the other two are running Windows 10 and IIS 10. The system the application fails on is running Windows Server 2019 and IIS 10. What I don't understand is why I would be getting this error, when I haven't changed anything other than the system and server my application is running on:
XmlValidator::ValidateXml() - Error Validating xml - System.Exception: Validation error: The element 'systemBalances' in namespace 'http://www.alpha.com/system/schemas/v1.0/' has invalid child element 'accessible' in namespace 'http://www.alpha.com/system/schemas/v1.0/'.
at ABC.Common.XMLValidators.XmlValidator.ValidationHandler(Object sender, ValidationEventArgs args) in C:\XYZ Src\Dev\ABC\ABC[Dev]\AlphaWebServices\ABC.Common\XmlValidator.cs:line 134
at System.Xml.Schema.XmlSchemaValidator.SendValidationEvent(ValidationEventHandler eventHandler, Object sender, XmlSchemaValidationException e, XmlSeverityType severity)
at System.Xml.Schema.XmlSchemaValidator.ElementValidationError(XmlQualifiedName name, ValidationState context, ValidationEventHandler eventHandler, Object sender, String sourceUri, Int32 lineNo, Int32 linePos, XmlSchemaSet schemaSet)
at System.Xml.Schema.XmlSchemaValidator.ValidateElementContext(XmlQualifiedName elementName, Boolean& invalidElementInContext)
at System.Xml.Schema.XmlSchemaValidator.ValidateElement(String localName, String namespaceUri, XmlSchemaInfo schemaInfo, String xsiType, String xsiNil, String xsiSchemaLocation, String xsiNoNamespaceSchemaLocation)
at System.Xml.XsdValidatingReader.ProcessElementEvent()
at System.Xml.XsdValidatingReader.ProcessReaderEvent()
at System.Xml.XsdValidatingReader.Read()
at ABC.Common.XMLValidators.XmlValidator.ValidateXml(String sXmlToValidate, JKLClass AlphaClass, Int32 hostId, String egmId, String& outXmlMsg) in C:\BMS Src\Dev\ABC\ABC[Dev]\AlphaWebServices\ABC.Common\XmlValidator.cs:line 34
I'm not sure if it matters but the three other systems that my application is running on successfully are all hosted on the same server while the one that errors is on a server separate from the initial three environments.

Invalid handle error when calling MQQueueManager constructor in .NET C#

I've got a seemingly random "The handle is invalid" error when calling the following piece of code :
var properties = new Hashtable();
properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
properties.Add(MQC.HOST_NAME_PROPERTY, "MyHost");
properties.Add(MQC.PORT_PROPERTY, "MyPort");
properties.Add(MQC.CHANNEL_PROPERTY, "MyChannel");
this.QueueManager = new MQQueueManager("MyName", properties);
The environment is .NET Framework 3.5 & MQ Client 7.1
The problem is that this error doesn't occur repeatedly, maybe just 15% of the time, only in the production environment, and I can't seem to reproduce it within my development environment. The only trail I've got is that I know than it is possible for this code to be called simultaneously by two different processes. If it helps, I've also got the following stack trace :
Win32Exception - The handle is invalid
at System.Diagnostics.NtProcessManager.GetModuleInfos(Int32 processId, Boolean firstModuleOnly)
at System.Diagnostics.ProcessManager.GetModuleInfos(Int32 processId)
at System.Diagnostics.Process.get_Modules()
at IBM.WMQ.Nmqi.UnmanagedNmqiMQ.MQCONNX(String pQMgrName, MQCNO& pConnectOpts, Hconn parentHconn, Phconn phconn, Int32& pCompCode, Int32& pReason)
at IBM.WMQ.Nmqi.UnmanagedNmqiMQ.MQCONNX(String pQMgrName, MQConnectOptions pConnectOpts, Phconn phconn, Int32& pCompCode, Int32& pReason)
at IBM.WMQ.MQQueueManager.Connect(String queueManagerName)
at IBM.WMQ.MQQueueManager..ctor(String queueManagerName, Hashtable properties)
...
Any ideas ? Anyone ? :)

This fix might be of help to you. Have you downloaded the latest MQ Client for v7.1?

Set write permissions for all users for my program folder

I have build a program in Visual Studio. The program creates a logfile and writes into it while the program is running. Therefore I constructed an installer (setup-project), that should set write permissions for my program-folder regardless which user works with the program.
currently it looks like this:
// ...
}
InitializeComponent();
string folder = Directory.GetCurrentDirectory();
DirectorySecurity ds = Directory.GetAccessControl(folder);
ds.AddAccessRule(new FileSystemAccessRule("Everyone", //Everyone is important
//because rights for all users!
FileSystemRights.Read | FileSystemRights.Write, AccessControlType.Allow));
}
// ...
In the last two rows I get a System.SystemException: “Die Vertrauensstellung zwischen der primären Domäne und der vertrauenswürdigen Domäne konnte nicht hergestellt werden.“
[Translation: "The trust relationship between the primary domain and the trusted domain could not be established."]
The stacktrace reads like this:
bei System.Security.Principal.NTAccount.TranslateToSids(IdentityReferenceCollection sourceAccounts, Boolean& someFailed)
bei System.Security.Principal.NTAccount.Translate(IdentityReferenceCollection sourceAccounts, Type targetType, Boolean& someFailed)
bei System.Security.Principal.NTAccount.Translate(IdentityReferenceCollection sourceAccounts, Type targetType, Boolean forceSuccess)
bei System.Security.Principal.NTAccount.Translate(Type targetType)
bei System.Security.AccessControl.CommonObjectSecurity.ModifyAccess(AccessControlModification modification, AccessRule rule, Boolean& modified)
bei System.Security.AccessControl.CommonObjectSecurity.AddAccessRule(AccessRule rule)
bei System.Security.AccessControl.FileSystemSecurity.AddAccessRule(FileSystemAccessRule rule)
Have you an idea what I can do?
thanks

Perhaps the best answer isn't what you've asked for. There's a good reason for not writing to the program files directory. Log data in particular is transient and shouldn't be written here.
It's a much better idea to write log data to the directory specified by the TEMP environment variable. If you do this you'll save your users a few troubles and prevent them cursing your software in the future. Please check out this answer which covers the same topic:
Allow access permission to write in Program Files of Windows 7

Aren't you missing the statement where you actually set the access control back to the directory?
Directory.SetAccessControl(Directory.GetCurrentDirectory(), ds);

This previously asked question should point you in the right direction. Basically, you do NOT want any user writing to the Program Files folder. UAC, security and other measures are there to try and prevent this as much as possible.
Essentially, if you want a single file which will be written to by all users, you will want it in the ProgramData folder, accessible through the %ALLUSERSPROFILE%, rather than the individual users' temporary folder, which is definitely what you want to do with a log file. Remember that the temporary folder's content should be considered volatile, and could be deleted at any time, such as by the Disk Cleanup Wizard.

Invalid Month in Date - Date format in Informix reverts sporadically

So we have an AIX (ugh) server that runs an ERP system. This system's built in reports suck so I am of course tasked with building the reports we need.
I've got many reports I run and most of course are for specific dates, etc. The coding is done in C# running in ASP.net on a Windows 2003 Standard Server box. It uses the Informix CSDK to connect via the .Net Data Adapter that comes with the CSDK. The server runs Informix 10 on the AIX 5.2 server.
What's weird is that anytime we start and stop Informix or reboot the server, etc, it seems that Informix decides to change the way it's handled the date via the CSDK. If it's currently expecting MM/DD/YYYY then it'll eventually decide after the above situation that is wants it in YYYY/MM/DD. This usually ends up giving me an "Invalid Month in Date" error. Then I go into my dateformat function (made to easily allow me to fall back and forth) and manually change it over. A couple reports I built in the handling of this error and then retry the same query with the other format of the date. This of course is less than ideal and I'd like to get to the bottom of it!
Here is some pasted text from the ASP.net page error. Thanks!
Server Error in '/' Application.
ERROR [HY000] [Informix .NET provider][Informix]Invalid month in date
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about theerror and where it originated in the code.
Exception Details: IBM.Data.Informix.IfxException: ERROR [HY000] [Informix .NET provider][Informix]Invalid month in date
Source Error:
Line 479:
Line 480: //aUsage = new IfxDataAdapter(sSelect_Usage, conn);
Line 481: aUsage.Fill(dsUsage, "Usage");
Line 482: aUsage.Dispose();
Line 483: dtUsage = dsUsage.Tables["Usage"];
Source File: D:\Inetpub\reports2.oscarwinski.com\App_Code\IMRShipClass.cs Line: 481
Stack Trace:
[IfxException: ERROR [HY000] [Informix .NET provider][Informix]Invalid month in date]
IBM.Data.Informix.IfxConnection.HandleError(IntPtr hHandle, SQL_HANDLE hType, RETCODE retcode) +26
IBM.Data.Informix.IfxCommand.ExecuteReaderObject(CommandBehavior behavior, String method) +654
IBM.Data.Informix.IfxCommand.ExecuteReader(CommandBehavior behavior) +117
IBM.Data.Informix.IfxCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior) +4
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +130
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +287
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, String srcTable) +92
IMRShipClass.Generate() in D:\Inetpub\reports2.oscarwinski.com\App_Code\IMRShipClass.cs:481
IMRShip.testIMR() in D:\Inetpub\reports2.oscarwinski.com\IMRShip.aspx.cs:114
IMRShip.btnExport2Excel_Click1(Object sender, EventArgs e) in D:\Inetpub\reports2.oscarwinski.com\IMRShip.aspx.cs:259
System.Web.UI.WebControls.Button.OnClick(EventArgs e) +111
System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +110
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +10
System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +13
System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +36
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1565
Version Information: Microsoft .NET Framework Version:2.0.50727.3082; ASP.NET Version:2.0.50727.3082

The date format is controlled by the client, not the server. Obviously it has a default (US format), but each client can dictate the format it wishes to use. This is done by setting the DBDATE environment variable. (There is also GL_DATE if you're using locales.)
If the date format has switched, you should check the value of this environment variable. Is it being set by your middleware in some circumstances?

Urgh! .NET is not my strong point - you may have to adapt what I say to work correctly.
On Windows, for better or worse, there is an Informix utility called SETNET32 which can be used to set Informix-related environment variables for Windows programs that connect to Informix databases. You don't mention whether you've set anything using this. Investigate whether it used by the Informix .NET provider.
You also do not mention whether the IDS server instance on the AIX box is restarted when the problem appears. I assume not. (The analysis might be different if it did.)
The main environment variable that controls the date format is DBDATE. It takes various notations, such as:
DBDATE=dmy4/ # 30/07/2009
DBDATE=mdy4/ # 07/30/2009
DBDATE=y4md- # 2009-07-30
DBDATE=mdy20 # 073009
Don't use the last one. If there are changes in the value of DBDATE, this could account some of your problems, and setting DBDATE would probably fix your problems.
There are other variables that affect date interpretation if DBDATE is not set; these include CLIENT_LOCALE and DB_LOCALE, and even GL_DATE. However, DBDATE has the highest priority and is the one most people set most usually.

This is probably unrelated, but I was receiving the same error. I ended up changing the single quotes in my SQL to parentheses. I was using parameterized SQL.
SELECT id, descriptor FROM Foo WHERE descriptor >= '?'
becomes
SELECT id, descriptor FROM Foo WHERE descriptor >= (?)

This may not be the case for your particular setup, but I have seen servers change the date format reported to databases when administrators have had their locale set to something differnt to the default, but only while they are logged in!
You may wish to check the country/locale settings for EVERY user on the system.

DB2/Informix overrides the client date format. You contact your Db2 admin and set the date format in the server side once for all. We faced exactly same problem in AS400 (DB2) and DB admin never accepted it as in your case. Once the ISO date format syntax was taken from IBM site and given to DB2 admin he changed it. After that our programs worked fine with dates.
Good luck.

Set DBDATE=MDY4/ in the client app that runs the reports and every other client as well.

Start the informix instance as root instead of informix user .... it solves my pb and my night lol