NullPointerException with MR2 in windows - c#

I have installed Hadoop 2.3.0 in windows and able to execute MR jobs successfully. But while trying with streaming sample in C# [with HadoopSDK's .Net assemblies] the app ends with the following exception
14/05/16 18:21:06 INFO mapreduce.Job: Task Id : attempt_1400239892040_0003_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
at org.apache.hadoop.mapred.Task.getFsStatistics(Task.java:347)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:478)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Update:
I was able to drill down the problem and found that the exception raised in the following line
matchedStats = getFsStatistics(FileOutputFormat.getOutputPath(job), job);
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:478)
In the above, the result of 'FileOutputFormat.getOutputPath(job)' returns null, which throws the null pointer exception. Below are the codes for the getOutputPath() function.
public static final String OUTDIR = "mapreduce.output.fileoutputformat.outputdir";
public static Path getOutputPath(JobConf conf) {
String name = conf.get(org.apache.hadoop.mapreduce.lib.output.
FileOutputFormat.OUTDIR);
return name == null ? null: new Path(name);
}
So Is it needed to set value for property "mapreduce.output.fileoutputformat.outputdir" anywhere in the configuration files to get this issue resolved?
Thanks

The problem is that the hadoop services have been started from a different user [SYSTEM in my case] and the mapreduce sample was submitted from my local user. So this makes the issue by returning the FileSystem statistics [for the local user] as Null.
Once i started Hadoop from my local user, the above issue gets resolved.

Related

Firebase AppCheck token always null

I'm having this issue since one week on a Xamarin Android application.
I get the token with this method:
public async void GetToken()
{
var AppCheckToken = await AppCheck.GetToken(false);
if (AppCheckToken is AppCheckTokenResult Tok)
{
string r;
r = Tok.Token;
Preferences.Set("AppCheckToken", r);
}
}
that always return this string eyJlcnJvciI6IlVOS05PV05fRVJST1IifQ== on both debug and physical (production) mode. I also tried to make internal tests in Play Console and then download the app from there, but result is always the same.
Here's how I initialize AppCheck:
auth = FirebaseAuth.GetInstance(app);
PlayIntegrityAppCheckProviderFactory f = new PlayIntegrityAppCheckProviderFactory();
f.Create(app);
AppCheck = FirebaseAppCheck.GetInstance(app);
AppCheck.InstallAppCheckProviderFactory(f);
AppCheck.SetTokenAutoRefreshEnabled(true);
I noticed also that AppCheckToken variable declared in GetToken() method is always null and returns this FirebaseException:
--- End of managed Firebase.FirebaseException stack trace --- com.google.firebase.FirebaseException: null at
com.google.firebase.appcheck.internal.DefaultFirebaseAppCheck.lambda$getToken$1(DefaultFirebaseAppCheck.java:205)
at
com.google.firebase.appcheck.internal.DefaultFirebaseAppCheck$$ExternalSyntheticLambda1.then(Unknown
Source:0) at
com.google.android.gms.tasks.zze.run(com.google.android.gms:play-services-tasks##18.0.2:1)
at android.os.Handler.handleCallback(Handler.java:942) at
android.os.Handler.dispatchMessage(Handler.java:99) at
android.os.Looper.loopOnce(Looper.java:201) at
android.os.Looper.loop(Looper.java:288) at
android.app.ActivityThread.main(ActivityThread.java:7892) at
java.lang.reflect.Method.invoke(Native Method) at
com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936) Caused
by: java.lang.NumberFormatException: null at
java.lang.Long.parseLong(Long.java:699) at
java.lang.Long.parseLong(Long.java:861) at
com.google.firebase.appcheck.playintegrity.internal.PlayIntegrityAppCheckProvider.lambda$getPlayIntegrityAttestation$4$com-google-firebase-appcheck-playintegrity-internal-PlayIntegrityAppCheckProvider(PlayIntegrityAppCheckProvider.java:105) at
com.google.firebase.appcheck.playintegrity.internal.PlayIntegrityAppCheckProvider$$ExternalSyntheticLambda1.then(Unknown
Source:4) at
com.google.android.gms.tasks.zzo.run(com.google.android.gms:play-services-tasks##18.0.2:1)
... 8 more
I tried also to generate a debug token from Firebase Console to use in the app in debug mode but when I make the call to Firebase Storage (that is already enforced) it keeps saying "AppCheck Token is invalid".
UPDATE:
I've tried also this approach: FirebaseAppCheck AppCheck = FirebaseAppCheck.Instance; AppCheck.InstallAppCheckProviderFactory(PlayIntegrityProviderFactory.Instance);
and for just one time FirebaseException was saying something like "PlayStore is not uptodate in your device". So I've updated PlayStore and now it throws the same error described above. I've tried to get the token with an AddOnSuccessListener(this) but result is the same.
Also I have to tell that var AppCheckToken declared before the if statement becomes always an object of class DefaultFirebaseAppCheckTokenResult.
I don't know what else to do...
IMHO there's a problem with declaration of objects (that are nulls), but since there aren't documents on how to use AppCheck on C# I'm screwed.

"Runtime error Exception has been thrown by the target of an invocation" from Script task

I have a SSIS package with a script task, I get the following error when i try to run it in my local system. It works fine for my collegues as well as in production. However, I am not able to run it locally, to test. I keep a debug point in the main method, but it is never reached, I get the error before it goes to main method.
I am using VS 2010, .Net framework 4.5.
The script task does compile. I get the following messages SSIS package "..\Test.dtsx" starting. Error: 0x1 at Test: Exception has been thrown by the target of an invocation. Task failed: Test SSIS package "..\Test.dtsx" finished: Success. The program '[2552] DtsDebugHost.exe: DTS' has exited with code 0 (0x0).
The following is the code:
public void Main()
{
try
{
LogMessages("Update Bug package execution started at :: " + DateTime.Now.ToLongTimeString());
LogMessages("Loading package configuration values to local variables.");
strDBConn = Dts.Variables["User::DBConnection"] != null ? Dts.Variables["User::DBConnection"].Value.ToString() : string.Empty;
strTPCUrl = Dts.Variables["User::TPCUrl"] != null ? Dts.Variables["User::TPCUrl"].Value.ToString() : string.Empty;
TfsTeamProjectCollection objTPC = new TfsTeamProjectCollection(new Uri(strTPCUrl));
WorkItemStore objWIS = new WorkItemStore(objTPC);
WorkItemCollection objWIC = objWIS.Query("SELECT...");
foreach (WorkItem wi in objWIC)
{
}
}
catch(Exception ex)
{
}
When I commented the code from TfsTeamProjectCollection objTPC = new TfsTeamProjectCollection(new Uri(strTPCUrl)); The script executes successfully. However, if i keep TfsTeamProjectCollection objTPC = new TfsTeamProjectCollection(new Uri(strTPCUrl)); and comment the rest, i get the exception.
I do have access to the URL.
I am using Microsoft.TeamFoundation.Client.dll and Microsoft.TeamFoundation.WorkItemTracking.Client.dll, in my script task. However the dll version in the package is 10.0, and the version of the dll in my GAC is 12.0. Would that cause a problem?
I had the same Problem (i.e. the same error code Error: 0x1 ...).
The issue was with some of the libraries referenced from a missing folder.
Removing the references and adding them back from the correct path fixed the issue.
The Microsoft Reference (https://msdn.microsoft.com/en-us/library/ms345164.aspx) related to the Error code is very generic and doesn't help you much.
However, reading other articles it is quite likely it indicates an unknown failure reason to run the Script Task.
Hexadecimal code: 0x1
Decimal Code: 1
Symbolic Name: DTS_MSG_CATEGORY_SERVICE_CONTROL
Description: Incorrect function.
I got this error message when I referred to a passed ssis variable in Dts.Variables["User::xxxx].Value(); where xxxx did not exist and was not passed from the calling program. It was a simple Console.Writeline referring to a passed variable that didn't exist.
I fixed this error by changing the TargetServerVersion of the SSIS Project.
Integration Services Project Property Pages
This is just a different situation and not intended to be the end all be all solution for everyone.
When I was installing my DLLs into the GAC I forgot to run my script as Administrator and the script ran silently without error as though it was working.
I felt really dumb when I realized that's what I did wrong. Hopefully this can help prevent other people from wasting time on something so silly.
For reference this is what I use for installing my DLLs into the GAC and I modified it to tell me when I am not running it as Administrator now:
#https://superuser.com/questions/749243/detect-if-powershell-is-running-as-administrator
$isAdmin = ([Security.Principal.WindowsPrincipal] `
[Security.Principal.WindowsIdentity]::GetCurrent() `
).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
if($isAdmin -eq $false)
{
Write-Host "You have to run this script as Administrator or it just won't work!" -ForegroundColor Red
return;
}
$strDllPath = "C:\PathToYourDllsHere\"
#Note that you should be running PowerShell as an Administrator
[System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")
$publish = New-Object System.EnterpriseServices.Internal.Publish
$arr = #(
"YourDLL01.dll",
"YourDLL02.dll",
"YourDLL03.dll"
)
get-date
foreach($d in $arr)
{
$p = ($strDllPath + $d);
$p
$publish.GacInstall($p);
}
#If installing into the GAC on a server hosting web applications in IIS, you need to restart IIS for the applications to pick up the change.
#Uncomment the next line if necessary...
#iisreset
Credit for how to determine if your PowerShell script is running in Admin mode or not:
https://superuser.com/questions/749243/detect-if-powershell-is-running-as-administrator
In my case it was missing DLLs or not having the correct version installed on the server.
Locally all tests were fine but on the server the error message Runtime error Exception has been thrown by the target of an invocation kept popping up.
No exception handler would be able to catch that error - the code in the script task would not even be executed, as soon as a DLL would be needed, that is not in the assembly cache on the server (it could happen on a local machine as well, with the same error).
The difficulty here is finding out what is missing and then either update the references to the correct version or install the missing DLL in the assembly cache with gacutil. The way I approached the debugging was to remove parts of the code in the script task until that error wouldn't appear, then analyze the missing part for references.
After not having any luck with the other answers here, I finally found that in my package, the problem was that I had created a new variable but not carried its name across into my new copy of the C# script. The variables were the ones to be used as connection string expressions. So it was ultimately a matter of changing:
Dts.Variables["Exists"].Value = File.Exists(Dts.Variables["OldSSISPackageVariableName"].Value.ToString());
to:
Dts.Variables["Exists"].Value = File.Exists(Dts.Variables["NewSSISPackageVariableName"].Value.ToString());
Once I kept them in sync, it worked fine.
Its fixed, when added reference to dll version 12.0.0 and changed Target Framework to .Net Framework 4.5
My problem was that I, in the script task, tried to fetch data like this:
public void Main()
{
using (var connection = Dts.Connections["localhost.Test"].AcquireConnection(Dts.Transaction) as SqlConnection)
{
connection.Open();
var command = new SqlCommand("select * from Table;", connection);
var reader = command.ExecuteReader();
while (reader.Read())
{
MessageBox.Show($"{reader[0]} {reader[1]} {reader[2]}");
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
However, my connection is of the type OLE DB, and therefore I needed to connect to it like this instead:
public void Main()
{
var connectionString = Dts.Connections["localhost.Test"].ConnectionString;
using (var connection = new OleDbConnection(connectionString))
{
connection.Open();
var command = new OleDbCommand("select * Table;", connection);
var reader = command.ExecuteReader();
while (reader.Read())
{
MessageBox.Show($"{reader[0]} {reader[1]} {reader[2]}");
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
Notice that I'm using OleDbConnection here.
For me what fixed the issue was updating the string parameter I passed to the script. it was missing "" at the end of the path (i.e. "e:\arcive" - needed to add "" at the end)

Azman gives same error using either local xml or SQL Server for storage

Recently two users in our system started getting this error when trying to add them to a role.
System.Runtime.InteropServices.COMException: Cannot create a file when that file already exists. (Exception from HRESULT: 0x800700B7)
What is interesting is that the same error occurs regardless of the configuration, running locally we use an XML store and in the test environment it uses SQL Server.
Here is code where is blows up - AddMemberName() - as you can see this is pretty straightforward stuff and it's worked well for a while, it's just these two users all of the sudden
public void AddUserToRole(string roleName, string userName, bool upn)
{
string uName = userName;
if (upn)
uName = getAltUserNames(userName).First();
AzAuthorizationStore store = new AzAuthorizationStoreClass();
store.Initialize(2, _provider.StoreLocation, null);
IAzApplication app = store.OpenApplication(_provider.ApplicationName, null);
IAzRole role = app.OpenRole(roleName, null);
role.AddMemberName(uName, null);
role.Submit(0, null);
Marshal.FinalReleaseComObject(role);
}
I tried googling various different terms but can't find much of anything regarding this. I did read this post but it doesn't seem to be the issue.
Thanks
Check you Active Directory usernames and the underlying OU name especially. Check for duplicates and mismatches.
I had an issue once where a user got married and her name changed.

ActiveDirectory error 0x8000500c when traversing properties

I got the following snippet (SomeName/SomeDomain contains real values in my code)
var entry = new DirectoryEntry("LDAP://CN=SomeName,OU=All Groups,dc=SomeDomain,dc=com");
foreach (object property in entry.Properties)
{
Console.WriteLine(property);
}
It prints OK for the first 21 properties, but then fail with:
COMException {"Unknown error (0x8000500c)"}
at System.DirectoryServices.PropertyValueCollection.PopulateList()
at System.DirectoryServices.PropertyValueCollection..ctor(DirectoryEntry entry, String propertyName)
at System.DirectoryServices.PropertyCollection.PropertyEnumerator.get_Entry()
at System.DirectoryServices.PropertyCollection.PropertyEnumerator.get_Current()
at ActiveDirectory.Tests.IntegrationTests.ObjectFactoryTests.TestMethod1() in MyTests.cs:line 22
Why? How can I prevent it?
Update
It's a custom attribute that fails.
I've tried to use entry.RefreshCache() and entry.RefreshCache(new[]{"theAttributeName"}) before enumerating the properties (which didn't help).
Update2
entry.InvokeGet("theAttributeName") works (and without RefreshCache).
Can someone explain why?
Update3
It works if I supply the FQDN to the item: LDAP://srv00014.ssab.com/CN=SomeName,xxxx
Bounty
I'm looking for an answer which addresses the following:
Why entry.Properties["customAttributeName"] fails with the mentioned exception
Why entry.InvokeGet("customAttributeName") works
The cause of the exception
How to get both working
If one wants to access a custom attribute from a machine that is not
part of the domain where the custom attribute resides (the credentials
of the logged in user don't matter) one needs to pass the fully
qualified name of the object is trying to access otherwise the schema
cache on the client machine is not properly refreshed, nevermind all
the schema.refresh() calls you make
Found here. This sounds like your problem, given the updates made to the question.
Using the Err.exe tool here
http://www.microsoft.com/download/en/details.aspx?id=985
It spits out:
for hex 0x8000500c / decimal -2147463156 :
E_ADS_CANT_CONVERT_DATATYPE adserr.h
The directory datatype cannot be converted to/from a native
DS datatype
1 matches found for "0x8000500c"
Googled "The directory datatype cannot be converted to/from a native" and found this KB:
http://support.microsoft.com/kb/907462
I have the same failure. I´m read and saw a lot of questions about the error 0x8000500c by listing attribute from a DirectoryEntry.
I could see, with the Process Monitor (Sysinternals), that my process has read a schema file. This schema file is saved under
C:\Users\xxxx\AppData\Local\Microsoft\Windows\SchCache\xyz.sch.
Remove this file and the program works fine :)
I just encountered the issue and mine was with a web application.
I had this bit of code which pulls the user out of windows authentication in IIS and pulls their info from AD.
using (var context = new PrincipalContext(ContextType.Domain))
{
var name = UserPrincipal.Current.DisplayName;
var principal = UserPrincipal.FindByIdentity(context, this.user.Identity.Name);
if (principal != null)
{
this.fullName = principal.GivenName + " " + principal.Surname;
}
else
{
this.fullName = string.Empty;
}
}
This worked fine in my tests, but when I published the website it would come up with this error on FindByIdentity call.
I fixed the issue by using correct user for the app-pool of the website. As soon as I fixed that, this started working.
I had the same problem with a custom attribute of a weird data type. I had a utility program that would extract the value, but some more structured code in a service that would not.
The utility was working directly with a SearchResult object, while the service was using a DirectoryEntry.
It distilled out to this.
SearchResult result;
result.Properties[customProp]; // might work for you
result.Properties[customProp][0]; // works for me. see below
using (DirectoryEntry entry = result.GetDirectoryEntry())
{
entry.Properties[customProp]; // fails
entry.InvokeGet(customProp); // fails as well for the weird data
}
My gut feel is that the SearchResult is a little less of an enforcer and returns back whatever it has.
When this is converted to a DirectoryEntry, this code munges the weird data type so that even InvokeGet fails.
My actual extraction code with the extra [0] looks like:
byte[] bytes = (byte[])((result.Properties[customProp][0]));
String customValue = System.Text.Encoding.UTF8.GetString(bytes);
I picked up the second line from another posting on the site.

How to set CCSID for MQQueueManager from code?

I have some strange problem. I think I followed documentation correctly but my code doesn't work. I have this very simple hard coded test (NUnit):
[TestFixture]
public class MQQueueTests {
public const string MessageContent = "<test>This is test message</test>";
public static void Main(string[] args) {
var tests = new MQQueueTests();
tests.PutAndGetMessage();
}
[Test]
public void PutAndGetMessage() {
var properties = new Hashtable
{
{MQC.HOST_NAME_PROPERTY, "TestServer"},
{MQC.CHANNEL_PROPERTY, "Test.Channel"},
{MQC.PORT_PROPERTY, 1415},
// Is this correct? It looks like it is not
// enough because adding this line didn't solve
// the problem.
{MQC.CCSID_PROPERTY, 437}
};
using (var manager = new MQQueueManager("Test.Queue.Manager", properties)) {
using (MQQueue queue = manager.AccessQueue("Test.Queue",
MQC.MQOO_OUTPUT | MQC.MQOO_INPUT_AS_Q_DEF)) {
MQMessage message = new MQMessage();
message.WriteUTF(MessageContent);
queue.Put(message);
MQMessage readMessage = new MQMessage();
queue.Get(readMessage);
Assert.AreEqual(MessageContent, readMessage.ReadUTF());
queue.Close();
}
manager.Disconnect();
}
}
}
I'm running the test application either from console or through Resharper 6 test runner. If I run the application in test runner I always get following exception:
IBM.WMQ.MQException : MQRC_CHANNEL_CONFIG_ERROR (reason code is 2539)
The exception is thrown by MQQueueManager.Connect (called by its constructor).
If I check MQ logs I see:
AMQ9541: CCSID supplied for data conversion not supported.
EXPLANATION: The program ended because, either the source CCSID '437'
or the target CCSID '852' is not valid, or is not currently supported.
ACTION: Correct the CCSID that is not valid, or ensure that the
requested CCSID can be supported.
If I run the application from the console I got the same error but if I change the code page for console by calling
chcp 437
My test application works. How can I configure code page from code?
Well I found a workaround - it can probably solve my problem but I'm not very satisfied with it. I can set up MQCCSID environment variable either globally or by calling:
Environment.SetEnvironmentVariable("MQCCSID", "437");
That will configure code page. Still I would like to use properties of a new MQQueueManager instance to setup code page.
Both of these are answers correct. For Windows Forms Project setting the environment variable MQCCSID the same as the ccsid of the Queue Manager that you are trying to connect will be enough.
- the 2nd solution
HKEY_LOCAL_MACHINE->SYSTEM->CurrentControlSet->Control->Nls->CodePage>OEMCP value.
i had a web application(web forms) that only worked with the 2nd solution
change your system locale to English(United States), on windows 7 Regional Settings -> Administrative->Change System locale. also after do that you can check it in regedit value.
regedit->HKEY_LOCAL_MACHINE->SYSTEM->CurrentControlSet->Control->Nls->CodePage check OEMCP value.

Categories