HDinsight Store Pig Result - c#

I want to submit a pig job with the .NET SDK to HDInsight (but it also happens when I submit it with PowerShell, tested that already). Using following Statement:
string queryString = "REGISTER wasb:///PigTestFolder/myudfs.jar;" +
" A = LOAD 'wasb:///PigTestFolder/wordlist.txt' USING PigStorage()AS Line:chararray;" +
" B =ORDER A BY Line asc;" +
" D= Limit B 20;" +
" Dump D;" +
" STORE D INTO 'wasb:///PigTestFolder/results/' USING PigStorage ('\t');";
When I remove the STORE command everything works perfectly fine. But Storing the result to the BLOB is a pain in the arse. It say in the error log: "AS was unexpected at this time". But if I remove the schema and define the column with $0 it says: B was unexpected at this time.
The following Code prints out: Dump unexpected at this time.
string queryString = " A = LOAD 'wasb:///PigTestFolder/wordlist.txt' USING PigStorage();" +
" Dump A;" +
" STORE A INTO 'wasb:///PigTestFolder/results/' USING PigStorage ('\t');";
I tested the code with the Hortonworks Sandbox and it works. No problem. Another mystery is, when using only the following two lines of PigLatin code it works as well:
string queryString = " A = LOAD 'wasb:///PigTestFolder/wordlist.txt' USING PigStorage();" +
"STORE D INTO 'wasb:///PigTestFolder/results/' USING PigStorage ('\t');";
Does anyone have any idea what I am doing wrong?

Related

How to display query in output

So I have a query and am trying to display it in the Debug Output, when I run the file it gives me a list of output starting with iisexpress.exe : https://gyazo.com/fd9eb832dfcc08571b31490103b85b49
but no actual result? I am trying to run a query on Visual Studios2015 for the first time using the dotnetRDF. My code is below:
public static void Main(String[] args)
{
Debug.WriteLine("SQLAQL query example");
//Define a remote endpoint
//Use the DBPedia SPARQL endpoint with the default Graph set to DBPedia
SparqlRemoteEndpoint endpoint = new SparqlRemoteEndpoint(new Uri("http://dbpedia.org/sparql"), "http://dbpedia.org");
//SPARQL query to show countries, population, capital for countries where population is more than 100000 and limit results to 50
String queryString = "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> " +
"PREFIX type: <http://dbpedia.org/class/yago/> " +
"PREFIX prop: <http://dbpedia.org/property/> " +
"SELECT ?country_name ?population ?cptl " +
"WHERE { " +
"?country rdf:type type:Country108544813. " +
"?country rdfs:label ?country_name. " +
"?country prop:populationEstimate ?population. " +
"?country dbo:capital ?cptl " +
"FILTER (?population > 1000000000) . " +
"}" +
"LIMIT 50 ";
Debug.WriteLine("queryString: [" + queryString + "]");
//Make a SELECT query against the Endpoint
SparqlResultSet results = endpoint.QueryWithResultSet(queryString);
foreach (SparqlResult result in results)
{
Debug.WriteLine(result.ToString());
}
}
Just learning SPARQL so this maybe a very basic question.
Many Thanks:)
You need to make sure that your code is compiled and run in Debug mode. If is not then Debug.WriteLine() will have no effect. The output you have provided is incomplete, future reference it is better to copy and paste into your question rather than posting a screenshot.
Since this appears to be a console application why not just use Console.WriteLine() instead?

C# Razor The name does not exist in the current context

I know I must be being stupid here, but I can't work out why this code isn't working. I'm very new to Razor, so please go easy on me.
I have the following code in my markup: (I've cut it down to be as simple as I can make it while still reproducing the issue so that I can hopefully diagnose the issue easier)
string testVar = "test";
#testVar
It returns the following:
error CS0103: The name 'testVar' does not exist in the current context
I've tried using different names for the variable, I've tried having it use "var" instead of "string" for it's declaration, I've tried assigning various different values to it, I've tried surrounding the variable in brackets like #(testVar), but the issue is still there. It's very frustrating, because in another part of my code I have
string prop = #p.Name + " (" + #p.PropertyType + ") - " + #p.GetValue(#page, null).ToString() + "\r\n";
#prop
which works fine.
I can't think what could be causing it, and it's starting to frustrate me.
Thanks,
YM
In Razor when we want to write c# statements we have to tell it that from here c# code starts:
#{ // from here c# block starts
string testVar = "test";
} // here ends
Now if you want to access this variable in html you can do like this:
<span>#testVar</span>
When you are writing:
string prop = #p.Name + " (" + #p.PropertyType + ") - " + #p.GetValue(#page, null).ToString() + "\r\n";
it is considered as plain text and will be rendered on browser like "string prop = ..."
you have to tell that it is c# code by following way:
#{
string prop = #p.Name + " (" + #p.PropertyType + ") - " + #p.GetValue(#page, null).ToString() + "\r\n";
}

How to check if a date falls between two other dates in XPath

I'm trying to pull event logs from a windows PC given certain information. This block of code is working fine but only pulls logs from the current day. I'd like to only pull logs that where created between dates that are specified earlier, "toDate" and "fromDate". This is my first time using xpath and I can't seem to find anything online that deals with something like this.
string queryString =
"<QueryList>" +
" <Query Id=\"0\" Path=\"Application\">" +
" <Select Path=\"Application\">" +
" *[System[(Level=3) and " +
" TimeCreated[timediff(#SystemTime) <= 86400000]]]" +
" </Select>" +
" </Query>" +
"</QueryList>";

MySQL leading whitespace with C#

When I update a field in my MySQL database, it always adds a whitespace to the value.
I tried to remove the whitespace with the trim-command and the replace-command. Neither of them worked. So I expect that it isn't a whitespace but some vague ASCII character. These are the commands I used:
this.foo = result.GetValue(0).ToString().Trim();
this.bar = result.GetValue(0).ToString().Replace(" ","");
The field it updates is a VARCHAR(xx). This is my MySQL update command:
MySqlCommand cmd = new MySqlCommand("UPDATE " + table + " SET " + new_field + " =' " + new_value+ "' WHERE " + field+ "= " + value + "",this.con);
this.con is my connection to the MySQL database.
FYI: I use .NET 3.5CF with a mysql.data.cf DLL in Visual Studio 2008.
Could someone help me out with this problem? It's driving me nuts.
Well yes, you've got a leading space in the SQL:
"UPDATE " + table + " SET " + new_field + " =' " + new_value+ "'
Note the bit straight after "=" - you've got a quote, then a space, then new_value.
However, you shouldn't be putting the values in the SQL directly in the first place - you should be using parameterized SQL statements... currently you've got a SQL injection attack waiting to happen, as well as potential problems for honest values with quotes in.
You should use parameterized SQL for both new_value and value here... I'm assuming that field and table come from more "trusted" sources?
This appears to have a space where the * is
" ='*" + new_value

int.Parse of "8" fails. int.Parse always requires CultureInfo.InvariantCulture?

We develop an established software which works fine on all known computers except one. The problem is to parse strings that begin with "8". It seems like "8" in the beginning of a string is a reserved character.
Parsing:
int.Parse("8") -> Exception message: Input string was not in a correct format.
int.Parse("80") -> 0
int.Parse("88") -> 8
int.Parse("8100") -> 100
CurrentCulture: sv-SE
CurrentUICulture: en-US
The problem is solved using int.Parse("8", CultureInfo.InvariantCulture). However, it would be nice to know the source of the problem.
Question: Why do we get this behaviour of "8" if we don't specify invariant culture?
Additional information:
I did send a small program to my client achieve the result above:
private int ParseInt(string s)
{
int parsedInt = -1000;
try
{
parsedInt = int.Parse(s);
textBoxMessage.Text = "Success: " + parsedInt;
}
catch (Exception ex)
{
textBoxMessage.Text =
string.Format("Error parsing string: '{0}'", s) + Environment.NewLine +
"Exception message: " + ex.Message;
}
textBoxMessage.Text += Environment.NewLine + Environment.NewLine +
"CurrentCulture: " + Thread.CurrentThread.CurrentCulture.Name + "\r\n" +
"CurrentUICulture: " + Thread.CurrentThread.CurrentUICulture.Name + "\r\n";
return parsedInt;
}
Update
I stumbled across this link, a bug in the microsoft connect database:
https://connect.microsoft.com/VisualStudio/feedback/details/253265/int32-parse-fails-to-convert-the-string-0-zero-on-some-systems
It seems like there's an issue with similiar symptoms, but no real root cause. If anyone could elaborate on this I would be grateful!
For the sv-SE culture 8 represents CurrencyNegativePattern and that's why you're getting the error you describe.
You can check this by running the following example:
var ci = new CultureInfo("sv-SE");
var nfi = (NumberFormatInfo)ci.GetFormat(typeof(NumberFormatInfo));
Console.WriteLine(nfi.CurrencyNegativePattern);
Console.WriteLine(nfi.CurrencyPositivePattern);
This will output:
// 8
// 3
You can explicitly say that you are parsing an integer and not a currency by using the Parse overload that accepts a NumberStyles enumeration.
Int32.Parse("8", NumberStyles.Integer, new CultureInfo("sv-SE"));
This time since you are specifying that you're parsing an integer no error will occur.
However, IIRC the Int32.Parse should interpret the input as an integer by default, so why you're getting to the error with that sample code is beyond me.
Update:
From the information you recently added it seems that you should make sure that the problem is not external. This is, if the user for example changed the positive sign setting of the windows locale to 8 it would be normal and make perfect sense for you to get the error you are obtaining. It would be just like setting the + as the positive sign and then trying to parse it:
var ci = new CultureInfo("sv-SE");
var nfi = (NumberFormatInfo)ci.GetFormat(typeof(NumberFormatInfo));
nfi.PositiveSign = "+";
Int32.Parse("+", nfi); // This will throw
Ask the user for it's locale registry settings like indicated in the Connect issue and check that they are what you would expect.
Side note: Welcome to SO and by the way next time you need to add further information to your question you should edit it instead of providing it in an answer.

Categories