We Keep Coding

Issues with writing a C# function to load BigQuery with CSV file from Google Cloud Storage

Thanks in advance!
I am trying to create a function in C# to truncate and reload a table in bigquery. The issue I am having is that I cannot for the life of me confirm it is actually accepting the jobOptions I am inputting. I know I am able to enter the else statement, but it just displays the number of rows within the table. I've verified that a specific file I am trying to load only has one record. I think there is something up with the job options with the write disposition setting.
public void LoadTableGcsCsv(string projectId , string datasetId )
{
BigQueryClient client = BigQueryClient.Create(projectId);
var gcsURI = "gs://{bucketname}/{filename}.csv";
var dataset = client.GetDataset(datasetId);
var schema = new TableSchemaBuilder {
{ "col1", BigQueryDbType.String }
,{ "col2", BigQueryDbType.String }
,{ "col3", BigQueryDbType.Date }
,{ "col4", BigQueryDbType.String }
,{ "col5", BigQueryDbType.String }
,{ "col6", BigQueryDbType.String }
,{ "col7", BigQueryDbType.Int64 }
,{ "col8", BigQueryDbType.Date }
,{ "col9", BigQueryDbType.String }
,{ "col10", BigQueryDbType.Int64 }
}.Build();
var destinationTableRef = dataset.GetTableReference(
tableId: "destinationTableName");
// Create job configuration
var jobOptions = new CreateLoadJobOptions()
{
// The source format defaults to CSV; line below is optional.
SourceFormat = FileFormat.Csv
,CreateDisposition = CreateDisposition.CreateIfNeeded
,WriteDisposition = WriteDisposition.WriteTruncate
,SkipLeadingRows = 1
};
// Create and run job
var loadJob = client.CreateLoadJob(
sourceUri: gcsURI, destination: destinationTableRef,
schema: schema, options: jobOptions);
loadJob.PollUntilCompleted(); // Waits for the job to complete.
if (loadJob.Status.ErrorResult != null)
{
foreach (ErrorProto error in loadJob.Status.Errors)
{
Console.WriteLine(error.Message);
Console.ReadLine();
}
}
else
{
// Display the number of rows uploaded
BigQueryTable table = client.GetTable(destinationTableRef);
Console.WriteLine(
$"Loaded {table.Resource.NumRows} rows to {table.FullyQualifiedId}");
Console.ReadLine();
}
}
}

Your job is probably failing but you don't know about it, PollUntilCompleted does not modify the instance it is called on, instead it returns a new instance, so before checking for errors you need to do:
var completedLoadJob = loadJob.PollUntilCompleted();
if (completedLoadJob.Status.ErrorResult != null) ...
Or, if you want to propagate an exception instead of writing the errors in the console, you can do:
loadJob.PollUntilCompleted().ThrowOnAnyError();

C# GoogleAPI - How to set a time duration when variable type is "object"?

I'm stuck with my problem using " Google.Apis.Testing.v1.Data " and their documentation doesn't help me.
I have to set a "timeout" value (= a duration), but the variable type is "object" instead of "float" for example. I tried to put an int, a float, and a string but that doesn't work.
The object API doc is here. My variable is "TestTimeout" which is definitely a duration.
When I searched for a solution, I saw in java the variable type is string but that doesn't help (here)
Just for your information, I'm using this lib to execute my android application on their test devices. It's a service called TestLab in Firebase. The timeout value needs to be higher because I don't have enough time to execute my test. Here is my code, everything is working well besides this TimeOut.
TestMatrix testMatrix = new TestMatrix();
testMatrix.TestSpecification = new TestSpecification();
testMatrix.TestSpecification.TestTimeout = 600.0f; // I tested 600, 600.0f, "600", "30m", "500s"
testMatrix.EnvironmentMatrix = new EnvironmentMatrix();
testMatrix.EnvironmentMatrix.AndroidDeviceList = new AndroidDeviceList();
testMatrix.EnvironmentMatrix.AndroidDeviceList.AndroidDevices = new List<AndroidDevice>();
foreach (TestMatrixModel.TestData testData in _model.ListTests)
{
if (testData.IsSelected)
{
//Here I'm using my own data class to set GoogleAPI objects, it's simple
//as it asks me strings even for integer numbers, and it's working
foreach (int indice in testData.ChosenAndroidVersionsIndices)
{
AndroidDevice device = new AndroidDevice();
device.AndroidModelId = testData.ModelID;
device.AndroidVersionId = testData.AvailableAndroidVersions[indice];
device.Locale = testData.AvailableLocales[testData.ChosenLocale];
device.Orientation = testData.Orientation;
testMatrix.EnvironmentMatrix.AndroidDeviceList.AndroidDevices.Add(device);
}
}
}
Ok and here is the result of the request :
{
"testMatrixId": "matrix-2dntrwio3kco7",
"testSpecification": {
"testTimeout": "300s",
"testSetup": {},
"androidTestLoop": {
"appApk": {
"gcsPath": "gs://myLinkIntoGoogleCloudStorage.apk"
}
}
},
"environmentMatrix": {
"androidDeviceList": {
"androidDevices": [
{
"androidModelId": "grandpplte",
"androidVersionId": "23",
"locale": "en_001",
"orientation": "landscape"
},
{
"androidModelId": "hero2lte",
"androidVersionId": "23",
"locale": "en_001",
"orientation": "landscape"
},
etc.....
As you can see, it seems to be a string set to "300s"... so why "500s" cannot enter in ?
Thanks a lot.

Ok I got my answer :
testMatrix.TestSpecification.TestTimeout = "600s";
So it was a string and needed to finish with "s". Why that didn't work when I tried ? Just because my code was overrided with another TestSpecification after... my bad.

ElasticSearch C# NEST - scriptscorefunction not working

I'm trying to write a script score function which boosts items who belong to just one category.
new ScriptScoreFunction
{
Script = new ScriptQuery
{
Lang = "painless",
Source = "doc['categories'].values.length == 1",
},
Weight = 10
}
The code below does not seem to work. I'm getting a "bad response code" from the API. What am I doing wrong?

solved by using this as the source of my script:
if (doc['categories.slug'].values.length == 1) { return 1; } return 0;
NOTE: 'slug' is a property of the property 'category'.

Filter EC2 Instance with "DescribeInstanceStatus" routine - AWS SDK

I'm trying to filter EC2 instances using the AWS SDK in .NET and, although I have seen inumerous threads on SO and on other websites of people resolving this issue, nothing I've tried on my end worked.
So, as a last resource, I'm coming to you guys for help. Can anyone shed some light on what I'm missing ? I know it's very likely that I'm doing something stupid, but I can't afford to waste too much time solving this issue.
This is the chunk of code I'm using to filter an EC2 instance (get it's metadata) by it's tag name:
DescribeInstanceStatusRequest req = new DescribeInstanceStatusRequest ();
req.Filters.Add (new Filter() { Name = "tag:Name", Values = new List <string> () { "some_random_name" } });
// Executing request & fetching response
DescribeInstanceStatusResponse resp = m_ec2Client.DescribeInstanceStatus (req);
But I keep on running into this exception:
The filter 'tag:Name' is invalid
I have replaced the filter name ("tag:Name" in the example) by several filters listed in the documentation (e.g. "tag-key", "tag-value", "tag:key=value"), but nothing works.
Thank you all in advance :)

After a more thorough research, I found out that the "DescribeInstanceStatus" routine doesn't support searching by tag, but I found a somewhat "simple" way of doing so. I'll post it in here in case anyone goes through the same situation.
Here's how:
DescribeInstancesRequest req = new DescribeInstancesRequest ();
req.Filters.Add (new Filter () { Name = "tag-value", Values = new List <string> () { "something" }});
// Executing request & fetching response
DescribeInstancesResponse resp = m_ec2Client.DescribeInstances (req);
return resp.Reservations.SelectMany (x => x.Instances).Where (y => y.State.Name == InstanceStateName.Pending || y.State.Name == InstanceStateName.Running).ToList (); {code}
In theory, with this routine you can use any of the filters listed under the "Supported Filters" table in the documentation.

Getting Number of running instance from AWS EC2
DescribeInstancesRequest req = new DescribeInstancesRequest();
req.Filters.Add(new Filter {
Name = "instance-state-name",
Values = new List<string>() { "running" }
});
DescribeInstancesResponse resp = _amazonEC2Client.DescribeInstances(req);

It's may be...
// Executing request & fetching response
DescribeInstancesResponse resp = m_ec2Client.DescribeInstances (
new DescribeInstancesRequest()
{
Filters = new List<Filter>()
{
new Filter("tag:Name", new List<string>(){"some_random_name"})
}
});

Write Callback Result to Page

I have been looking for hours trying to figure out how to write the result of my callback to my webpage. It is a rather specific problem but no solution I have ran across actually works. Here are the details:
Psuedo Code: (ideally)
User interacts with webpage (in this case, changing the bounds
within a google map).
Upon bounds_changed, javascript grabs the
bounds and sends them via a jquery callback in the JSON format to my
C# code.
The C# code processes the bounds and returns the searchResults in the form of JSON to webpage to be manipulated by javascript.
The second part of step 3 is the trouble spot (everything else works). I cannot seem to actually write the result dynamically on completion of the callback.
Here is the relevant code:
Send message from client to server: (this works but just including it for completeness)
function toServer(data) {
var dataPackage = data + "~";
jQuery('form').each(function () {
document.getElementById('payload').value = JSON.stringify({ sendData: dataPackage });
$.ajax({
type: "POST",
async: true,
url: window.location.href.toString(),
data: jQuery(this).serialize(),
success: function (result) {
console.log("callback compelete");
},
error: function(error) {
console.log("callback Error");
}
});
});
}
Server Code on page load: (note this does not work [the writing of the searchResults that is])
public Map_ResultsViewer_Beta(...)
{
...
//holds actions from page
string payload = HttpContext.Current.Request.Form["payload"] ?? String.Empty;
// See if there were hidden requests
if (!String.IsNullOrEmpty(payload))
{
//temp: process the request //2do: make this dynamic
string temp_AggregationId = CurrentMode.Aggregation;
string[] temp_AggregationList = temp_AggregationId.Split(' ');
Perform_Aggregation_Search(temp_AggregationList, true, Tracer);
//NOTHING BELOW THIS REALLY WORKS, ignore the the 'placeholder' method of displaying the results, it was just one attempt.
#region
// Start to build the response
StringBuilder mapSearchBuilder = new StringBuilder();
PlaceHolder MainPlaceHolder = new PlaceHolder();
mapSearchBuilder.AppendLine(" ");
mapSearchBuilder.AppendLine(" <script type=\"text/javascript\"> ");
mapSearchBuilder.AppendLine(" function processCallbackResult(){ ");
if (!string.IsNullOrEmpty(HttpContext.Current.Session["SearchResultsJSON"].ToString()))
{
mapSearchBuilder.AppendLine(" var searchResults=" + HttpContext.Current.Session["SearchResultsJSON"] + ";");
}
else
{
mapSearchBuilder.AppendLine(" var searchResults; ");
}
mapSearchBuilder.AppendLine(" } ");
mapSearchBuilder.AppendLine(" </script> ");
mapSearchBuilder.AppendLine(" ");
// Add this to the page
Literal writeData = new Literal { Text = mapSearchBuilder.ToString() };
MainPlaceHolder.Controls.Add(writeData);
#endregion
}
else
{
//string temp_AggregationId = CurrentMode.Aggregation;
//string[] temp_AggregationList = temp_AggregationId.Split(' ');
//Perform_Aggregation_Search(temp_AggregationList, true, Tracer);
HttpContext.Current.Session["SearchResultsJSON"] = "";
}
}
Actually performing the search: (this works, just putting it here for completeness)
public void Perform_Aggregation_Search(...)
{
//search the database for all items in all of the provided aggregations and merge them into one datatable
List<DataTable> temp_Tables = new List<DataTable>();
DataTable searchResults = new DataTable();
foreach (string aggregationId in aggregationIds)
{
temp_Tables.Add(Database.Get_All_Coordinate_Points_By_Aggregation(aggregationId, Tracer));
}
foreach (DataTable temp_Table in temp_Tables)
{
searchResults.Merge(temp_Table);
}
//assign and hold the current search result datatable, from now on we will be using this as the base layer...
HttpContext.Current.Session["SearchResults"] = searchResults;
//call json writer with these results
string searchResultsJSON = Create_JSON_Search_Results_Object(searchResults,Tracer);
//send json obj to page and tell page to read it (via callback results)
HttpContext.Current.Session["SearchResultsJSON"] = searchResultsJSON;
}
Code to create the JSON object: (this works, just putting it here for completeness)
public string Create_JSON_Search_Results_Object(...)
{
//take the search results from db query (incoming) and parse into JSON
JavaScriptSerializer serializer = new JavaScriptSerializer();
List<Dictionary<string, object>> rows = new List<Dictionary<string, object>>();
Dictionary<string, object> row = null;
foreach (DataRow dr in searchResults.Rows)
{
row = new Dictionary<string, object>();
foreach (DataColumn col in searchResults.Columns)
{
row.Add(col.ColumnName.Trim(), dr[col]);
}
rows.Add(row);
}
//return JSON object
return serializer.Serialize(rows);
}
Did I miss anything? What am I doing wrong? Can you point me in the write direction?
Once again, the problem is in writing the result of the callback to the page.
Thanks!

You need to actually do something with the JSON in your success method of you ajax post.
You could use JQuery to target a DOM node like this:
$('.example').html(result);