We Keep Coding

C# GoogleAPI - How to set a time duration when variable type is "object"?

I'm stuck with my problem using " Google.Apis.Testing.v1.Data " and their documentation doesn't help me.
I have to set a "timeout" value (= a duration), but the variable type is "object" instead of "float" for example. I tried to put an int, a float, and a string but that doesn't work.
The object API doc is here. My variable is "TestTimeout" which is definitely a duration.
When I searched for a solution, I saw in java the variable type is string but that doesn't help (here)
Just for your information, I'm using this lib to execute my android application on their test devices. It's a service called TestLab in Firebase. The timeout value needs to be higher because I don't have enough time to execute my test. Here is my code, everything is working well besides this TimeOut.
TestMatrix testMatrix = new TestMatrix();
testMatrix.TestSpecification = new TestSpecification();
testMatrix.TestSpecification.TestTimeout = 600.0f; // I tested 600, 600.0f, "600", "30m", "500s"
testMatrix.EnvironmentMatrix = new EnvironmentMatrix();
testMatrix.EnvironmentMatrix.AndroidDeviceList = new AndroidDeviceList();
testMatrix.EnvironmentMatrix.AndroidDeviceList.AndroidDevices = new List<AndroidDevice>();
foreach (TestMatrixModel.TestData testData in _model.ListTests)
{
if (testData.IsSelected)
{
//Here I'm using my own data class to set GoogleAPI objects, it's simple
//as it asks me strings even for integer numbers, and it's working
foreach (int indice in testData.ChosenAndroidVersionsIndices)
{
AndroidDevice device = new AndroidDevice();
device.AndroidModelId = testData.ModelID;
device.AndroidVersionId = testData.AvailableAndroidVersions[indice];
device.Locale = testData.AvailableLocales[testData.ChosenLocale];
device.Orientation = testData.Orientation;
testMatrix.EnvironmentMatrix.AndroidDeviceList.AndroidDevices.Add(device);
}
}
}
Ok and here is the result of the request :
{
"testMatrixId": "matrix-2dntrwio3kco7",
"testSpecification": {
"testTimeout": "300s",
"testSetup": {},
"androidTestLoop": {
"appApk": {
"gcsPath": "gs://myLinkIntoGoogleCloudStorage.apk"
}
}
},
"environmentMatrix": {
"androidDeviceList": {
"androidDevices": [
{
"androidModelId": "grandpplte",
"androidVersionId": "23",
"locale": "en_001",
"orientation": "landscape"
},
{
"androidModelId": "hero2lte",
"androidVersionId": "23",
"locale": "en_001",
"orientation": "landscape"
},
etc.....
As you can see, it seems to be a string set to "300s"... so why "500s" cannot enter in ?
Thanks a lot.

Ok I got my answer :
testMatrix.TestSpecification.TestTimeout = "600s";
So it was a string and needed to finish with "s". Why that didn't work when I tried ? Just because my code was overrided with another TestSpecification after... my bad.

Deserialize Avro Spark

I'm pushing a stream of data to Azure EventHub with the following code leveraging Microsoft.Hadoop.Avro.. this code runs every 5 seconds, and simply plops the same two Avro serialised items 👍🏼:
var strSchema = File.ReadAllText("schema.json");
var avroSerializer = AvroSerializer.CreateGeneric(strSchema);
var rootSchema = avroSerializer.WriterSchema as RecordSchema;
var itemList = new List<AvroRecord>();
dynamic record_one = new AvroRecord(rootSchema);
record_one.FirstName = "Some";
record_one.LastName = "Guy";
itemList.Add(record_one);
dynamic record_two = new AvroRecord(rootSchema);
record_two.FirstName = "A.";
record_two.LastName = "Person";
itemList.Add(record_two);
using (var buffer = new MemoryStream())
{
using (var writer = AvroContainer.CreateGenericWriter(strSchema, buffer, Codec.Null))
{
using (var streamWriter = new SequentialWriter<object>(writer, itemList.Count))
{
foreach (var item in itemList)
{
streamWriter.Write(item);
}
}
}
eventHubClient.SendAsync(new EventData(buffer.ToArray()));
}
The schema used here is, again, v. simple:
{
"type": "record",
"name": "User",
"namespace": "SerDes",
"fields": [
{
"name": "FirstName",
"type": "string"
},
{
"name": "LastName",
"type": "string"
}
]
}
I have validated this is all good, with a simple view in Azure Stream Analytics on the portal:
So far so good, but i cannot, for the life of me correctly deserialize this in Databricks leverage the from_avro() command under Scala..
Load (the exact same) schema as a string:
val sampleJsonSchema = dbutils.fs.head("/mnt/schemas/schema.json")
Configure EventHub
val connectionString = ConnectionStringBuilder("<CONNECTION_STRING>")
.setEventHubName("<NAME_OF_EVENT_HUB>")
.build
val eventHubsConf = EventHubsConf(connectionString).setStartingPosition(EventPosition.fromEndOfStream)
val eventhubs = spark.readStream.format("eventhubs").options(eventHubsConf.toMap).load()
Read the data..
// this works, and i can see the serialised data
display(eventhubs.select($"body"))
// this fails, and with an exception: org.apache.spark.SparkException: Malformed records are detected in record parsing. Current parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as 'PERMISSIVE'.
display(eventhubs.select(from_avro($"body", sampleJsonSchema)))
So essentially, what is going on here.. i am serialising the data with the same schema as deserializing, but something is malformed.. the documentation is incredibly sparse on this front (very very minimal on the Microsoft website).

The issue
After additional investigation, (and mainly with the help of this article) I found what my problem was: from_avro(data: Column, jsonFormatSchema: String) expects spark schema format and not avro schema format. The documentation is not very clear on this.
Solution 1
Databricks provides a handy method from_avro(column: Column, subject: String, schemaRegistryUrl: String)) that fetches needed avro schema from kafka schema registry and automatically converts to correct format.
Unfortunately, it is not available for pure spark, nor is it possible to use it without a kafka schema registry.
Solution 2
Use schema conversion provided by spark:
// define avro deserializer
class AvroDeserializer() extends AbstractKafkaAvroDeserializer {
override def deserialize(payload: Array[Byte]): String = {
val genericRecord = this.deserialize(payload).asInstanceOf[GenericRecord]
genericRecord.toString
}
}
// create deserializer instance
val deserializer = new AvroDeserializer()
// register deserializer
spark.udf.register("deserialize_avro", (bytes: Array[Byte]) =>
deserializer.deserialize(bytes)
)
// get avro schema from registry (but I presume that it should also work with schema read from a local file)
val registryClient = new CachedSchemaRegistryClient(kafkaSchemaRegistryUrl, 128)
val avroSchema = registryClient.getLatestSchemaMetadata(topic + "-value").getSchema
val sparkSchema = SchemaConverters.toSqlType(new Schema.Parser().parse(avroSchema))
// consume data
df.selectExpr("deserialize_avro(value) as data")
.select(from_json(col("data"), sparkSchema.dataType).as("data"))
.select("data.*")

Parsing/Iterating over Json

I'm fairly new to parsing Json with C# and i'm having a little issue i can't work my head around.
My data looks something like this:
{
"languages": {
"ja_lang": {
"data": {
"name": "Japanese"
},
"files": [["ja",
"Japanese File",
"lang_ja.txt"]]
},
"en_lang": {
"data": {
"name": "English"
},
"files": [["en",
"English File",
"lang_en.txt"]]
}
}
}
Now i want to iterate over the items in languages and only work with the one where the object-name starts with "ja_" (in this case it would only work with "ja_lang" and ignore "en_lang"), then extract the name inside data and the "lang_ja.txt" in files.
To Parse the Json in C# i downloaded the Newtonsoft.Json library and came up with this:
dynamic json_obj = JsonConvert.DeserializeObject("json string");
// when debugging language holds { "ja_lang": { "data": { "name": "Japanese" }, "files": [["ja", "Japanese File", "lang_ja.txt"]] } }
foreach (var language in json_obj.languages)
{
// not sure how i can access the object-name
/*if(!language.StartsWith("ja_"))
continue;*/
// Exception: 'Newtonsoft.Json.Linq.JProperty' does not contain a definition for 'data' - Not sure why it is treated as a property?
var name = language.data.name;
var file = language.files[2];
}
I'm sorry for this probably dumb question, but i've been trying to cast it to different types and searched the web for solutions, but i just couldn't figure it out. So if someone could help me out with this i would be really greatful.
Thanks in advance!

Since you're stating in a comment (on an answer that has been deleted) that the data changes so a fixed model won't work, you can still fix what is known:
Here's a LINQPad program that demonstrates:
void Main()
{
var collection = JsonConvert.DeserializeObject<LanguagesCollection>(File.ReadAllText(#"c:\temp\test.json"));
foreach (var keyValuePair in collection.Languages)
if (keyValuePair.Key.StartsWith("ja_"))
keyValuePair.Value.Dump();
}
public class LanguagesCollection
{
public Dictionary<string, JObject> Languages { get; } = new Dictionary<string, JObject>();
}
This will deserialize the outer object, with the "languages" key, and inside you have a dictionary with the keys, "ja_lang", "en_lang", and you can just process the values as you see fit. These are left as JObject which means they will contain whatever json was present as a value for that key in the dictionary.

Using a site like json2sharp you can just pass your json data in and get a ready to use c# model out.
Then you can easily deserialize your json data into that c# model and use the properties for much easier handling:
string jsonData = #"{
'languages': {
'ja_lang': {
'data': {
'name': 'Japanese'
},
'files': [['ja',
'Japanese File',
'lang_ja.txt']]
},
'en_lang': {
'data': {
'name': 'English'
},
'files': [['en',
'English File',
'lang_en.txt']]
}
}
}";
RootObject data = JsonConvert.DeserializeObject<RootObject>(jsonData);
foreach(Languages lang in data.languages) //would work if Languages was a listing
{
}
Although I admit that your Json is a bit strange and that Languages most likly should be a listing and not a property for each language.

ASP.Net JSON configuration file transforms with arrays

I have a base configuration file eg.
appsettings.json
{
"Values": {
"Test": ["one", "two"]
}
}
and
appsettings.dev.json
{
"Values": {
"Test": ["three"]
}
}
and after transforming, the array would be
["three", "two"]
How do I make sure the transformed array is shrunk to a smaller number of elements rather than each element changing individually?

To understand the cause of such 'strange' behavior for overridden array settings you need to understand how those settings are stored inside configuration providers.
The reality is that all loaded settings are stored in dictionaries, own for each configuration provider. Keys are built from setting paths where nested sections are delimited with a colon.
Array settings are stored in the same dictionary with an index in setting path (:0, :1, ...).
For configuration you described you will have 2 configuration providers with following sets of settings:
provider1[Values:Test:0] = "one"
provider1[Values:Test:1] = "two"
and
provider2[Values:Test:0] = "three"
Now it's clear why the final value of array setting is ["three", "two"]. Values:Test:0 from the second provider overrides the same setting from the first provider, and Values:Test:1 is left untouched.
Unfortunately, there is now a built-in possibility to overcome this problem. Fortunately, .net core configuration model is flexible enough for adjusting this behavior for your needs.
Idea is the following:
Enumerate configuration providers in reverse order.
For each provider get all its setting keys. You could call IConfigurationProvider.GetChildKeys() method recursively for this purpose. See GetProviderKeys() in below snippet.
With a regular expression check whether current key is an array entry.
If it is and some of previous providers overrides this array, then just suppress current array entry by setting it to null value.
If it's unseen array then current provider is marked as the only provider of values for this array. Arrays from all other providers will be suppressed (step #4).
For convenience you could wrap all this logic into extension method on IConfigurationRoot.
Here is a working sample:
public static class ConfigurationRootExtensions
{
private static readonly Regex ArrayKeyRegex = new Regex("^(.+):\\d+$", RegexOptions.Compiled);
public static IConfigurationRoot FixOverridenArrays(this IConfigurationRoot configurationRoot)
{
HashSet<string> knownArrayKeys = new HashSet<string>();
foreach (IConfigurationProvider provider in configurationRoot.Providers.Reverse())
{
HashSet<string> currProviderArrayKeys = new HashSet<string>();
foreach (var key in GetProviderKeys(provider, null).Reverse())
{
// Is this an array value?
var match = ArrayKeyRegex.Match(key);
if (match.Success)
{
var arrayKey = match.Groups[1].Value;
// Some provider overrides this array.
// Suppressing the value.
if (knownArrayKeys.Contains(arrayKey))
{
provider.Set(key, null);
}
else
{
currProviderArrayKeys.Add(arrayKey);
}
}
}
foreach (var key in currProviderArrayKeys)
{
knownArrayKeys.Add(key);
}
}
return configurationRoot;
}
private static IEnumerable<string> GetProviderKeys(IConfigurationProvider provider,
string parentPath)
{
var prefix = parentPath == null
? string.Empty
: parentPath + ConfigurationPath.KeyDelimiter;
List<string> keys = new List<string>();
var childKeys = provider.GetChildKeys(Enumerable.Empty<string>(), parentPath)
.Distinct()
.Select(k => prefix + k).ToList();
keys.AddRange(childKeys);
foreach (var key in childKeys)
{
keys.AddRange(GetProviderKeys(provider, key));
}
return keys;
}
}
The last thing is to call it when building the configuration:
IConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
configurationBuilder.AddJsonFile("AppSettings.json")
.AddJsonFile("appsettings.dev.json");
var configuration = configurationBuilder.Build();
configuration.FixOverridenArrays();

#Matt in my opinion it's just an unnecessary logic. Flow 'KISS'.
appsettings.json should contain only common settings. If your production mode or development mode must have some same values in one key, just duplicate their. Like appsettings.Development.json
"Values": {
"Test": ["one", "two"]
}
and appsettings.Production.json
"Values": {
"Test": ["one", "two","three"]
}
And if you need same values for both modes you should put it in appsettings.json.
"SomeValues": {
"Test": ["1", "2","3"]
}
In final settings you will have for production
"SomeValues": {
"Test": ["1", "2","3"]
},
"Values": {
"Test": ["one", "two","three"]
}
and for development
"SomeValues": {
"Test": ["1", "2","3"]
},
"Values": {
"Test": ["one", "two"]
}
anyway If previous answer solve your problem it's ok, it's just my opinion. Thanks)

I recommend use appsettings.Development.json and appsettings.Production.json to separate environments. And keep common settings in appsettings.json for both environments.
Just rename your appsettings.dev.json to appsettings.Development.json. Add Stage or Prodaction mode of appsettings.{#mode}.json. And modify ConfigurationBuilder in Startup.cs.
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
I think it's more common practice and could save your time from unnecessary logic of merging

How remove element for json file in C#

I have json file Users.json and I want delete one element. This is the structure:
{
"1": {
"Username": "1",
"Name": "1",
"AccessGroup": "Administrators"
},
"2": {
"Username": "2",
"Name": "2",
"AccessGroup": "Supervisors"
},
"3": {
"Username": "3",
"Name": "3",
"AccessGroup": "Administrators"
}
}
And the code:
public void DeleteUser(Users.User user)
{
String filename = USERS_PATH;
try
{
if (File.Exists(filename))
{
var data = DeserializeFromFile<Dictionary<String, User>>(USERS_PATH);
foreach (var item in data)
{
if (user.Username == data[item.Key].Username)
{
data.Remove(user.Username);
break;
}
}
}
}
catch (Exception)
{
throw new ArgumentException("User has not deleted");
}
}
After that iteration the file is the same like before. Where do I wrong? Thanks in advance.

There are a few things wrong with this code:
After loading the file into an object, the Dictionary<string,User> no longer corresponds to the file: if the dictionary is updated, the file isn't and vice versa, you you need to serialize the data again and save it to the file with:
String encoded = JsonConvert.SerializeObject(data);
System.IO.File.WriteAllText(filename,encode);
Dictionary.Remove asks to provide a key, you can remove with the username, it's not guaranteed (especially not in your example file), that the key is the username. The result is that nothing is removed. So you should use:
data.Remove(item.Key);
instead of:
data.Remove(user.Username);
Next, you shouldn't remove data in an iterator. Although this works in this case because you do a break, in general it's a very bad idea to do this since most enumerators are not designed to enumerate over changing collections. In this case you can store a reference to the key to be removed:
var torem = null;
foreach (var item in data) {
if (user.Username == data[item.Key].Username) {
torem = item.Key;
break;
}
}
if(torem != null) {
data.Remove(torem);
}
You can also save some CPU cycles by using item.Value instead of data[item.Key].Username since what you do is a lookup for a value where you already have the pointer so use:
if(user.Username == item.Value.Username)
instead of:
if (user.Username == data[item.Key].Username)

You have to save the data to he same file.
File.WriteAllText(USERS_PATH, JsonConvert.SerializeObject(data));
// I don't know what JSON parser you're using, but replace it with
// whatever that library provides for serializing data/saving to file :)
or something like this :)
Basically you're working on an in-memory copy of what was read from file. If you want to use a JSON like a DB, you might want to write some kind of DAO/Repository that will just expose basic CRUD methods (open file -> perform changes -> write file).

I could be wrong, but don't you have to delete an item by its key, not by a property of the item?
So:
data.Remove(item.Key);