We Keep Coding

how to create api to receive raw nmea messages

I have a requirement to create an API to receive GPS data from an IOT device and insert that data into a SQL table. Below is the raw data format. I have tried to create an API controller, but that will work only if the data format is Json.
$GPGLL,4826.67566,N,12322.19605,W,022314.000,A,A*4A
$GPGSA,A,3,30,29,10,21,24,26,15,,,,,,2.9,1.9,2.2*3D
$GPGST,022314.000,8.8,13.0,6.1,65.6,7.1,11.1,14.0*63
$GPGSV,3,1,11,05,09,179,,02,10,072,25,30,28,194,38,29,77,118,42*72
$GPGSV,3,2,11,10,42,059,36,16,24,315,27,21,45,256,43,24,84,024,40*79*
Can anyone help to solve this?
I have tried to create an API controller but that will work only if the data format is Json.
public class GPSStatusController : ApiController
{
public HttpResponseMessage Post([FromBody]GPSStatu Bts)
{
using (LocationEntities entities = new LocaionEntities())
{
var ins = new GPSStatu();
ins.ID = Bts.EvID;
ins.GPSData = Bts.GPSData;
entities.BatteryStatus.Add(ins);
entities.SaveChanges();
}
}
}

If the raw data is just a csv string then you can do:
public async Task<IActionResult> Post()
{
var list = new List<GPSStatu>()
MemoryStream mem = new MemoryStream();
await Request.Body.CopyToAsync(mem);
// parse Memorystream
var table = GenericCsvParse(mem,',',0,false,'"')
foreach (DataRow row in table.Rows)
{
list.Add(new GPSStatu{ID = row[0].toString(),.....}
}
}
public static DataTable GenericCsvParse(Stream fs, char delimiter, int skip, bool firstrowheader, char textqualifier)
{
using (StreamReader sr = new StreamReader(fs))
{
using (GenericParserAdapter parser = new GenericParserAdapter(sr)
{
ColumnDelimiter = delimiter,
SkipStartingDataRows = skip,
FirstRowHasHeader = firstrowheader,
TextQualifier = textqualifier
})
{
return parser.GetDataTable();
}
}
}
Here I used https://www.nuget.org/packages/GenericParsing, but you can do it also with plain split string.

Clone a JsonNode and attach it to another one in .NET 6

I'm using System.Text.Json.Nodes in .NET 6.0 and what I'm trying to do is simple: Copy a JsonNode from one and attach the node to another JsonNode.
The following is my code.
public static string concQuest(string input, string allQuest, string questId) {
JsonNode inputNode = JsonNode.Parse(input)!;
JsonNode allQuestNode = JsonNode.Parse(allQuest)!;
JsonNode quest = allQuestNode.AsArray().First(quest =>
quest!["id"]!.GetValue<string>() == questId) ?? throw new KeyNotFoundException("No matching questId found.");
inputNode["quest"] = quest; // Exception occured
return inputNode.ToJsonString(options);
}
But when I try to run it, I got a System.InvalidOperationException said "The node already has a parent."
I've tried edit
inputNode["quest"] = quest;
to
inputNode["quest"] = quest.Root; // quest.Root is also a JsonNode
Then the code runs well but it returns all nodes instead of the one I specified which is not the result I want. Also since the code works fine, I think it is feasible to set a JsonNode to another one directly.
According to the exception message, it seems if I want to add a JsonNode to another one, I must unattach it from its parent first, but how can I do this?
Note that my JSON file is quite big (more than 6MB), so I want to ensure there are no performance issues with my solution.

Easiest option would be to convert json node into string and parse it again (though possibly not the most performant one):
var destination = #"{}";
var source = "[{\"id\": 1, \"name\":\"some quest\"},{}]";
var sourceJson = JsonNode.Parse(source);
var destinationJson = JsonNode.Parse(destination);
var quest = sourceJson.AsArray().First();
destinationJson["quest"] = JsonNode.Parse(quest.ToJsonString());
Console.WriteLine(destinationJson.ToJsonString(new() { WriteIndented = true }));
Will print:
{
"quest": {
"id": 1,
"name": "some quest"
}
}
UPD
Another trick is to deserialize the JsonNode to JsonNode:
...
var quest = sourceJson.AsArray().First();
var clone = quest.Deserialize<JsonNode>();
clone["name"] = "New name";
destinationJson["quest"] = clone;
Console.WriteLine(quest["name"]);
Console.WriteLine(destinationJson.ToJsonString(new() { WriteIndented = true }));
Prints:
some quest
{
"quest": {
"id": 1,
"name": "New name"
}
}

As JsonNode has no Clone() method as of .NET 6, the easiest way to copy it is probably to invoke the serializer's JsonSerializer.Deserialize<TValue>(JsonNode, JsonSerializerOptions) extension method to deserialize your node directly into another node. First, introduce the following extension methods to copy or move a node:
public static partial class JsonExtensions
{
public static TNode? CopyNode<TNode>(this TNode? node) where TNode : JsonNode => node?.Deserialize<TNode>();
public static JsonNode? MoveNode(this JsonArray array, int id, JsonObject newParent, string name)
{
var node = array[id];
array.RemoveAt(id);
return newParent[name] = node;
}
public static JsonNode? MoveNode(this JsonObject parent, string oldName, JsonObject newParent, string name)
{
parent.Remove(oldName, out var node);
return newParent[name] = node;
}
public static TNode ThrowOnNull<TNode>(this TNode? value) where TNode : JsonNode => value ?? throw new JsonException("Null JSON value");
}
Now your code may be written as follows:
public static string concQuest(string input, string allQuest, string questId)
{
var inputObject = JsonNode.Parse(input).ThrowOnNull().AsObject();
var allQuestArray = JsonNode.Parse(allQuest).ThrowOnNull().AsArray();
concQuest(inputObject, allQuestArray, questId);
return inputObject.ToJsonString();
}
public static JsonNode? concQuest(JsonObject inputObject, JsonArray allQuestArray, string questId)
{
// Enumerable.First() will throw an InvalidOperationException if no element is found satisfying the predicate.
var node = allQuestArray.First(quest => quest!["id"]!.GetValue<string>() == questId);
return inputObject["quest"] = node.CopyNode();
}
Alternatively, if you aren't going to keep your array of quests around, you could just move the node from the array to the target like so:
public static string concQuest(string input, string allQuest, string questId)
{
var inputObject = JsonNode.Parse(input).ThrowOnNull().AsObject();
var allQuestArray = JsonNode.Parse(allQuest).ThrowOnNull().AsArray();
concQuest(inputObject, allQuestArray, questId);
return inputObject.ToJsonString();
}
public static JsonNode? concQuest(JsonObject inputObject, JsonArray allQuestArray, string questId)
{
// Enumerable.First() will throw an InvalidOperationException if no element is found satisfying the predicate.
var (_, index) = allQuestArray.Select((quest, index) => (quest, index)).First(p => p.quest!["id"]!.GetValue<string>() == questId);
return allQuestArray.MoveNode(index, inputObject, "quest");
}
Also, you wrote
since my json file is quite big (more than 6MB), I was worried there might be some performance issues.
In that case I would avoid loading the JSON files into the input and allQuest strings because strings larger than 85,000 bytes go on the large object heap which can cause subsequent performance degradation. Instead, deserialize directly from the relevant files into JsonNode arrays and objects like so:
var questId = "2"; // Or whatever
JsonArray allQuest;
using (var stream = new FileStream(allQuestFileName, new FileStreamOptions { Mode = FileMode.Open, Access = FileAccess.Read }))
allQuest = JsonNode.Parse(stream).ThrowOnNull().AsArray();
JsonObject input;
using (var stream = new FileStream(inputFileName, new FileStreamOptions { Mode = FileMode.Open, Access = FileAccess.Read }))
input = JsonNode.Parse(stream).ThrowOnNull().AsObject();
JsonExtensions.concQuest(input, allQuest, questId);
using (var stream = new FileStream(inputFileName, new FileStreamOptions { Mode = FileMode.Create, Access = FileAccess.Write }))
using (var writer = new Utf8JsonWriter(stream, new JsonWriterOptions { Indented = true }))
input.WriteTo(writer);
Or, if your app is asynchronous, you can do:
JsonArray allQuest;
await using (var stream = new FileStream(allQuestFileName, new FileStreamOptions { Mode = FileMode.Open, Access = FileAccess.Read, Options = FileOptions.Asynchronous }))
allQuest = (await JsonSerializer.DeserializeAsync<JsonArray>(stream)).ThrowOnNull();
JsonObject input;
await using (var stream = new FileStream(inputFileName, new FileStreamOptions { Mode = FileMode.Open, Access = FileAccess.Read, Options = FileOptions.Asynchronous }))
input = (await JsonSerializer.DeserializeAsync<JsonObject>(stream)).ThrowOnNull();
JsonExtensions.concQuest(input, allQuest, questId);
await using (var stream = new FileStream(inputFileName, new FileStreamOptions { Mode = FileMode.Create, Access = FileAccess.Write, Options = FileOptions.Asynchronous }))
await JsonSerializer.SerializeAsync(stream, input, new JsonSerializerOptions { WriteIndented = true });
Notes:
Microsoft's documentation explicitly recommends against serializing from and to strings instead of UTF-8 byte sequences for performance reasons which is another reason not to load your large JSON files into temporary string buffers.
Demo fiddles:
For copying the node, see https://dotnetfiddle.net/cwKDen.
For moving the node, see https://dotnetfiddle.net/cI8DuB.
For async reading and writing, see https://dotnetfiddle.net/VjKstQ

Trouble Fetching Value in Variable

Basically here's my code which I'm having trouble with. Insanely new to mongoDB and would love to understand how to get values out of a JSON string that is returns in the variable 'line'.
public string get_data()
{
var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<BsonDocument>("metacorp");
var cursor = collection.Find("{'movie_name' : 'Hemin'}").ToCursor();
var line = "";
foreach (var document in cursor.ToEnumerable())
{
using (var stringWriter = new StringWriter())
using (var jsonWriter = new JsonWriter(stringWriter))
{
var context = BsonSerializationContext.CreateRoot(jsonWriter);
collection.DocumentSerializer.Serialize(context, document);
line = stringWriter.ToString();
}
}
var js = new JavaScriptSerializer();
var d = js.Deserialize<dynamic>(line);
var a = d["movie_name"];
return line;
}
This is the output I get if I return line:
{ "_id" : ObjectId("58746dcafead398e4d7233f5"), "movie_name" : "Hemin"
}
I want to be able to fetch the value 'Hemin' into 'a'.

I know this is not what you're asking for but since you're using the c# driver then I would recommend the following. Assumes you have a c# class corresponding to metacorp collection or at least a serializer that handles it. Hope it helps.
var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<MetaCorp>("metacorp");
var m = collection.SingleOrDefault(x => x.Movie_Name == "Hemin"); // Assuming 0 or 1 with that name. Use Where otherwise
var movieName = "Not found";
if(m!= null)
movieName = m.Movie_Name;

You could have your dto class for movie ans just fetch the data from collection:
public class Movie
{
public ObjectId Id { get; set; }
public string movie_name { get; set;}
}
...
var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<BsonDocument>("metacorp");
var movies = collection.Find(x=>x.movie_name == "Hemin").ToEnumerable();

Inserting json documents in DocumentDB

In DocumentDB documentation examples, I find insertion of C# objects.
// Create the Andersen family document.
Family AndersenFamily = new Family
{
Id = "AndersenFamily",
LastName = "Andersen",
Parents = new Parent[] {
new Parent { FirstName = "Thomas" },
new Parent { FirstName = "Mary Kay"}
},
IsRegistered = true
};
await client.CreateDocumentAsync(documentCollection.DocumentsLink, AndersenFamily);
In my case, I'm receiving json strings from application client and would like to insert them in DocumentDB without deserializing them. Could not find any examples of doing something similar.
Any help is sincerely appreciated..
Thanks

Copied from the published .NET Sample code -
private static async Task UseStreams(string colSelfLink)
{
var dir = new DirectoryInfo(#".\Data");
var files = dir.EnumerateFiles("*.json");
foreach (var file in files)
{
using (var fileStream = new FileStream(file.FullName, FileMode.Open, FileAccess.Read))
{
Document doc = await client.CreateDocumentAsync(colSelfLink, Resource.LoadFrom<Document>(fileStream));
Console.WriteLine("Created Document: ", doc);
}
}
//Read one the documents created above directly in to a Json string
Document readDoc = client.CreateDocumentQuery(colSelfLink).Where(d => d.Id == "JSON1").AsEnumerable().First();
string content = JsonConvert.SerializeObject(readDoc);
//Update a document with some Json text,
//Here we're replacing a previously created document with some new text and even introudcing a new Property, Status=Cancelled
using (var memoryStream = new MemoryStream(Encoding.UTF8.GetBytes("{\"id\": \"JSON1\",\"PurchaseOrderNumber\": \"PO18009186470\",\"Status\": \"Cancelled\"}")))
{
await client.ReplaceDocumentAsync(readDoc.SelfLink, Resource.LoadFrom<Document>(memoryStream));
}
}

Json.net deserialize DateTime from HTTPClient result

I am using HTTPCLient to call RestFul service. My problem when parsing DateTime.
Because in my class I have DateTime Property. Which in Json it is type long. Json key is: exp
{
"resultCodes": "OK",
"description": "OK",
"pans": {
"maskedPan": [
{
"id": "4b533683-bla-bla-3517",
"pan": "67*********98",
"exp": 1446321600000,
"isDefault": true
},
{
"id": "a3093f00-zurna-01e18a8d4d72",
"pan": "57*********96",
"exp": 1554058800000,
"isDefault": false
}
]
}
}
In documentation i read that
To minimize memory usage and the number of objects allocated Json.NET supports serializing and deserializing directly to a stream.
So =>
WAY 1 (Reading via GetStringAsync). In documentation has written that use StreamReader instead.
return Task.Factory.StartNew(() =>
{
var client = new HttpClient(_handler);
var url = String.Format(_baseUrl + #"list/{0}", sessionId);
BillsList result;
var rrrrr = client.GetStringAsync(url).Result;
result = JsonConvert.DeserializeObject<BillsList>(rrrrr,
new MyDateTimeConverter());
return result;
}, cancellationToken);
WAY 2(Good way. I read via StreamReader. Bu in line var rTS = sr.ReadToEnd(); it creates new string. It is not good. Because i have used GetStreamAsync to avoid of creating string variable.)
return Task.Factory.StartNew(() =>
{
var client = new HttpClient(_handler);
var url = String.Format(_baseUrl + #"list/{0}", sessionId);
BillsList result;
using (var s = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(s))
using (JsonReader reader = new JsonTextReader(sr))
{
var rTS = sr.ReadToEnd();
result = JsonConvert.DeserializeObject<BillsList>(rTS,
new MyDateTimeConverter());
}
return result;
}, cancellationToken);
WAY 3(The best. But it gives exception if property is DateTime in my class. )
return Task.Factory.StartNew(() =>
{
var client = new HttpClient(_handler);
var url = String.Format(_baseUrl + #"list/{0}", sessionId);
BillsList result;
using (var s = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(s))
using (JsonReader reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
result = serializer.Deserialize<BillsList>(reader);
}
return result;
}, cancellationToken);
So my question. I want to continue with 3-rd way. But have there any way to set some handler as MyDateTimeConverter for JsonSerializer to convert it automatically?

You can set up default JsonSerializerSettings when your app is initialized:
// This needs to be done only once, so put it in an appropriate static initializer.
JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
Converters = new List<JsonConverter> { new MyDateTimeConverter() }
};
Then later you can use JsonSerializer.CreateDefault
JsonSerializer serializer = JsonSerializer.CreateDefault();
result = serializer.Deserialize<BillsList>(reader);

You can add your MyDateTimeConverter to the Converters collection on the JsonSerializer; that should allow you to use your third approach without getting errors.
var serializer = new JsonSerializer();
serializer.Converters.Add(new MyDateTimeConverter());
result = serializer.Deserialize<BillsList>(reader);