How to add CSV reader "records" to database? - c#

I am currently building a .NET service that takes a CSV file, parses it, and stores the parsed data into the database. I've already built the parse logic using CSVhelper. I am fairly new to Backend languages and cannot figure out how to store the data into the database. There isn't a lot of code, but I know i'm making a mistake somewhere.
Parse Service code:
public void ReadMonitoredEvent( IFormFile file)
{
MonitoredEvent monitoredEvent = new MonitoredEvent();
TextReader reader = new StreamReader(file.OpenReadStream());
var csvReader = new CsvReader(reader);
var records = csvReader.GetRecords<MonitoredEvent>();
this.dbContext.MonitoredEvents.Add(monitoredEvent);
this.dbContext.SaveChanges();
}
I've tried changing out the parameters for the "this.dbContext.MonitoredEvent...." but can't seem to find the proper entity to put as parameters
I tested in Postman without the dbContext part and I get a successful 200 code returned, but with the dbContext statements I am getting:
"NullReferenceException: Object reference not set to an instance of an
object."
at line: this.dbContext.MonitoredEvents.Add(monitoredEvent);
If someone can help me get this sorted out so I can store it into my database I'd appreciate it. I already have my entity model built and all the headers in the CSV match up.

Related

Excel interop CopyFromRecordset/ADODB (custom data/no database) in .net core

We have this poor man's weekly task, which essentially loads a bunch of CSVs, massages them a bit, dumps them into preloaded Excel template and sends them as an attachment to mgmt. Then they play with it using excel filters and charts, etc (not important here).
What is important is that this used to be a .NET Framework app which we have to migrate to .net core, and the initial author did this by constructing an ADODB recordset and using CopyFromRecordset method on a Range to dump the entire table. For example, this constructs the recordset:
internal static ADODB.Recordset ConvertToADODBRecordSet(this List<MyData> data)
{
var result = new ADODB.Recordset { CursorLocation = ADODB.CursorLocationEnum.adUseClient };
var resultFields = result.Fields;
resultFields.Append("Date", ADODB.DataTypeEnum.adDate);
resultFields.Append("Hour", ADODB.DataTypeEnum.adInteger);
resultFields.Append("Code", ADODB.DataTypeEnum.adVarWChar, 16);
result.Open(CursorType: ADODB.CursorTypeEnum.adOpenStatic, LockType: ADODB.LockTypeEnum.adLockOptimistic);
foreach (var dr in data)
{
result.AddNew();
resultFields[0].Value = dr.Date;
resultFields[1].Value = dr.Hour;
resultFields[2].Value = dr.Code;
}
return result;
}
...
var table = sheet.ListObjects["CodesTable"];
table.InsertRowRange.CopyFromRecordset(data.ConvertToADODBRecordSet());
As far as I see, there is not ADODB support in .net framework, I couldn't even try it with supported database objects because, well, there is no database here, recordset is manual and judging by other peoples' attempts it wouldn't work anyway. Entering values in cells one by one is not feasible, there are a dozen sets, each with over 100K rows, it takes forever.
So, how does one go about injecting a manual range of values in Excel using a single call C# interop on .net core?
PS: I cannot use any library which directly just modifies xlsx file. Upon injecting data, a bunch of charts and pivot tables are refreshed by Excel, so whatever alternative solution might come up, it needs to act as a full Excel and update all the sheets before saving.
Ok, I found a solution, in case anyone needs it. I have abandoned the data transfer route and leaned on clipboard to do the job. "Copying" tab delimited data and then pasting it in the table inject region did the trick. You will however have to convert the pure .net console project into *-windows targeted project and add OutputType of WinExe and UseWindowsForms to true (or WPF) so that Clipboard class can be available (its not in pure console project). You do not have to create any forms or Run the winForms/WPF though, just let it run as console app in Main.
using var writer = new StringWriter();
var cfg = new CsvConfiguration(CultureInfo.GetCultureInfo("en-us"))
{
HasHeaderRecord = false,
Delimiter = "\t"
};
using var csv = new CsvWriter(writer, cfg);
csv.WriteRecords(data);
var dataObject = new DataObject();
dataObject.SetText(writer.ToString());
System.Windows.Forms.Clipboard.SetDataObject(dataObject, true);
// paste to excel
var table = sheet.ListObjects["CodesTable"];
table.InsertRowRange.PasteSpecial(XlPasteType.xlPasteAll, XlPasteSpecialOperation.xlPasteSpecialOperationAdd);

ParquetWriter not sending all information to blob storage

public async Task UploadParquetFromObjects<T>(string fileName, T objects)
{
var stringJson = JArray.FromObject(objects).ToString();
var parsedJson = ChoJSONReader.LoadText(stringJson);
var desBlob = blobClient.GetBlockBlobClient(fileName);
using (var outStream = await desBlob.OpenWriteAsync(true).ConfigureAwait(false))
using (ChoParquetWriter parser = new ChoParquetWriter(outStream))
{
parser.Write(parsedJson);
}
}
I'm using this code to send some data to a file on an Azure Blob Storage. At first, it worked fine, it created the file, put some information on it and it was readable, but with some investigation, it only write a fraction of the data I send. For example, I send a list of 15 items and it only writes 3. I tried different datasets, with different sizes and composed of different objects, the writer varies on the number of registers written, but it never gets to 100%.
Am I doing something wrong?
This issue is being tracked and addressed in GitHub issues section.
https://github.com/Cinchoo/ChoETL/issues/230
The issue was the input JSON has inconsistent members, hence missing datetime members are set as null by JSON reader. Parquet writer couldn't handle such null datetime values. Applied fix.
Sample fiddle: https://dotnetfiddle.net/PwxNWX
Packages used:
ChoETL.JSON.Core v1.2.1.49 (beta2)
ChoETL.Parquet v1.0.1.23 (beta6)

How to handle issue with some rows in csv files for pipefittings in revit api?

I am facing a weird issue in Revit API c# while working with pipefittings and CSV files. The base model I am working on is a pipefitting where I need to upload a CSV file.
FamilySizeTableManager TableManager = FamilySizeTableManager.GetFamilySizeTableManager(famDoc, famDoc.OwnerFamily.Id);
if (TableManager.IsValidObject)
{
using (Transaction trans = new Transaction(famDoc))
{
trans.Start("import lookup table");
FamilySizeTableErrorInfo errorInfo = new FamilySizeTableErrorInfo();
result = TableManager.ImportSizeTable(famDoc, csvPath, errorInfo);
trans.Commit();
}
}
I added the CSV file name as follows, where tableDimensions is the name of the CSV file:
if (!String.IsNullOrWhiteSpace(tableDimensions))
{
Parameters.SetParameterFamily(famManager, famDoc, "DATA", ParameterType.Text, false, "TABLE_DIMENSIONS", tableDimensions);
}
This CSV file contains 15-20 rows. Up to 10 rows, it works properly, but on the 11 rows I get the error as:
Line is too short.
Can't create the surface of revolution
The weird part is that, when I do this task manually from the Revit UI, it works properly.
So the CSV format seems to be correct, because 10 rows are executing correctly, and the value on this particular row is correct because it works properly on trying it manually via Revit UI.
Did someone face a similar issue that works manually but not programmatically? Also is there a better way to handle this issue or get more details to pinpoint this issue?
Edit:
small extract of the csv
;DN##LENGTH##MILLIMETERS;PN##LENGTH##MILLIMETERS;LMX##LENGTH##MILLIMETERS;DE##LENGTH##MILLIMETERS;M_THICKNESS##LENGTH##MILLIMETERS;I_COAT_THICKNESS##LENGTH##MILLIMETERS;E_COAT_THICKNESS##LENGTH##MILLIMETERS;MASS##MASS##KILOGRAMS;P_THICKNESS##LENGTH##MILLIMETERS;P_SIZE##LENGTH##MILLIMETERS
BBA1;60;10;500;90;6.0;4.0;0;11;16.0;175
BBA2;80;10;500;98;6.0;4.0;0;15;16.0;200
BBA3;100;10;500;118;6.0;4.0;0;18;16.0;220
BBA4;125;10;500;144;6.0;4.0;0;22;16.0;250
BBA5;600;10;550;635;9.9;5.0;0;220;26.0;780
The last row fails but works in Revit UI.

Caching posted data and fall-backs

I'm currently working on a project that has an external site posting xml data to a specified url on our site. My initial thoughts were to first of all save the xml data to a physical file on our server as a backup. I then insert the data into the cache and from then on, all requests for the data will be made to the cache instead of the physical file.
At the moment I have the following:
[HttpPost]
public void MyHandler()
{
// filePath = path to my xml file
// Delete the previous file
if (File.Exists(filePath))
File.Delete(filePath));
using (Stream output = File.OpenWrite(filePath))
using (Stream input = request.InputStream)
{
input.CopyTo(output);
}
// Deserialize and save the data to the cache
var xml = new XmlTextReader(filePath);
var serializer = new XmlSerializer(typeof(MyClass));
var myClass = (MyClass)serializer.Deserialize(xml);
HttpContext.Current.Cache.Insert(myKey,
myClass,
null,
myTimespan,
Cache.NoSlidingExpiration,
CacheItemPriority.Default, null);
}
The issue I have is that I'm always getting exceptions thrown because the file that I'm saving to 'is in use' when I try a second post to update the data.
A colleague suggested using a Mutex class just before I left work on the Friday so I wonder if that is the correct approach here?
Basically I'm just trying to sanity check that this is a good way of managing the data? I can see there's clearly an issue with how I'm writing the data to a file but aside from this, does my approach make sense?
Thanks

write to specific position in .json file + serilaize size limit issue C#

I have a method that retrieves data from a json serialized string and writes it to a .json file using:
TextWriter writer = new StreamWriter("~/example.json");
writer2.Write("{\"Names\":" + new JavaScriptSerializer().Serialize(jsonData) + "}");
data(sample):
{"People":{"Quantity":"4"}, ,"info" :
[{"Name":"John","Age":"22"}, {"Name":"Jack","Age":"56"}, {"Name":"John","Age":"82"},{"Name":"Jack","Age":"95"}]
}
This works perfectly however the jsonData variable has content that is updated frequently. Instead of always deleting and creating a new example.json when the method is invoked,
Is there a way to write data only to a specific location in the file? in the above example say to the info section by appending another {"Name":"x","Age":"y"}?
My reasoning for this is I ran into an issue when trying to serialize a large amount of data using visual studio in C#. I got "The length of the string exceeds the value set on the maxJsonLength propertyā€¯ error. I tried to increase the max allowed size in the web.config using a few suggested methods in this forum but they never worked. As the file gets larger I feel I may run into the same issue again. Any other alternatives are always welcome. Thanks in advance.
I am not aware of a JSON serializer that works with chunks of JSON only. You may try using Json.NET which should work with larger data:
var data = JsonConvert.SerializeObject(new { Names = jsonData });
File.WriteAllText("example.json", data);

Categories