Unable to insert using TableBatchOperation - c#

I'm trying to insert some test values into an Azure table using storage connection string. When I tried to perform insert operation it showing error as cannot convert Entities.SyncJob to Microsoft.Azure.CosmosDB.Table.ITableEntity.
public async Task BackupJobsAsync(List<SyncJob> syncJobs)
{
var tableName = "Table";
var table = await GetCloudTableAsync("connectionString", tableName);
if (!await table.ExistsAsync())
{
await table.CreateIfNotExistsAsync();
}
var backupCount = 0;
var batchSize = 100;
var currentSize = 0;
var groups = syncJobs.GroupBy(x => x.PartitionKey).ToList();
foreach (var group in groups)
{
var batchOperation = new TableBatchOperation();
foreach (var job in group)
{
batchOperation.Insert(job);
if (++currentSize == batchSize)
{
var result = await table.ExecuteBatchAsync(batchOperation);
backupCount += result.Count(x => IsSuccessStatusCode(x.HttpStatusCode));
batchOperation = new TableBatchOperation();
currentSize = 0;
}
}
if (batchOperation.Any())
{
var result = await table.ExecuteBatchAsync(batchOperation);
backupCount += result.Count(x => IsSuccessStatusCode(x.HttpStatusCode));
}
}
}
SyncJob class:
namespace Entities
{
public class SyncJob : ITableEntity
{
public SyncJob()
{
}
public SyncJob(string partitionKey, string rowKey)
{
PartitionKey = partitionKey;
RowKey = rowKey;
}
public string PartitionKey { get; set; }
public string RowKey { get; set; }
public string ABC { get; set; }
public string DEF { get; set; }
public string GHI { get; set; }
public string JKL { get; set; }
}
}
Error Screenshot for reference. Can anyone let me know how can we overcome this? Have tried by casting the value, but the result is same.
UPDATE:
I just noticed that this is using a deprecated package nuget package Microsoft.Azure.Cosmos.Table. How do I update this code to use new package Azure.Data.Tables?

Related

No members mapped for type - CSV Helper

I am new to using CSVHelper and AutoMapper and am getting the following error when trying to:
public async Task<IActionResult> OnPostAsync()
{
if (BolCsv != null)
{
try
{
using (var reader = new StreamReader(BolCsv.OpenReadStream()))
using (var csvr = new CsvReader(reader, System.Globalization.CultureInfo.CurrentCulture))
{
csvr.Configuration.Delimiter = "\t";
csvr.Configuration.HeaderValidated = null;
csvr.Configuration.MissingFieldFound = null;
var bolDtos = csvr.GetRecords<BOLDto>().ToList();
var bols = _mapper.Map<IEnumerable<BOL>>(bolDtos);
await _context.SaveChangesAsync();
return RedirectToPage("./Index");
}
}
var bolDtos = csvr.GetRecords<BOLDto>();
Error: No members are mapped for type 'BOLDto'.
BOLDto:
{
public class BOLDto
{
[Name("BOLNumber")]
public int BOLNumber { get; set; }
[Name("ProductID")]
public int ProductID { get; set; }
[Name("ProductDescription")]
public string ProductDescription { get; set; }
etc...
}
}
BOL.cs:
{
public class BOL
{
public int BOLNumber { get; set; }
public int ProductId { get; set; }
public string ProductDescription { get; set; }
etc...
}
}
As I mentioned Im new to ASP.Net Core AutoMapper, and CSVHelper... how do I solve this issue?

It looks like your BOLDto class has properties, but that error message is the same as I would get if the class had fields instead of properties. So you might want to try CsvHelper.Configuration.MemberTypes.Fields. Also that must be an older version of CsvHelper you are using, because that is not how you would need to set up the configuration in the current version. But it should still work for you to add csvr.Configuration.MemberTypes = CsvHelper.Configuration.MemberTypes.Fields.
void Main()
{
var config = new MapperConfiguration(cfg => cfg.CreateMap<BOLDto, BOL>());
var _mapper = config.CreateMapper();
var csvConfig = new CsvConfiguration(CultureInfo.InvariantCulture)
{
Delimiter = "\t",
HeaderValidated = null,
MissingFieldFound = null,
MemberTypes = CsvHelper.Configuration.MemberTypes.Fields
};
using (var reader = new StringReader("BOLNumber\tProductID\tProductDescription\t\n1\t2\tMy Product"))
using (var csvr = new CsvReader(reader, csvConfig))
{
var bolDtos = csvr.GetRecords<BOLDto>().ToList();
var bols = _mapper.Map<IEnumerable<BOL>>(bolDtos);
}
}
public class BOLDto
{
[Name("BOLNumber")]
public int BOLNumber;
[Name("ProductID")]
public int ProductID;
[Name("ProductDescription")]
public string ProductDescription;
}
public class BOL
{
public int BOLNumber { get; set; }
public int ProductId { get; set; }
public string ProductDescription { get; set; }
}

Mongo time-series inserts using c# produce exception after short time

I want to test if mongo db can have a collection up to 50 000 000 000.
So I insert 10K elements every second using method:
public async Task InsertManyAsync(List<DBRoutLine> list)
{
await _collRouts.InsertManyAsync(list);
}
Data looks like this:
namespace DbLayer.Models
{
public class DBRoutLineMeta
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public string id { get; set; }
public int counter { get; set; }
}
[BsonIgnoreExtraElements]
public class DBRoutLine
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public string id { get; set; }
public DBRoutLineMeta meta { get; set; } = new DBRoutLineMeta();
public DateTime timestamp { get; set; } = DateTime.UtcNow;
public string some_data { get; set; } = DateTime.Now.ToString();
}
}
id members not required actually but I have them, just for testing.
So I've got exception like this:
"A bulk write operation resulted in one or more errors. WriteErrors: [ { Category : "DuplicateKey", Code : 11000, Message : "E11000 duplicate key error collection: TSTest.system.buckets.TSTable dup key: { _id: ObjectId('634e87301297fa65b7df9923') }" } ]."
after sometime. It can be also like this:
"time-series insert failed: TSTest.TSTable :: caused by :: Expected
And it will never recover from error even if I recreate connection to mongo server. Only application restart helps to insert records again.
Test code:
using DbLayer.Models;
using DbLayer.Services;
using MongoDB.Bson;
Console.WriteLine("Hello, World!");
var service = new RoutService();
try
{
CancellationTokenSource tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
var list = new List<DBRoutLine>();
for (int i = 0; i < 10000; i++)
{
DBRoutLine line = new DBRoutLine();
list.Add(line);
}
Task task = Task.Run(async () => {
int max_counter = await service.GetMaxCount();
bool recover = false;
while (!token.IsCancellationRequested)
{
try
{
if (!recover)
{
foreach (DBRoutLine line in list)
{
line.meta.counter = ++max_counter;
line.id = ObjectId.GenerateNewId().ToString();
line.meta.id = line.id;
}
}
var t1 = DateTime.Now;
await service.InsertManyAsync(list);
var t2 = DateTime.Now;
max_counter = await service.GetMaxCount();
var t3 = DateTime.Now;
Console
.WriteLine(
$"{max_counter}->Insert:{(int)(t2 - t1).TotalMilliseconds}, GetMax:{(int)(t3 - t2).TotalMilliseconds}");
recover = false;
}
catch(Exception ex)
{
recover = true;
await Task.Delay(3000);
Console.WriteLine(ex.Message.ToString());
service = new RoutService();
max_counter = await service.GetMaxCount();
}
}
}, token);
Console.WriteLine("Press any key to stop emulation\n");
Console.ReadKey();
tokenSource.Cancel();
Task.WaitAll(task);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
Service code:
using DbLayer.Models;
using MongoDB.Bson;
using MongoDB.Driver;
namespace DbLayer.Services
{
public class RoutService:IDisposable
{
private readonly IMongoCollection<DBRoutLine> _collRouts;
private readonly MongoClient _mongoClient;
private readonly string CollName = "TSTable";
public RoutService(
)
{
var ConnectionString = "mongodb://mongoservice:27017";
_mongoClient = new MongoClient(
ConnectionString);
var mongoDatabase = _mongoClient.GetDatabase(
"TSTest");
var filter = new BsonDocument("name", CollName);
var options = new ListCollectionNamesOptions { Filter = filter };
if (!mongoDatabase.ListCollectionNames(options).Any())
{
var createOptions = new CreateCollectionOptions();
var timeField = nameof(DBRoutLine.timestamp);
var metaField = nameof(DBRoutLine.meta);
createOptions.TimeSeriesOptions =
new TimeSeriesOptions(timeField, metaField, TimeSeriesGranularity.Minutes);
mongoDatabase.CreateCollection(
CollName,
createOptions);
}
_collRouts =
mongoDatabase.GetCollection<DBRoutLine>(
CollName
);
CreateIndexes();
}
private void CreateIndexes()
{
{
IndexKeysDefinition<DBRoutLine> keys =
new IndexKeysDefinitionBuilder<DBRoutLine>()
.Descending(d => d.meta.counter);
var indexModel = new CreateIndexModel<DBRoutLine>(
keys, new CreateIndexOptions()
{ Name = "counter" }
);
_collRouts.Indexes.CreateOneAsync(indexModel);
}
////////////////////////////////////////////////
{
IndexKeysDefinition<DBRoutLine> keys =
new IndexKeysDefinitionBuilder<DBRoutLine>()
.Ascending(d => d.meta.id);
var indexModel = new CreateIndexModel<DBRoutLine>(
keys, new CreateIndexOptions()
{ Name = "id" }
);
_collRouts.Indexes.CreateOneAsync(indexModel);
}
}
public async Task InsertManyAsync(List<DBRoutLine> list)
{
await _collRouts.InsertManyAsync(list);
}
public async Task<int> GetMaxCount()
{
var last = await _collRouts
.Find(i=> i.meta.counter > 0)
.SortByDescending( i => i.meta.counter).FirstOrDefaultAsync();
if (last == null)
{
return 0;
}
return last.meta.counter;
}
public void Dispose()
{
}
}
}
project repository:
github.com/pruginkad/TestMongo

Ok, I found the bug. I changed timestamp only once when I created List of documents.
in this code:
foreach (DBRoutLine line in list)
{
line.meta.counter = ++max_counter;
line.id = ObjectId.GenerateNewId().ToString();
line.meta.id = line.id;
line.timestamp = DateTime.UtcNow;//missing line
}
I had to update timestamp, my mistake.
Anyway it's kind of strange that exception happen after every 17M documents and disappear after restart of mongo db

Error while reading data from azure table storage

I'm writing a code to read values from a table storage. The code is similar to printing nodes in a tree level by level.
eg:
root
Level1child1 -> Level1child2 -> Level1child3
string tablename = "<table-name">;
string storageAccountName = "<storage-account-name";
var baseurl = #$"https://{storageAccountName}.table.core.windows.net/{tableName}()";
var sastoken = getAccountSASToken();
string filter = #"&$filter=PartitionKey%20eq%20'123'%20and%20RowKey%20eq%20'abc'";
baseurl = $"{baseurl}{sastoken}{filter}";
var data = HttpHelper.GetForOData(baseurl);
var responseData = data.Data.Replace(".", "_");
var odata = JsonConvert.DeserializeObject<ODataResponse>(responseData);
Queue<int> strQ = new Queue<int>();
Console.WriteLine(odata.value[0].Email);
strQ.Enqueue(odata.value[0].TreeNodeID);
while (strQ.Any())
{
var url = #$"https://{storageAccountName}.table.core.windows.net/{tableName}()";
var token = _tableStorageRepository.GetAccountSASToken();
filter = #"&$filter=ParentNodeId%20eq%20" + strQ.Peek();
url = $"{url}{token}{filter}";
data = HttpHelper.GetForOData(url);
responseData = data.Data.Replace(".", "_");
odata = JsonConvert.DeserializeObject<ODataResponse>(responseData);
foreach (var m in odata?.value)
{
Console.WriteLine(m.Email);
strQ.Enqueue(m.TreeNodeID);
}
strQ.Dequeue();
}
public class ODataResponse
{
public string odata_metadata { get; set; }
public List<ODatavalue> value { get; set; }
}
public class ODatavalue
{
public string odata_type { get; set; }
public string odata_id { get; set; }
public string odata_etag { get; set; }
public string odata_editLink { get; set; }
public string RowKey { get; set; }
public string Email { get; set; }
public int ParentNodeID { get; set; }
public int TreeNodeID { get; set; }
}
Code for HttpHelper class: https://github.com/xyz92/httphelper/blob/master/HttpHelper.cs
The first time when I ran this code, it only printed root node.
The second time when I ran this code, it printed root node and Level1child1 node.
For the next runs, it printed root node, Level1child1 node & Level1child2 node.
The last node Level1child3 node is getting printed very rarely on some runs.
What am I missing in this code?
UPDATE:
Sample responseData:
{
"odata_metadata": "https://<storage-account-name>_table_core_windows_net/$metadata#<table-name>",
"value": [{
"odata_type": "<storage-account-name>_<table-name>",
"odata_id": "https://<storage-account-name>_table_core_windows_net/<table-name>(PartitionKey='123',RowKey='abc')",
"odata_etag": "W/\"datetime'2020-09-01T16%3A34%3A21_3342187Z'\"",
"odata_editLink": "<table-name>(PartitionKey='123',RowKey='abc')",
"PartitionKey": "123",
"RowKey": "abc",
"Timestamp#odata_type": "Edm_DateTime",
"Timestamp": "2020-09-01T16:34:21_3342187Z",
"Email": "email",
"ParentNodeID": 1,
"TreeNodeID": 2
}
]
}
Table columns:
Sample data in Table:
Sample outputs while running code:

According to my test, I cannot reproduce your issue in my environment. My test is as below
Table columns:
Sample data in table
Code I use your HttpHelper class to send request
class Program
{
static async Task Main(string[] args)
{
string storageAccountName = "andyprivate" ;
string tableName = "test";
var baseurl = #$"https://{storageAccountName}.table.core.windows.net/{tableName}()";
var sastoken = "";
string filter = #"&$filter=PartitionKey%20eq%20'123'%20and%20RowKey%20eq%20'abc'";
baseurl = $"{baseurl}{sastoken}{filter}";
var data = HttpHelper.GetForOData(baseurl);
var responseData = data.Data.Replace(".", "_");
var odata = JsonConvert.DeserializeObject<ODataResponse>(responseData);
Queue<int> strQ = new Queue<int>();
Console.WriteLine("----root----");
Console.WriteLine(odata.value[0].Email);
strQ.Enqueue(odata.value[0].TreeNodeID);
while (strQ.Any())
{
int i = 0;
var url = #$"https://{storageAccountName}.table.core.windows.net/{tableName}()";
var token = "";
filter = #$"&$filter=ParentNodeID eq {strQ.Peek()}";
url = $"{ url}{token}{filter}";
data = HttpHelper.GetForOData(url);
responseData = data.Data.Replace(".", "_");
odata = JsonConvert.DeserializeObject<ODataResponse>(responseData);
Console.WriteLine($"----TreeNode{strQ.Peek()}----");
foreach (var m in odata?.value)
{
if (i == 0) {
strQ.Enqueue(m.TreeNodeID);
i = 1;
}
Console.WriteLine(m.Email);
}
strQ.Dequeue();
}
Console.ReadLine();
}
}
public class ODataResponse
{
public string odata_metadata { get; set; }
public List<ODatavalue> value { get; set; }
}
public class ODatavalue
{
public string odata_type { get; set; }
public string odata_id { get; set; }
public string odata_etag { get; set; }
public string odata_editLink { get; set; }
public string RowKey { get; set; }
public string Email { get; set; }
public int ParentNodeID { get; set; }
public int TreeNodeID { get; set; }
}
Update
My test code
Rest API (I use your HttpHelper class to send request)
class Program
{
static async Task Main(string[] args)
{
string storageAccountName = "andyprivate" ;
string tableName = "test";
var baseurl = #$"https://{storageAccountName}.table.core.windows.net/{tableName}()";
var sastoken = "";
string filter = #"&$filter=PartitionKey%20eq%20'123'%20and%20RowKey%20eq%20'abc'";
baseurl = $"{baseurl}{sastoken}{filter}";
var data = HttpHelper.GetForOData(baseurl);
var responseData = data.Data.Replace(".", "_");
var odata = JsonConvert.DeserializeObject<ODataResponse>(responseData);
Queue<int> strQ = new Queue<int>();
Console.WriteLine("----root----");
Console.WriteLine(odata.value[0].Email);
strQ.Enqueue(odata.value[0].TreeNodeID);
while (strQ.Any())
{
int i = 0;
var url = #$"https://{storageAccountName}.table.core.windows.net/{tableName}()";
var token = "";
filter = #$"&$filter=ParentNodeID eq {strQ.Peek()}";
url = $"{ url}{token}{filter}";
data = HttpHelper.GetForOData(url);
responseData = data.Data.Replace(".", "_");
odata = JsonConvert.DeserializeObject<ODataResponse>(responseData);
Console.WriteLine($"----TreeNode{strQ.Peek()}----");
foreach (var m in odata?.value)
{
if (i == 0) {
strQ.Enqueue(m.TreeNodeID);
i = 1;
}
Console.WriteLine(m.Email);
}
strQ.Dequeue();
}
Console.ReadLine();
}
}
public class ODataResponse
{
public string odata_metadata { get; set; }
public List<ODatavalue> value { get; set; }
}
public class ODatavalue
{
public string odata_type { get; set; }
public string odata_id { get; set; }
public string odata_etag { get; set; }
public string odata_editLink { get; set; }
public string RowKey { get; set; }
public string Email { get; set; }
public int ParentNodeID { get; set; }
public int TreeNodeID { get; set; }
}
SDK. I use the package Microsoft.Azure.Cosmos.Table
string storageAccountName = "andyprivate";
string accountKey ="";
string tableName = "test";
CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials(storageAccountName, accountKey), true);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient(new TableClientConfiguration());
CloudTable table = tableClient.GetTableReference(tableName);
TableOperation retrieveOperation = TableOperation.Retrieve<CustomEntity>("123", "abc");
TableResult result = await table.ExecuteAsync(retrieveOperation);
CustomEntity entity = result.Result as CustomEntity;
Queue<int> strQ = new Queue<int>();
Console.WriteLine("----root----");
Console.WriteLine(entity.Email);
strQ.Enqueue(entity.TreeNodeID);
while (strQ.Any()) {
int i = 0;
TableQuery<CustomEntity> query = new TableQuery<CustomEntity>()
.Where(
TableQuery.GenerateFilterConditionForInt("ParentNodeID", QueryComparisons.Equal,strQ.Peek())
);
Console.WriteLine($"----TreeNode{strQ.Peek()}----");
foreach (CustomEntity m in table.ExecuteQuery(query) ) {
Console.WriteLine(m.Email);
if (i == 0) {
strQ.Enqueue(m.TreeNodeID);
i = 1;
}
}
strQ.Dequeue();
}

Composite join on two CSV files in C#

Starting from a table of daily fruit prices
fruits.csv
Day,Name,Kind,Price
2019-09-04,"apple","red",63.09
2019-09-04,"apple","yellow",52.14
2019-09-04,"orange","navel",41.18
2019-09-04,"orange","blood",41.18
2019-09-03,"apple","red",63.07
2019-09-03,"apple","yellow",52.11
2019-09-03,"orange","navel",41.13
2019-09-03,"orange","blood",41.13
I'd like to insert the reference prices by name and kind
fruit_ref_prices.csv
Name,Kind,Reference_Price
"apple","red",60.00
"apple","yellow",50.00
"orange","navel",40.00
"orange","blood",42.00
to result in the following table
Day,Name,Kind,Price,Reference_Price
2019-09-04,"apple","red",63.09,60.00
2019-09-04,"apple","yellow",52.14,50.00
2019-09-04,"orange","navel",41.18,40.00
2019-09-04,"orange","blood",41.18,42.00
2019-09-03,"apple","red",63.07,60.00
2019-09-03,"apple","yellow",52.11,50.00
2019-09-03,"orange","navel",41.13,40.00
2019-09-03,"orange","blood",41.13,42.00
The solution should be simple using C#'s built-in SQL-like syntax, and I'm sure the answer lies in one of the following tutorial pages:
Join clause
Perform custom join operations
Join by using composite keys
but I'm having a hard time identifying the syntax of this language.
In my attempt below instead of writing
join fruit_ref in fruit_refs on fruit.name equals fruit_ref.name
I should be able to write
join fruit_ref in fruit_refs on fruit.name equals fruit_ref.name
and fruit.kind equals fruit_ref.kind
but the Boolean expression is not accepted. Why?
My attempt is:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data;
using System.IO;
namespace MyConsoleApplication
{
class Program
{
const string root = #"c:\path\to\here\";
const string file1_in = root + #"fruits.csv";
const string file2_in = root + #"fruit_ref_prices.csv";
static void Main(string[] args)
{
Fruit_Basket fruit_basket = new Fruit_Basket(file1_in, file2_in);
fruit_basket.PrintFruits();
}
}
public class Fruit
{
public DateTime day { get; set; }
public string name { get; set; }
public string kind { get; set; }
public decimal price { get; set; }
public Fruit(DateTime newFruit_day,
string newFruit_name,
string newFruit_kind,
decimal newFruit_price)
{
this.day = newFruit_day;
this.name = newFruit_name;
this.kind = newFruit_kind;
this.price = newFruit_price;
}
}
public class Fruit_Ref
{
public string name;
public string kind;
public decimal reference_price;
public Fruit_Ref(string newName, string newKind, decimal newRef_Price)
{
this.name = newName;
this.kind = newKind;
this.reference_price = newRef_Price;
}
}
public class Fruit_Basket {
public List<Fruit> fruits { get; set; }
public List<Fruit_Ref> fruit_refs { get; set; }
public Fruit_Basket(string file1_in, string file2_in) {
build_fruit_list(file1_in);
build_fruit_ref_list(file2_in);
}
public void build_fruit_list(string file_in)
{
fruits = new List<Fruit>();
int count = 0;
StreamReader reader = new StreamReader(file_in);
string line = "";
while ((line = reader.ReadLine()) != null)
{
if (++count > 1)
{
string[] splitLine = line.Split(new char[] { ',' }).ToArray();
var newFruit_day = DateTime.Parse(splitLine[0]);
var newFruit_name = splitLine[1];
var newFruit_kind = splitLine[2];
var newFruit_price = decimal.Parse(splitLine[3]);
Fruit newFruit = new Fruit(newFruit_day,
newFruit_name,
newFruit_kind,
newFruit_price);
fruits.Add(newFruit);
}
}
reader.Close();
}
public void build_fruit_ref_list(string file_in)
{
fruit_refs = new List<Fruit_Ref>();
int count = 0;
StreamReader reader = new StreamReader(file_in);
string line = "";
while ((line = reader.ReadLine()) != null)
{
if (++count > 1)
{
string[] splitLine = line.Split(new char[] { ',' }).ToArray();
var newFruit_name = splitLine[0];
var newFruit_kind = splitLine[1];
var newFruit_ref_price = decimal.Parse(splitLine[2]);
Fruit_Ref newFruit_ref = new Fruit_Ref(newFruit_name,
newFruit_kind,
newFruit_ref_price);
fruit_refs.Add(newFruit_ref);
}
}
reader.Close();
}
public void PrintFruits()
{
var innerJoinQuery =
from fruit in fruits
join fruit_ref in fruit_refs on fruit.name equals fruit_ref.name
select new { Day = fruit.day, Name = fruit.name, Kind = fruit.kind,
Price = fruit.price, Reference_Price = fruit_ref.reference_price };
Console.WriteLine($#"""Date"",""Name"",""Kind"",""Price"",""Ref Price""");
foreach (var i in innerJoinQuery)
{
Console.WriteLine($#"{i.Day},{i.Kind},{i.Price},{i.Reference_Price}");
}
}
}
}

You could also refactor your code to use the CsvHelper NuGet package for reading/writing CSV files.
First, You can make these classes to reflect the fruits, fruit references and final fruit structure.
public class Fruit
{
public string Day { get; set; }
public string Name { get; set; }
public string Kind { get; set; }
public string Price { get; set; }
}
public class FruitReferencePrice
{
public string Name { get; set; }
public string Kind { get; set; }
public string Reference_Price { get; set; }
}
public class FruitFinal
{
public string Day { get; set; }
public string Name { get; set; }
public string Kind { get; set; }
public string Price { get; set; }
public string ReferencePrice { get; set; }
public override string ToString()
{
return $"Day={Day},Name={Name},Kind={Kind},Price={Price},Reference_Price={ReferencePrice}";
}
}
Then you can make two methods to return the rows of each CSV file into List<Fruit> and List<FruitReferencePrice>.
private static IEnumerable<Fruit> BuildFruitList(string csvFilePath)
{
if (!File.Exists(csvFilePath))
{
throw new FileNotFoundException("Could not locate CSV at path " + csvFilePath, csvFilePath);
}
try
{
using var fileReader = File.OpenText(csvFilePath);
using var csv = new CsvReader(fileReader);
return csv.GetRecords<Fruit>().ToList();
} catch (Exception ex)
{
Console.WriteLine(ex.Message);
return Enumerable.Empty<Fruit>();
}
}
private static IEnumerable<FruitReferencePrice> BuildFruitReferenceList(string csvFilePath)
{
if (!File.Exists(csvFilePath))
{
throw new FileNotFoundException("Could not locate CSV at path " + csvFilePath, csvFilePath);
}
try
{
using var fileReader = File.OpenText(csvFilePath);
using var csv = new CsvReader(fileReader);
return csv.GetRecords<FruitReferencePrice>().ToList();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
return Enumerable.Empty<FruitReferencePrice>();
}
}
Then you can perform a grouped join and output the merged result.
var path1 = "PATH\\fruits.csv";
var path2 = "PATH\\fruit_ref_prices.csv";
var fruitList = BuildFruitList(path1);
var fruitReferencePrices = BuildFruitReferenceList(path2);
var groupedJoin = from fruit in fruitList
join fruit_ref in fruitReferencePrices
on new { fruit.Name, fruit.Kind } equals new { fruit_ref.Name, fruit_ref.Kind }
select new FruitFinal
{
Day = fruit.Day,
Name = fruit.Name,
Kind = fruit.Kind,
Price = fruit.Price,
ReferencePrice = fruit_ref.Reference_Price
};
foreach (var fruit in groupedJoin)
{
Console.WriteLine(fruit.ToString());
}
Merged results:
Day=2019-09-04,Name=apple,Kind=red,Price=63.09,Reference_Price=60.00
Day=2019-09-04,Name=apple,Kind=yellow,Price=52.14,Reference_Price=50.00
Day=2019-09-04,Name=orange,Kind=navel,Price=41.18,Reference_Price=40.00
Day=2019-09-04,Name=orange,Kind=blood,Price=41.18,Reference_Price=42.00
Day=2019-09-03,Name=apple,Kind=red,Price=63.07,Reference_Price=60.00
Day=2019-09-03,Name=apple,Kind=yellow,Price=52.11,Reference_Price=50.00
Day=2019-09-03,Name=orange,Kind=navel,Price=41.13,Reference_Price=40.00
Day=2019-09-03,Name=orange,Kind=blood,Price=41.13,Reference_Price=42.00

Please change the equals clause as on new { fruit.name, fruit.kind } equals new { fruit_ref.name, fruit_ref.kind }
Why you require this
The query has two anonymous types (one for left table and one for right table). So to compare those anonymous types, the linq statement should use new keyword
Query :
var innerJoinQuery = from fruit in fruits
join fruit_ref in fruit_refs on new { fruit.name, fruit.kind } equals new { fruit_ref.name, fruit_ref.kind }
select new { Day = fruit.day, Name = fruit.name, Kind = fruit.kind,
Price = fruit.price, Reference_Price = fruit_ref.reference_price };

Trying concurrent updates(AddToSet) to collection in mongodb and its not working correctly

I have FileAggregateReport class:
public class FileAggregateReport
{
public Guid FileId { get; set; }
public Guid CorrelationId { get; set; }
public int NumberOfSupportedEngines { get; set; }
public int EnginesCompleted { get; set; }
public string SHA256 { get; set; }
public DateTime ScanDate { get; set; }
public List<EngineReport> EngineReports { get; set; }
}
and i am trying to add to the list EngineReports reports as update operation.
my insert report function is:(FileAggregateReportCollection is IMongoCollection<FileAggregateReport>)
public void InsertNewEngineReport(Guid CorrelationId, Guid FileId, EngineReport engineReport, string sha256, int numberOfSupportedEngines)
{
var update = Builders<FileAggregateReport>.Update
.Inc(report => report.EnginesCompleted, 1)
.Set(report => report.CorrelationId, CorrelationId)
.Set(report => report.FileId, FileId)
.SetOnInsert(report => report.ScanDate, engineReport.ScanDate)
.SetOnInsert(report => report.SHA256, sha256)
.AddToSet( report => report.EngineReports, engineReport)
.Set(report => report.NumberOfSupportedEngines, numberOfSupportedEngines);
UpdateOptions options = new UpdateOptions { IsUpsert = true };
try
{
_mongoDbHelper.FileAggregateReportCollection.UpdateOne(e => e.FileId == FileId, update, options);
}
catch (Exception e)
{
throw new StateRepositoryException(e.Message);
}
}
}
when i'm doing test like this:
Guid cor = Guid.NewGuid();
Guid fileID = Guid.NewGuid();
EngineReport report = new EngineReport()
{
FileId = fileID
};
_fileResultRepository.InsertNewEngineReport(cor, fileID, report, "dfsdfsdf", 15);
Task[] tasks = new Task[5];
for (int i = 0; i < 5; i++)
{
Task task = Task.Run(() =>
_fileResultRepository.InsertNewEngineReport(cor, fileID, new EngineReport()
{
FileId = fileID,
EngineType = EngineType.Clamav,
MaliciousScore = 70,
ScanDate = DateTime.Now,
ShortSummary = "ATD"
}, "dfsdfsdf", 5));
tasks[i]=task;
}
Task.WaitAll(tasks);
but in my result of this test i have in EnginesReport only 3 element and its need to be 6.
And sometimes i got E11000 duplicate key error collection: FileResultAggregationRepository.FileAggregateReport index: _id_ dup key: { : BinData(3, 2A03C999127FDF4AAF293C5C086BEC66) but i cannot reproduce this.

I saw from mongodb docs the definition of AddToSet=> "The $addToSet operator adds a value to an array unless the value is already present, in which case $addToSet does nothing to that array."
I added new Id for FileAggregateReport because FileId was identical to all EngineReports, after i changed that in my test i got 6 EngineReports as i expected.

Categories