We Keep Coding

Data Context error when saving data to database .NET 5.0 [duplicate]

This question already has answers here:
DbContextOptionsBuilder.EnableSensitiveDataLogging Doesn't Do Anything
(3 answers)
Closed last year.
I have a little problem saving data in the database. In a given method, I make a request to an API, serialize the data, and try to save to the database, as shown in the image, but I end up getting an error referring to the application's DataContext, saying:
System.InvalidOperationException: The instance of entity type 'Launch ' cannot be tracked because another instance with the same key value for {'id'} is already being tracked. When attaching existing entities, ensure that only one entity instance with a given key value is attached.
Consider using 'DbContextOptionsBuilder.EnableSensitiveDataLogging' to see the conflicting key values.
Any tips on how to resolve this issue?
Save Database Method
using System.Collections.Generic;
using System.Threading.Tasks;
using Newtonsoft.Json;
using RestSharp;
using SpaceFlightApiChallenge.Data.Context;
using SpaceFlightApiChallenge.Domain.Entities;
namespace SpaceFlightApiChallenge.Data
{
public class DatabaseImport
{
private readonly SpaceFlightContext _context;
public DatabaseImport(SpaceFlightContext context)
{
_context = context;
}
public async Task ImportData()
{
string url = "https://api.spaceflightnewsapi.net/v3/";
var client = new RestClient(url);
var apiRequest = new RestRequest("articles", Method.Get);
apiRequest.AddHeader("Accept", "application/json");
var response = await client.ExecuteAsync(apiRequest);
var content = response.Content;
var articles = JsonConvert.DeserializeObject<List<Article>>(content);
_context.Articles.AddRange(articles);
await _context.SaveChangesAsync();
}
}
}
Article Class
using System.Collections.Generic;
namespace SpaceFlightApiChallenge.Domain.Entities
{
public class Article
{
public int id { get; set; }
public bool featured { get; set; }
public string title { get; set; }
public string url { get; set; }
public string imageUrl { get; set; }
public string newsSite { get; set; }
public string summary { get; set; }
public string publishedAt { get; set; }
public List<Launch> launches { get; set; }
public List<Event> events { get; set; }
}
}
Launch Class
namespace SpaceFlightApiChallenge.Domain.Entities
{
public class Launch
{
public string id { get; set; }
public string provider { get; set; }
}
}
Event Class
namespace SpaceFlightApiChallenge.Domain.Entities
{
public class Event
{
public string id { get; set; }
public string provider { get; set; }
}
}
EDIT: I believe the whole problem is with the id's. All objects that the external API returns me (Article, event, launch) have their own id, but when they enter the database, EF Core wants to assign a new id, due to the identity property. I don't need to change the id's, the data must be saved in the database as it comes from the API, so that later I can consult them. Here's the API link from where I'm getting the data: https://api.spaceflightnewsapi.net/v3/documentation

Maybe your list contains duplicates. EF does not allow to track entites with same key. If you have that error, try
_context.Entry(<entity>).State = EntityState.Detached;

I can see in your Article class
[DatabaseGenerated(DatabaseGeneratedOption.None)]
public int id { get; set; }
Try to change it
[Key]
public int id { get; set; }
I think the promblem is a primary key, but for testing pourposes try to add one by one , to see what which item has the problem.Try it using a debugger, and see is the first item will get an exception or not
foreach(var article in articles)
{
context.Articles.Add (article);
context.SaveChanges();
}

I'd say it's because the external service is giving you repeated IDs, and your JSON deserialize routine is just making new objects with the same ID and attaching them to the object graph. This means EF ends up receiving something like:
{
id: 13479,
title: Engineers taking more time ...
events: [ { id: 482 } ] //object at memory address 1234
},
{
id: 13477,
title: NASA takes break in JWST deployment ...
events: [ { id: 482 } ] //object at memory address 4567
}
Two different Event objects with the same ID; EF would expect those to be the same object instance
Whether you can get the JSON deser routine to fix this up (by e.g. keeping a dictionary of seen-before objects based on ID) without a custom deser, I don't know*..
..but it should be possible to fix the objects up so the graph entities are unique:
var events = articles.SelectMany(a => a.Events).ToLookup(e => e.Id);
var launches = articles.SelectMany(a => a.Launches).ToLookup(l=> l.Id);
foreach(var a in articles){
a.Events = a.Events.Select(e => events[e.Id].First()).ToList();
a.Launches = a.Launches.Select(l => launches[l.Id].First()).ToList();
}
Turning the events into a Lookup on id (something like a Dictionary<int, List<Event>>) will group up all the different event objects into enumerations accessible by id. This can then be used to recreate an event list where all the events within refer to a common Event. If we enumerate the event list, pull the id out of the Event we find in the list and use that to lookup the First event in the lookup, it points everything to the same event instance for that id
It means you have a lookup called events that is:
events -> lookup of [ { id:482 /*mem address 1234*/ }, {id: 482 /*mem address 4567*/ } ]
And where you once had an Article 13477 whose Events had one Event (with id 482, at memory address 4567), that list is replaced with an event list whose event is again 482, but this time pointing to the Event at memory address 1234
{
id: 13479,
title: Engineers taking more time ...
events: [ { id: 482 } ] //object at memory address 1234: first element in lookup 482
},
{
id: 13477,
title: NASA takes break in JWST deployment ...
events: [ { id: 482 } ] //object at memory address 1234: first element in lookup 482
}
Now.. I hope that these events aren't repeated across invocations of the download (i.e. next week there will be another different article that also has id 482), because if they do you'll probably get a primary key violation.
The solution to that one is probably going to be to extend the loop that is patching up the object graph so that it first looks up in the DB the event with 482, and put that one into the Article (and then fall back to a new event 482 if the DB doesn't know it)
Note; you can use a [JsonProperty("json name here")] attribute to declare the name in json vs the name in C#, so you don't have to have C# that violates normal PascalCase convention for its props
Here's a bunch of code generated from http://quicktype.io (no affiliation) that is C# naming convention square, and deser's your JSON. You can blend it with your entities if you want (my code block above uses PascalCase syntax derived from this set of deser classes, or if you don't plan to change your deser classes you can adjust the code above to camel case) :
// <auto-generated />
//
// To parse this JSON data, add NuGet 'Newtonsoft.Json' then do:
//
// using SpaceX;
//
// var article = Article.FromJson(jsonString);
namespace SpaceX
{
using System;
using System.Collections.Generic;
using System.Globalization;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
public partial class Article
{
[JsonProperty("id")]
public long Id { get; set; }
[JsonProperty("title")]
public string Title { get; set; }
[JsonProperty("url")]
public Uri Url { get; set; }
[JsonProperty("imageUrl")]
public Uri ImageUrl { get; set; }
[JsonProperty("newsSite")]
public string NewsSite { get; set; }
[JsonProperty("summary")]
public string Summary { get; set; }
[JsonProperty("publishedAt")]
public DateTimeOffset PublishedAt { get; set; }
[JsonProperty("updatedAt")]
public DateTimeOffset UpdatedAt { get; set; }
[JsonProperty("featured")]
public bool Featured { get; set; }
[JsonProperty("launches")]
public List<Launch> Launches { get; set; }
[JsonProperty("events")]
public List<Event> Events { get; set; }
}
public partial class Event
{
[JsonProperty("id")]
public long Id { get; set; }
[JsonProperty("provider")]
public string Provider { get; set; }
}
public partial class Launch
{
[JsonProperty("id")]
public Guid Id { get; set; }
[JsonProperty("provider")]
public string Provider { get; set; }
}
public partial class Article
{
public static List<Article> FromJson(string json) => JsonConvert.DeserializeObject<List<Article>>(json, SpaceX.Converter.Settings);
}
public static class Serialize
{
public static string ToJson(this List<Article> self) => JsonConvert.SerializeObject(self, SpaceX.Converter.Settings);
}
internal static class Converter
{
public static readonly JsonSerializerSettings Settings = new JsonSerializerSettings
{
MetadataPropertyHandling = MetadataPropertyHandling.Ignore,
DateParseHandling = DateParseHandling.None,
Converters =
{
new IsoDateTimeConverter { DateTimeStyles = DateTimeStyles.AssumeUniversal }
},
};
}
}
* I expect it would be possible but I've never deser'd directly to a DB entity (nor would i recommend to do so; I'd generally have one set of Dtos for API shuttling and another set of entities for DB work)
Edit:
Actually, try this, which is a slight rethink of the above:
//your code
var articles = JsonConvert.DeserializeObject<List<Article>>(content);
//my code
foreach(var a in articles){
a.Events = a.Events.Select(e => _context.Events.Find(e.Id) ?? e).ToList();
a.Launches = a.Launches.Select(l => _context.Launches.Find(l.Id) ?? l).ToList();
_context.Articles.Add(a);
}
//your code
//_context.Articles.AddRange(articles); - not needed now
Ok, so that above is what I mean with the tweaks : after you deserialize the json you have articles with repeated same-id events and launches. For each article we'll rebuild the events. This code also incorporates the idea of "look up the event by id in the db first" - using Find should be fairly quick as it will first look locally for the entity- if it has been downloaded before then it is returned from local cache in the context. If Find doesn't get the item locally it hits the db, if that also returns null then we use the event/launch that we're currently processing. This should then become the event that Find will find next time (because we add to the context soon after) which resolves the duplicate entity problem by reusing the one the context knows about
If that doesn't work as expected, this one should:
var events = articles.SelectMany(a => a.Events).ToLookup(e => e.Id);
var launches = articles.SelectMany(a => a.Launches).ToLookup(l=> l.Id);
foreach(var a in articles){
a.Events = a.Events.Select(e => _context.Events.Find(e.Id) ?? events[e.Id].First()).ToList();
a.Launches = a.Launches.Select(l => _context.Launches.Find(l.Id) ?? launches[l.Id].First()).ToList();
_context.Articles.Add(a);
}
//your code
//_context.Articles.AddRange(articles); - not needed now

Mongodb collection as dynamic

I have an application that has two similar but different objects and I want to store those objects in the same collection. What is the best way to do this? And how can I query this collection?
Today my collections is represented by:
public IMongoCollection<Post> Posts
{
get
{
return _database.GetCollection<Post>("posts");
}
}
And I have this class:
public class Post
{
public string Id { get; set; }
public string Message { get; set; }
}
public class NewTypePost
{
public string Id { get; set; }
public string Image { get; set; }
}
So, today I just can save and query using Post class. Now I want to store and retrive the both classes, Post and NewTypePost.
I tried to change the class type from Post to dynamic. But when I did this, I could not query the collections.

MongoDB .NET driver offers few possibilites in such cases:
Polymorphism
You can build a hierarchy of classes and MongoDB driver will be able to determine a type of an object it gets retrieved from the database:
[BsonKnownTypes(typeof(Post), typeof(NewTypePost))]
public abstract class PostBase
{
[BsonId]
public string Id { get; set; }
}
public class Post: PostBase
{
public string Message { get; set; }
}
public class NewTypePost: PostBase
{
public string Image { get; set; }
}
MongoDB driver will create additional field _t in every document which will represent corresponding class.
Single Class
You can still have Post class and use BsonIgnoreIfNull attribute to avoid serialization exception. MongoDB .NET driver will set those properties to null if they don't exist in your database.
public class Post
{
[BsonId]
public string Id { get; set; }
[BsonIgnoreIfNull]
public string Message { get; set; }
[BsonIgnoreIfNull]
public string Image { get; set; }
}
BsonDocument
You can also drop strongly-typed approach and use BsonDocument class which is dynamic dictionary-like structure that represents your Mongo documents
var collection = db.GetCollection<BsonDocument>("posts");
More details here
dynamic
Specifying dynamic as generic parameter of ICollection you should get a list of ExpandoObject that will hold all the values you have in your database.
var collection = db.GetCollection<dynamic>("posts");
var data = collection.Find(Builders<dynamic>.Filter.Empty).ToList();
var firstMessage = data[0].Message; // dynamically typed code

Suppose I have the next conn to a test database:
var mongoClient = new MongoClient(new MongoClientSettings
{
Server = new MongoServerAddress("localhost"),
});
var database = mongoClient.GetDatabase("TestDb");
Then I can do something like:
var col = database.GetCollection<Post>("posts");
var col2 = database.GetCollection<NewTypePost>("posts");
To get two different instances of IMongoCollection but pointing to the same collection in the database. Further I am able to save to each collection in the usual way:
col.InsertOne(new Post { Message = "m1" });
col2.InsertOne(new NewTypePost { Image = "im1" });
Then, I'm also able to query from those collection base on the specific fields:
var p1= col.Find(Builders<Post>.Filter.Eq(x=>x.Message, "m1")).FirstOrDefault();
var p2 =col2.Find(Builders<NewTypePost>.Filter.Eq(x=>x.Image, "im1")).FirstOrDefault();
Console.WriteLine(p1?.Message); // m1
Console.WriteLine(p2?.Image); // im1
I don't know if that's what you want but it uses the same collection. BTW, change the Id properties to be decorated with [BsonId, BsonRepresentation(BsonType.ObjectId)]. Hope it helps.

Use the BsonDocument data type. It can do all of that. BsonDocument and dynamic back and forth is very convenient.
public class CustomObject{
public long Id{get;set;}
public string Name{get;set;}
public List<(string,object)> CollectionDynamic{get;set;}
}
// inserted in mongo
//public class CustomObject_in_Db{
// public long Id {get;set;}
// public string Name {get;set;}
// public string field2 {get;set;}
// public string field3 {get;set;}
// public string field4 {get;set;}
// public string field5 {get;set;}
// }
// something code... mapper(config)
Automapper.Mapper.CreateMap<BsonDocument,CustomObject>()
.ForMember(dest=>dest.Id, a=>a.MapFrom(s=>s.Id.GetValue(nameof(CustomObject.Id)).AsInt64)
.ForMember(dest=>dest.Name, a=>a.MapFrom(s=>s.Id.GetValue(nameof(CustomObject.Name)).AsString)
.ForMember(dest=>dest.CollectionDynamic, a=>a.MapFrom(s=>_getList(s));
// .......
private List<(string, object)> _getList(BsonDocument source){
return source.Elements.Where(e=>!typeof(CustomObject).GetProperties().Select(s=>s.Name).Any(a=>a ==e.Name)).Select(e=>e.Name, BsonTryMapper.MapToDotNetValue(e.Value)));
}

LINQ convert SQL server Datetime to string using a DTO

I've been tasked to add a page to an API that we didn't build but are tasked to work on. The API is using C# MVC5. I don't really know MVC5, and but I'm attempting to keep the same design pattern that the rest of the API is using. I have to add functionality that will allow a user on the front end to upload a file to a server that will be processed for inserting into a SQL Server DB. This functionality will also return a list of all the files names and status of the imported files from a table in the DB.
The issue I'm having is converting a Datetime to a string in the LINQ query that is pulling the list of files.
I've attempted to try using this answer LINQ convert DateTime to string, but with no luck.
This is what I have so far, I've marked the line that is causing the issue:
[Route("ImportLogs", Name ="GetImportLogs")]
[HttpGet]
[ResponseType(typeof(List<IHttpActionResult>))]
public IHttpActionResult GetImportLogs()
{
var query =
from dbitem in db.ImportLogs
orderby dbitem.import_log_id
select new ImportLogDto()
{
Id = dbitem.import_log_id,
FileName = dbitem.import_file_name,
ImportTimeStamp = dbitem.import_timeStamp,
ImportStatus = dbitem.import_status,
ImportMessage = dbitem.import_message
};
query.ToList()
.Select(o => new ImportLogDto
{
Id = o.Id,
FileName = o.FileName,
ImportMessage = o.ImportMessage,
ImportStatus = o.ImportStatus,
ImportTimeStamp = o.ImportTimeStamp.ToString() //PROBLEM LINE
});
return Ok(query);
}
The error that I'm getting is
Cannot implicitly convert type 'string' to 'System.DateTime'
What am doing wrong? Any help would be appreciated. TIA.
EDIT:
Here is the DTO:
public class ImportLogDto
{
public int Id { get; set; }
public string FileName { get; set; }
public DateTime ImportTimeStamp { get; set; }
public string ImportStatus { get; set; }
public string ImportMessage { get; set; }
}

You are trying to assign a string into a DateTime so you get that exception. If you want to cast it to a string change your model as follows:
public class ImportLogDto
{
public int Id { get; set; }
public string FileName { get; set; }
public string ImportTimeStamp { get; set; } // Changed type
public string ImportStatus { get; set; }
public string ImportMessage { get; set; }
}
And then your query:
var query = (from dbitem in db.ImportLogs
orderby dbitem.import_log_id
select new {
Idbitem.import_log_id,
dbitem.import_file_name,
dbitem.import_timeStamp, // Still DateTime
dbitem.import_status,
dbitem.import_message
}).AsEnumerable(); // Retrieved from Database
.Select(o => new ImportLogDto
{
Id = o.import_log_id,
FileName = o.import_file_name,
ImportMessage = o.import_message,
ImportStatus = o.import_status,
ImportTimeStamp = o.import_timeStamp.ToString() // Changes to string
});
If you wnat to change the format of the DateTime for the API then then use its overload of ToString and specify a format:
ImportTimeStamp = o.ImportTimeStamp.ToString("dd/MM/yyyy HH24:MI:ss")
For more on the overload read: Custom Date and Time Format Strings

Your types are already DateTime, and since these types are tied to the backing data* you probably shouldn't change them. But where you return the values on the API you can really return anything you want. So an anonymous type could be a quick solution:
.Select(o => new // <--- notice no type definition here
{
Id = o.Id,
FileName = o.FileName,
ImportMessage = o.ImportMessage,
ImportStatus = o.ImportStatus,
ImportTimeStamp = o.ImportTimeStamp.ToString()
})
The compiler will know based on the type returned by .ToString() that you want ImportTimeStamp to be a string. You can then add formatters to .ToString() to customize the output however you like.
*If your DTO isn't actually tied to the database then you can change the type there from DateTime to string, of course. It's not really clear from the context of the code shown whether these are data DTOs or application DTOs.

MongoDB: Querying for a referenced document

I am new to MongoDB and I read the MongoDB documentation. I have following structure: -
public class User
{
[BsonId]
public long UserId { get; set; }
public string LoginId { get; set; }
public string Password { get; set; }
public List<string> Gateways { get; set; }
}
public class Gateway
{
[BsonId]
public string MACAddress { get; set; }
public string SerialNumber { get; set; }
public List<Device> Devices { get; set; }
}
I referenced Gateway MAC Addresses in the USER document as Gateway is also having separate existence in absence of USER. For getting all the gateways for a given UserId I am writing queries as follows: -
var userQuery = Query<User>.EQ(u => u.UserId, aUserId);
var userCursor = mMongoUserCollection.Find(userQuery);
var gateways = mMongoGatewayCollection.AsQueryable<Gateway>().Where(g => userCursor.FirstOrDefault().Gateways.Contains(g.MACAddress));
But I am getting an Exception that
"Unable to determine Serialization Information for the expression: Enumerable.FirstOrDefault<User>"
But when I write my query as follows all goes well
var userQuery = Query<User>.EQ(u => u.UserId, aUserId);
var userCursor = mMongoUserCollection.Find(userQuery);
List<string> desiredGateways = userCursor.FirstOrDefault().Gateways;
var gateways = mMongoGatewayCollection.AsQueryable<Gateway>().Where(g => desiredGateways.Contains(g.MACAddress));
I just want to know the difference between the above two.

The difference is that the MongoDB C# driver can't translate the first snippet into a MongoDB query, while it can the second one.
When you call Where on an IQueryable it stores all the lambda expressions until you materialize the query be using foreach or ToList. At that stage the provider (the driver) tries to generate relevant queries to be performed in the database.
The database can't use userCursor.FirstOrDefault() as it doesn't know what FirstOfDefault is and it can't receive a cursor. It can however serialize a User instance that you have retrieved from the database previously.

Automapper not populating destination from source

I have a simple test solution which consists of two projects (a 'business' layer and a Data Access layer) using Catel to tie the two together - works fine, no problems.
However, have been reading about how useful AutoMapper can be for helping to move data around such a setup by allowing easy population of DTO's and decided to give it a look...that's when my problems started!
I'm using Entity Framework 6.1, VS 2013 Express for Desktop and accessing a SQL Server Express 14 db - no problems with data retrieval and data displays correctly in my views.
AutoMapper was added using NuGet.
In order to use AutoMapper I've set up the following in my App.xaml.cs
private void InitializeAutomapper()
{
Mapper.CreateMap<Result, ResultDto>();
Mapper.AssertConfigurationIsValid();
}
This code is the first item called inside my 'OnStartup'.
A service in my business layer makes a call to the Data Access layer and retrieves a list of Result entites.
Subsequently, I take a single entity from this list and use that in the AutoMapper mapping call.
I'm trying to populate a resultDTO from this single entity, using the following
Result res = ResultList.First();
ResultDto resultDTO = Mapper.Map<Result, ResultDto>(res);
'res' is correctly populated with data but resultDTO is filled with the default values for the individual data types (in = 0, string = null, DateTime = {01/01/0001 00:00:00}) ie; no values are mapped from the source to the destination.
There are References in both projects to AutoMapper and AutoMapper.Net and no errors are raised - it just doesn't work as advertised...
I'm not slagging off the software, just asking what I'm doing wrong!
I realise there isn't much code to work on here but, in truth, what is posted here is pretty much all I've added to try out AutoMapper. I can see, conceptually how useful it could be - I just need to figure out how to make it happen so any help/comments gratefully received...:)
EDIT
#Andrew, as requested -
Result Class:
public partial class Result
{
public int Div { get; set; }
public System.DateTime Date { get; set; }
public string HomeTeam { get; set; }
public string AwayTeam { get; set; }
public int FTHG { get; set; }
public int FTAG { get; set; }
public string FTR { get; set; }
}
ResultDTO Class:
public class ResultDto
{
int Div { get; set; }
DateTime Date { get; set; }
string HomeTeam { get; set; }
string AwayTeam { get; set; }
int FTHG { get; set; }
int FTAG { get; set; }
string FTR { get; set; }
// Added tonight to try and get it to work
public ResultDto()
{
Div = 0;
Date = DateTime.Now;
HomeTeam = null;
AwayTeam = null;
FTHG = 0;
FTAG = 0;
FTR = null;
}
}
#stuartd, the following is used to retrieve the ResultList from which Result is obtained:
// Produce a list of DataLayer.Result entities.
var ResultList = (from x in dbContext.Results.Local
where x.HomeTeam == team.TeamName.ToString() || x.AwayTeam == team.TeamName.ToString()
orderby x.Date
select x).ToList();
Please note 'team.Teamname' is passed into the above from an external source - seems to be working fine.
So to sum up -
I produce ResultList as a list of Result entities.
Fill Result with the first entity in the list.
Try to map this Result entity to ResultDTO
Fail :(
Hope this helps!

By default, class members are declared private unless otherwise specified so the ResultDto properties aren't visible outside of the class.
public class ResultDto
{
int Div { get; set; }
....
}
needs to be
public class ResultDto
{
public int Div { get; set; }
....
}

AutoMapper can work out the type you are mapping from from the arguments provided. Try this:
ResultDto resultDTO = Mapper.Map<ResultDto>(res);
UPDATE
This is wrong, or at least won't help. We need to see the source and destination classes as mentioned in the comments.