We Keep Coding

Avoiding duplicates when inserting multiple data into MongoDB

Hi I want to insert multiple data and avoid duplicates in my programme. I have implemented the code to insert multiple data to mongoDB using asp.net core web api and it's working fine. I tried so many things but still I couldn't find a way to avoid duplicate records when inserting multiple records. Can you guys help me please :)
I want to avoid inserting duplicates by using employeeid.
this is my controller
using HelloApi.Models;
using HelloApi.Services;
using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace HelloApi.Controllers
{
[Route("api/[controller]")]
[ApiController]
[Produces("application/json")]
public class EmployeeController : ControllerBase
{
private readonly EmployeeService _employeeService;
public EmployeeController(EmployeeService employeeService)
{
_employeeService = employeeService;
}
//.....................................Create..............................................
public Task Create(IEnumerable<Employee> employees)
{
return _employeeService.Create(employees);
}
}
}
This is my model class
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using System.ComponentModel.DataAnnotations;
namespace HelloApi.Models
{
public class Employee
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
[Key]
public string Id { get; set; }
[Required]
[BsonElement("employeeid")]
public string employeeid { get; set; }
[Required]
[BsonElement("firstname")]
public string firstname { get; set; }
[Required]
[BsonElement("lastname")]
public string lastname { get; set; }
[Required]
[BsonElement("age")]
public int age { get; set; }
[Required]
[BsonElement("address")]
public string address { get; set; }
[Required]
[BsonElement("telephone")]
public int telephone { get; set; }
}
}
This is my service class
using HelloApi.Models;
using MongoDB.Driver;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
namespace HelloApi.Services
{
public class EmployeeService
{
private readonly IMongoCollection<Employee> _employee;
public EmployeeService(IHelloApiDatabaseSettings settings)
{
var client = new MongoClient(settings.ConnectionString);
var database = client.GetDatabase(settings.DatabaseName);
_employee = database.GetCollection<Employee>(settings.employeeCollectionName);
}
//.....................................Create..............................................
public Task Create(IEnumerable<Employee> employees)
{
return _employee.InsertManyAsync(employees);
}
}
}
This is the POST request in Postman and it is working fine but submit duplicates.
Really appreciate if you guys can help me with this :)

This Might be helpful. Use below InsertORUpdateEmployee function to find and update using employeeid.
public Task<string> Create(IEnumerable<Employee> employees)
{
return Task.Run(() =>
{
return InsertORUpdateEmployee(employees);
});
}
//Insert or Update Employees
private string InsertORUpdateEmployee(IEnumerable<Employee> employees)
{
try
{
foreach (Employee emp in employees)
{
var empId = emp.employeeid;
var DB = Client.GetDatabase("Employee");
var collection = DB.GetCollection<Employee>("EmployeeDetails");
//Find Employee using employeeid
var filter_id = Builders<Employee>.Filter.Eq("employeeid", empId);
var entity = collection.Find(filter_id).FirstOrDefault();
//Insert
if (entity == null)
{
collection.InsertOne(emp);
}
else
{
//Update
var update = collection.FindOneAndUpdateAsync(filter_id, Builders<Employee>.Update
.Set("firstname", emp.firstname)
.Set("lastname", emp.lastname)
.Set("age", emp.age)
.Set("address", emp.address)
.Set("telephone", emp.telephone));
}
}
return "Insert or Updated Succesfully";
}
catch (Exception ex)
{
return ex.ToString();
}
}

I think you need to add an extra step to declare your Id field as a Primary Key. [Key] should only work for RDBMS. For MongoDB i found another post here to define a field as a unique key.
(Actually MongoDB doesn't really allow for a Primary Key besides the auto-generated '_id' field, afaik, but you can create unique keys. )
Try looking through the answers posted here:
How to set a primary key in MongoDB?

It seems you need to create unique index on employeeId field to avoid duplicates :
db.employee.createIndex( { "employeeId": 1 }, { unique: true } )

Raw queries with overridden column names

I'm trying to retrieve some entities using Entity Framework by querying an XML column. Entity Framework doesn't support this so I had to use raw SQL.
var people = context.People.SqlQuery("SELECT * FROM [People] WHERE [DataXML].value('Properties/Age', 'int') = 21").AsQueryable().AsNoTracking();
My person class:
public class Person
{
public int Id { get; set; }
public string Name { get; set; }
[Column("YearsSinceBirth")]
public int Age { get; set; }
[Column(TypeName = "xml")]
public string DataXML { get; set; }
}
This should work, however, it falls over when trying to map it back to an object. Specifically, it's falling over on the Age property, which has it's column name overridden to "YearsSinceBirth".
'The data reader is incompatible with the specified
'MyProject.CodeBase.DataModel.DbEntities.Person'. A member of the
type, 'Age', does not have a corresponding column in the data reader
with the same name.'
I'm guessing that Entity Framework doesn't map database column names to object property names and therefore is expecting the column to be named 'Age' rather than 'YearsSinceBirth'.
I don't want to have to list each column and their mapping in the SQL query (like SELECT YearsSinceBirth As Age) as the actual project I'm working on which has this column has a lot more columns and that would mean this query would break every time the schema changed (kinda defeating the purpose of Entity Framework).

If this is EF Core, your problem is not that SqlQuery() doesn't support mapping column names (it does). Rather your problem is that your table doesn't contain a column called YearsSinceBirth, and you are returning 'select *'.
If you have a column called YearsSinceBirth, this works fine. Although you will be retrieving the value in the YearsSinceBirth column, not the value in the XML document. EG
using Microsoft.EntityFrameworkCore;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
//using Microsoft.Samples.EFLogging;
using System.ComponentModel.DataAnnotations.Schema;
using System.ComponentModel.DataAnnotations;
using System.Data.SqlClient;
namespace EFCore2Test
{
public class Person
{
public int Id { get; set; }
public string Name { get; set; }
[Column("YearsSinceBirth")]
public int Age { get; set; }
[Column(TypeName = "xml")]
public string DataXML { get; set; }
}
public class Location
{
public string LocationId { get; set; }
}
public class Db : DbContext
{
public DbSet<Person> People { get; set; }
public DbSet<Location> Locations { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer("Server=(local);Database=EFCoreTest;Trusted_Connection=True;MultipleActiveResultSets=true");
base.OnConfiguring(optionsBuilder);
}
}
class Program
{
static void Main(string[] args)
{
using (var db = new Db())
{
db.Database.EnsureDeleted();
//db.ConfigureLogging(s => Console.WriteLine(s));
db.Database.EnsureCreated();
var p = new Person()
{
Name = "joe",
Age = 2,
DataXML = "<Properties><Age>21</Age></Properties>"
};
db.People.Add(p);
db.SaveChanges();
}
using (var db = new Db())
{
var people = db.People.FromSql("SELECT * FROM [People] WHERE [DataXML].value('(/Properties/Age)[1]', 'int') = 21").AsNoTracking().ToList() ;
Console.WriteLine(people.First().Age);
Console.ReadLine();
}
Console.WriteLine("Hit any key to exit");
Console.ReadKey();
}
}
}
You can use a pattern similar to this to project entity attributes from an XML or JSON column:
public class Person
{
private XDocument xml;
public int Id { get; set; }
public string Name { get; set; }
[NotMapped]
public int Age
{
get
{
return int.Parse(xml.Element("Properties").Element("Age").Value);
}
set
{
xml.Element("Properties").Element("Age").Value = value.ToString();
}
}
[Column(TypeName = "xml")]
public string DataXML
{
get
{
return xml.ToString();
}
set
{
xml = XDocument.Parse(value);
}
}
}

You can dynamically create select query with aliases, if they needed, with the help of reflection and ColumnAttribute checking:
public string SelectQuery<T>() where T : class
{
var selectQuery = new List<string>();
foreach (var prop in typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance))
{
var attr = prop.GetAttribute<ColumnAttribute>();
selectQuery.Add(attr != null ? $"{attr.Name} as {prop.Name}" : prop.Name);
}
return string.Join(", ", selectQuery);
}
Usage:
var people = context.People.SqlQuery($"SELECT {SelectQuery<Person>()} FROM [People] WHERE [DataXML].value('Properties/Age', 'int') = 21")
.AsQueryable().AsNoTracking();

How avoid adding duplicates to database managed by Entity Framework caused by Seed?

Every time I run the application same objects are added to the database (duplicates).
My Configuration.cs:
namespace SklepInternetowy1.Migrations
{
using SklepInternetowy1.DAL;
using System;
using System.Data.Entity;
using System.Data.Entity.Migrations;
using System.Linq;
public sealed class Configuration : DbMigrationsConfiguration<SklepInternetowy1.DAL.KursyContext>
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
ContextKey = "SklepInternetowy1.DAL.KursyContext";
}
protected override void Seed(SklepInternetowy1.DAL.KursyContext context)
{
KursyInitializer.SeedKursyData(context);
// This method will be called after migrating to the latest version.
// You can use the DbSet<T>.AddOrUpdate() helper extension method
// to avoid creating duplicate seed data. E.g.
//
// context.People.AddOrUpdate(
// p => p.FullName,
// new Person { FullName = "Andrew Peters" },
// new Person { FullName = "Brice Lambson" },
// new Person { FullName = "Rowan Miller" }
// );
//
}
}
}
My KursyInitializer.cs:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Data.Entity;
using SklepInternetowy1.Models;
using SklepInternetowy1.DAL;
using SklepInternetowy1.Migrations;
using System.Data.Entity.Migrations;
namespace SklepInternetowy1.DAL
{
public class KursyInitializer : MigrateDatabaseToLatestVersion<KursyContext, Configuration>
{
public static void SeedKursyData(KursyContext context)
{
var kategorie = new List<Kategoria>
{
new Kategoria() {KategoriaID=1, NazwaKategorii="Asp123",NazwaPlikuIkony="asp.png", OpisKategorii="opis asp net mvc" },
new Kategoria() {KategoriaID=2, NazwaKategorii="Java",NazwaPlikuIkony="java.png", OpisKategorii="opis java" },
new Kategoria() {KategoriaID=3, NazwaKategorii="Php",NazwaPlikuIkony="php.png", OpisKategorii="opis php" },
new Kategoria() {KategoriaID=4, NazwaKategorii="Html",NazwaPlikuIkony="html.png", OpisKategorii="opis html" },
new Kategoria() {KategoriaID=5, NazwaKategorii="Css",NazwaPlikuIkony="css.png", OpisKategorii="opis css" },
new Kategoria() {KategoriaID=6, NazwaKategorii="Xml",NazwaPlikuIkony="xml.png", OpisKategorii="opis xml" },
new Kategoria() {KategoriaID=7, NazwaKategorii="C#",NazwaPlikuIkony="c#.png", OpisKategorii="opis c#" }
};
kategorie.ForEach(k => context.Kategorie.AddOrUpdate(k));
context.SaveChanges();
var kursy = new List<Kurs>
{
new Kurs() {AutorKursu="Tomek", TytulKursu="asp.net mvc1", KategoriaID=1, CenaKursu=100, Bestseller=true, NazwaPlikuObrazka="asd.png", DataDodania=DateTime.Now, OpisKursu="opis1" },
new Kurs() {AutorKursu="Jacek", TytulKursu="asp.net mvc2", KategoriaID=2, CenaKursu=101, Bestseller=true, NazwaPlikuObrazka="asd1.png", DataDodania=DateTime.Now, OpisKursu="opis2" },
new Kurs() {AutorKursu="Jarek", TytulKursu="asp.net mvc3", KategoriaID=3, CenaKursu=102, Bestseller=true, NazwaPlikuObrazka="asd2.png", DataDodania=DateTime.Now, OpisKursu="opis3" },
new Kurs() {AutorKursu="Romek", TytulKursu="asp.net mvc4", KategoriaID=4, CenaKursu=103, Bestseller=true, NazwaPlikuObrazka="asd3.png", DataDodania=DateTime.Now, OpisKursu="opis4" }
};
kursy.ForEach(k => context.Kursy.AddOrUpdate(k));
context.SaveChanges();
}
}
}
My Kurs.cs:
namespace SklepInternetowy1.Models
{
public class Kurs
{
public int KursID { get; set; }
public int KategoriaID { get; set; }
[Required(ErrorMessage = "Wprowadz nazwę kursu")]
[StringLength(100)]
public string TytulKursu { get; set; }
[Required(ErrorMessage = "Wprowadz nazwę autora")]
[StringLength(100)]
public string AutorKursu { get; set; }
public DateTime DataDodania { get; set; }
[StringLength(100)]
public string NazwaPlikuObrazka { get; set; }
public string OpisKursu { get; set; }
public decimal CenaKursu { get; set; }
public bool Bestseller { get; set; }
public bool Ukryty { get; set; }
public string OpisSkrocony { get; set; }
public virtual Kategoria Kategoria { get; set; }
}
}
Every time I run the app, Seed adds duplicate records:
duplicates
Question: what must I change to avoid duplicates in the table Kurs?

As stated in the AddOrUpdate MSDN article this method uses the key property to check the existence of an entity. In your case, the default value for the integer property is 0. So the EF checks whether there is a record with such key and decided that it should create a new record for the entity.
So, you need to explicitly define the KursID property in the seed method for each record in order to make Entity Framework know that the related item already persists in the data base when the AddOrUpdate is executed.

You forgot to bind the key(s) for which the seed method will check either it is an add or update.
Just change the following line......
kursy.ForEach(k => context.Kursy.AddOrUpdate(x => x.AutorKursu, k));
I am assuming that "AutorKursu" is your primary key. You can setup this condition as you like & depending on that lamda function seed method will determine the require action.
From Entity Framework doc:
identifierExpression:
// An expression specifying the properties that should be used when determining
// whether an Add or Update operation should be performed.
MSDN: https://msdn.microsoft.com/en-us/library/hh846514(v=vs.103).aspx

Delete nodes from returned from response.content using deserialization in RestSharp

I know the Response.Content from my GET RestRequest is xml UTF-8. However I want to delete unwanted child nodes from the response.content. I can't assign the response to an xml document (because it returns a string apparently ?) which I could use system.xml to delete unwanted child nodes.
I have a situation (below) where if I have three space_reservation nodes [1401, 1402, and 1401 & 1402], I want to delete the space_reservation nodes for MH-1402 and (MH1401 & MH-1402), and keep the space_reservation node for MH-1401. I want to do this before I use the a REST API to schedule the rooms because these room will present a duplicate schedule situation.
Here is the example of the response.content from the request:
<?xml version="1.0" encoding="UTF-8"?>
<r25:reservations xmls:xsi="http://www.w3.org/2001/XMLSchemainstance">
<r25:reservation xl:href="reservation.xml?rsrv_id=731397">
<r25:reservation_id>7313</r25:reservation_id>
<r25:reservation_state>1</r25:reservation_state>
<r25:event_start_dt>2016-04-12T09:00:00-07:00</r25:event_start_dt>
<r25:event_end_dt>2016-04-12T12:00:00-07:00</r25:event_end_dt>
<r25:event_id xl:href="event.xml?event_id=197559">197559</r25:event_id>
<r25:event_locator>2016-ABAHZP</r25:event_locator>
<r25:event_name>Spring Grand Rounds</r25:event_name>
<r25:event_type_name>Department Meetings & Events</r25:event_type_name>
<r25:organization_name>Sciences</r25:organization_name>
<r25:profile_name>April 12th Capture</r25:profile_name>
<r25:space_reservation xl:href="space.xml?space_id=335">
<r25:space_name>MH-1401</r25:space_name>
<r25:space_id>335</r25:space_id>
<r25:space_instruction_id>94367</r25:space_instruction_id>
</r25:space_reservation>
<r25:space_reservation xl:href="space.xml?space_id=336">
<r25:space_name>MH-1402</r25:space_name>
<r25:space_id>336</r25:space_id>
<r25:space_instruction_id>94368</r25:space_instruction_id>
</r25:space_reservation>
<r25:space_reservation xl:href="space.xml?space_id=337">
<r25:space_name>MH-1401 & 1402</r25:space_name>
<r25:space_id>337</r25:space_id>
<r25:space_instruction_id>94366</r25:space_instruction_id>
</r25:space_reservation>
<r25:resource_reservation xl:href="resource.xml?resource_id=55">
<r25:resource_id>55</r25:resource_id>
<r25:resource_name>Live plus on-demand</r25:resource_name>
<r25:resource_count>1</r25:resource_count>
<r25:resource_instruction_id/>
<r25:resource_instructions/>
</r25:resource_reservation>
</r25:reservation>
</r25:reservations>
Here are my deserialization classes:
public class Mreservation : List<reservation> { }
public class reservation
{
public string event_name { get; set; }
public DateTime reservation_start_dt { get; set; }
public DateTime reservation_end_dt { get; set; }
public DateTime event_start_dt { get; set; }
public DateTime event_end_dt { get; set; }
public string event_locator { get; set; }
public int organization_id { get; set; }
public List<space_reservation> spaceNodes { get; set; }
public List<resource_reservation> resourceNodes { get; set; }
}
public class reservations
{
public string pubdate { get; set; }
public List<reservation> ReservationNodes { get; set; }
}
public class space_reservation
{
public string space_name { get; set; }
public int space_id { get; set; }
}
public class resource_reservation
{
public int resource_id { get; set; }
public string resource_name { get; set; }
}
Here is the code I have for eliminating nodes that have title 1401. I save to xml so I can confirm that the node has been deleted but still the same.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using RestSharp;
using System.Xml;
namespace EliminateDuplicates
{
class Program
{
static void Main(string[] args)
{
var R25Client = R25_Rest_Login.R25Login();
String testDate = new DateTime(2016, 4, 12).ToString("yyyyMMdd");
var CaptureRequest = new RestRequest("reservations.xml", Method.GET);
CaptureRequest.AddParameter("resource_query_id", "35304");
CaptureRequest.AddParameter("start_dt", testDate);
CaptureRequest.AddParameter("end_dt", testDate);
CaptureRequest.RequestFormat = DataFormat.Xml;
var CaptureResponse = R25Client.Execute<Mreservation>(CaptureRequest);
XmlDocument xdoc = new XmlDocument();
xdoc.LoadXml(CaptureResponse.Content);
xdoc.Save("beforeRemoval.xml");
foreach (var x in CaptureResponse.Data)
{
if ((x.spaceNodes[0].space_id == 335) && (x.spaceNodes[1].space_id == 336) && (x.spaceNodes[2].space_id == 337))
{
x.spaceNodes.RemoveAll(i => i.space_id == 335);
}
}
XmlDocument xdocA = new XmlDocument();
xdocA.LoadXml(CaptureResponse.Content);
xdocA.Save("afterRemoval.xml");
}
}
}
Im looking for the proper way to delete these nodes using RestSharp ?

The proper way of transforming structure of responses in RestSharp would be implementing a custom IDeserializer.
However, it seems you want to apply business logic (i.e. remove some reservations) instead of just deserializing. In that case, you should use the already-deserialized object as shown in the example below.
var reservations = client.Execute<Mreservation>(request).Data;
foreach(var reservation in reservations)
{
reservation.SpaceNodes.RemoveAll((space) => someCondition(space));
}
// ...go on using reservations

Entity Framework 5 Won't Fetch Relationships With Include()

I am quite certain that questions like this have been answered a number of times before, but I can't get any of the suggestions to work.
I am building a MVC 4 application with Entity Framework 5, where the entities were generated from existing tables. I have entity classes that look like this:
namespace RebuildingModel
{
using System;
using System.Collections.Generic;
public partial class StandardCodeTable
{
public StandardCodeTable()
{
this.StandardCodeTableTexts = new HashSet<StandardCodeTableText>();
}
public int TableCode { get; set; }
public string RefTableName { get; set; }
public virtual ICollection<StandardCodeTableText> StandardCodeTableTexts { get; set; }
}
}
namespace RebuildingModel
{
using System;
using System.Collections.Generic;
public partial class StandardCodeTableText
{
public int TableCode { get; set; }
public string LanguageCode { get; set; }
public string TextVal { get; set; }
public virtual StandardCodeTable StandardCodeTable { get; set; }
}
}
namespace RebuildingSite.Models
{
public class CodeTableJoined
{
public int TableCode { get; set; }
public string ReferenceTableName { get; set; }
public string LanguageCode { get; set; }
public string TextValue { get; set; }
}
}
I have a DAO that looks like this:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace RebuildingModel.Dao
{
public class CodeTableDao
{
public CodeTableDao() { }
public ISet<StandardCodeTableText> GetCode(string refTableName)
{
HashSet<StandardCodeTableText> codes = new HashSet<StandardCodeTableText>();
using (var db = new RebuildingTogetherEntities())
{
db.StandardCodeTableTexts.Include("StandardCodeTables");
var query = from c in db.StandardCodeTableTexts
where c.StandardCodeTable.RefTableName == refTableName
orderby c.TableCode
select c;
foreach (var item in query)
{
codes.Add(item);
}
}
return codes;
}
}
I have a controller that looks like this:
namespace RebuildingSite.Controllers
{
public class CodeTableController : Controller
{
public ActionResult Index(string refTableName)
{
CodeTableDao dao = new CodeTableDao();
ICollection<StandardCodeTableText> codes = dao.GetCode(refTableName);
HashSet<CodeTableJoined> joins = new HashSet<CodeTableJoined>();
foreach (var code in codes)
{
CodeTableJoined join = new CodeTableJoined();
join.TableCode = code.TableCode;
join.LanguageCode = code.LanguageCode;
join.TextValue = code.TextVal;
join.ReferenceTableName = code.StandardCodeTable.RefTableName;
joins.Add(join);
}
ISet<string> refTableNames = dao.GetReferenceTables();
ViewBag.RefTableNames = refTableNames;
return View(joins);
}
}
}
When I run the view attached to the controller, an ObjectDisposedException is thrown at this line, where the relationship is used:
join.ReferenceTableName = code.StandardCodeTable.RefTableName;
This has to be something simple. What am I doing wrong? I have tried adding that Include() call in from the context in many different places, even multiple times.
I've also tried adding an explicit join in the Linq query. I can't get EF to fetch that relationship.

Copying my comment to an answer - Put the include be in the actual query
var query = from c in
db.StandardCodeTableTexts.include("StandardCodeTables"). where
c.StandardCodeTable.RefTableName == refTableName orderby c.TableCode
select c;