What is the most efficient way to sync documents in a RavenDB?
From an external source I get an IEnumerable of BlogPosts that I want to do the following with:
Add new objects that are new to RavenDB
Update existing objects
Remove objects that were removed in the external source
The code that needs implementation:
public void SyncIntoRaven(IEnumerable<BlogPost> postsToSync, IDocumentStore store) {
// TODO: Implement
// AddNewItems(postsToSync);
// TODO: Implement
// RemoveDeletedItems(postsToSync);
// TODO: Implement
// UpdateExistingItems(postsToSync);
}
One could just pull out all BlogPosts from RavenDB and sync locally to then push all the changes back, but I want to minimize traffic to RavenDB. But maybe that's not the right approach either?
If you are sharing the same ID between your external source and RavenDB, you can do this quite easily, in an ACID fashion, and within one transaction.
Keep track of IDs that changed between sync operations, and once you have that list of ID's you can easily do this:
Open a session, add the new documents using session.Store(), load all the documents need updating or deleting using session.Load(string[]) session.Load().Lazily, make the updates (and deletions using the Deferred option), and once you are done call session.SaveChanges().
That should get you covered, and happen in only one roundtrip to the server.
Either way, you never want to do complete sync every time. You always want to use deltas.
With the help in description-form from synhershko I figured it out and wanted to share the code, simplified to show the concepts.
private void RefreshBlogPosts(IDocumentSession session, IList<BlogPost> parsedPosts) {
var parsedPostsIds = parsedPosts.Select(x => x.Id);
var storePosts = session.Load<BlogPost>(parsedPostsIds);
// Update existing or create new posts
for(int i = 0; i < storePosts.Count(); i++) {
var parsedPost = parsedPosts[i];
var storePost = storePosts[i];
if(storePost == null) {
storePost = parsedPost;
session.Store(storePost);
} else {
// Update post's properties
}
}
// Find posts IDs no longer in database
var removedPostIds = session.Query<BlogPost>().Select(x => x.Id)
.Where(postId => !parsedPostsIds.Contains(postId));
foreach(var removedPostId in removedPostIds) {
session.Advanced.Defer(new DeleteCommandData() { Key = removedPostId });
}
session.SaveChanges();
}
Related
Using NopCommerce 3.8, Visual Studio 2015 proff.
I have created a plugin that is responsible for making restful calls to my Web API that exposes a different DB to that of Nop.
The process is run via a nop Task, it successfully pulls the data back and i can step through and manipulate as i see fit, no issues so far.
Issue comes when i try to update a record on the product table, i perform the update... but nothing happens no change, no error.
I believe this is due to the Context having no idea about my newly instantiated product object, however I'm drawing a blank on what i need to do in relation to my particular example.
Similar questions usually reference a "model" object that is part of the parameter of the method call, "model" has the method ToEntity which seems to be the answer in similar question in stack.
However my example doesn't have the ToEntity class/method possibly because my parameter is actually a list of products. To Clarify here my code.
Method in RestClient.cs
public async Task<List<T>> GetAsync()
{
try
{
var httpClient = new HttpClient();
var json = await httpClient.GetStringAsync(ApiControllerURL);
var taskModels = JsonConvert.DeserializeObject<List<T>>(json);
return taskModels;
}
catch (Exception e)
{
return null;
}
}
Method in my Service Class
public async Task<List<MWProduct>> GetProductsAsync()
{
RestClient<MWProduct> restClient = new RestClient<MWProduct>(ApiConst.Products);
var productsList = await restClient.GetAsync();
InsertSyncProd(productsList.Select(x => x).ToList());
return productsList;
}
private void InsertSyncProd(List<MWProduct> inserted)
{
var model = inserted.Select(x =>
{
switch (x.AD_Action)
{
case "I":
//_productService.InsertProduct(row);
break;
case "U":
UpdateSyncProd(inserted);
.....
Then the method to bind and update
private void UpdateSyncProd(List<MWProduct> inserted)
{
var me = inserted.Select(x =>
{
var productEnt = _productRepos.Table.FirstOrDefault(ent => ent.Sku == x.Sku.ToString());
if(productEnt != null)
{
productEnt.Sku = x.Sku.ToString();
productEnt.ShortDescription = x.ShortDescription;
productEnt.FullDescription = x.FullDescription;
productEnt.Name = x.Name;
productEnt.Height = x.Pd_height != null ? Convert.ToDecimal(x.Pd_height) : 0;
productEnt.Width = x.Pd_width != null ? Convert.ToDecimal(x.Pd_width) : 0;
productEnt.Length = x.Pd_depth != null ? Convert.ToDecimal(x.Pd_depth) : 0;
productEnt.UpdatedOnUtc = DateTime.UtcNow;
}
//TODO: set to entity so context nows and can update
_productService.UpdateProduct(productEnt);
return productEnt;
});
}
So as you can see, I get the data and pass data through to certain method based on a result. From that list in the method I iterate over, and pull the the entity from the table, then update via the product service using that manipulated entity.
So what am I missing here, I'm sure its 1 step, and i think it may be either be because 1) The context still has no idea about the entity in question, or 2) Its Incorrect calls.
Summary
Update is not updating, possibly due to context having no knowledge OR my methodology is wrong. (probably both).
UPDATE:
I added some logger.inertlog all around my service, it runs through fine, all to the point of the call of update. But again I check the product and nothing has changed in the admin section.
plugin
I have provided the full source as i think maybe this has something to do with the rest of the code setup possibly?
UPDATE:
Added the following for testin on my execute method.
var myprod = _productRepos.GetById(4852);
myprod.ShortDescription = "db test";
productRepos.Update(myprod);
This successfully updates the product description. I moved my methods from my service into the task class but still no luck. The more i look at it the more im thinking that my async is killing off the db context somehow.
Turned of async and bound the getbyid to a new product, also removed the lambda for the switch and changed it to a foreach loop. Seems to finally update the results.
Cannot confirm if async is the culprit, currently the web api seems to be returning the same result even though the data has changed (some wierd caching by deafult in .net core? ) so im creating a new question for that.
UPDATE: It appears that the issue stems from poor debugging of async. Each instance I am trying to iterate over an await call, simply put im trying to iterate over a collection that technically may or may not be completed yet. And probably due to poor debugging, I was not aware.
So answer await your collection Then iterate after.
I am getting users and their data from external webservice. I cache those items because I don't want to hit web service every time. Now, If user update any of their information, I am saving it through webservice. But I don't want to get the latest data from web service as it takes lot of time. Instead I want to update my cache. Can I do that ? If so, what would be the best way ? Here is my Code
List<User> users = appSecurity.SelectUsers();
var CacheKey = string.Format("GetUserList_{0}", currentUser);
CacheFactory.AddCacheItem(CacheKey, users, 300);
CacheFactory is a class where I handle Adding, Clearing and Removing cache. Below is the code
public static void RemoveCacheItem(string key)
{
Cache.Remove(key);
}
public static void ClearCache()
{
System.Collections.IDictionaryEnumerator enumerator = Cache.GetEnumerator();
while (enumerator.MoveNext())
{
RemoveCacheItem(enumerator.Key.ToString());
}
}
public static void AddCacheItem<T>(string key, T value, double timeOutInSeconds)
{
var Item = GetCacheItem<T>(key);
if (Item != null)
{
RemoveCacheItem(key);
Item = value;
}
Cache.Insert(key, value, null, DateTime.Now.AddSeconds(timeOutInSeconds), System.Web.Caching.Cache.NoSlidingExpiration);
}
The answer is yes, it can be done. It can also be done in many different ways depending on what you want to solve. At the basic level you can create a cache by using a List<T> or Dictionary<T,T> to store your data.
When you get information from the external web-service, you push the data into your List or Dictionary. You can then use that data throughout your application. When you need to update that cache, you update the value in the List/Dictionary.
You can update your dictonary like so
Dictionary<string, int> list = new Dictionary<string, int>();
then you can set the value for the key "test" as follows
list["test"] = list["test"] + 1;
When you are ready to push the updated data to the external source. All you need to do is properly parse that data into the format the source is expecting and send away.
Like I said there are many different ways to do this, but this is a basic sample way to accomplishing it. You can use this example to build off and go from there.
I am trying to devise a good way to perform updates to a SQL Server database using WCF Data Services and Entity Framework. The problem I'm having is that it seems overly complex to perform update, delete, and insert operations using the service.
I'll use typical Customer / Invoices scenario to help explain my current approach. I'm using WPF MVVM for the application. My view model contains a customer object that receives updates from the user. When saving, I pass the customer object to the service. The service must then load the customer object, transfer the property values from the updated customer, then perform the save.
Something like this:
public static int SaveProgram(Customer entity)
int returnValue = 0;
// Setup the service Uri
Uri serviceUri = new Uri(Properties.Settings.Default.DataUri);
try
{
// Get the DB context
var context = new DevEntities(serviceUri);
Customer dbCustomer;
if (entity.CustomerId == 0)
{
dbCustomer = new Customer();
context.AddToCustomers(dbCustomer);
}
else
{
dbCustomer = context.Customers.Where(p => p.CustomerId == entity.CustomerId).FirstOrDefault();
}
if (dbCustomer != null)
{
dbCustomer.StatusId = entity.StatusId;
dbCustomer.FirstName = entity.FirstName;
dbCustomer.LastName = entity.LastName;
dbCustomer.Address = entity.Address;
...
}
context.UpdateObject(dbCustomer);
// Submit Changes
DataServiceResponse response = context.SaveChanges(SaveChangesOptions.Batch);
// Check for errors
...
returnValue = response.Count();
}
... Catch exceptions
return returnValue;
}
Is it really necessary to go through all of this? It seems there should be an easier way.
Adding an invoice requires something like this:
var newInvoice = Invoice.CreateInvoice(0, customerId, etc...);
context.AddRelatedObject(dbCustomer, "Invoices", newInvoice);
Having already added a new invoice to the Customer.Invoices collection, this seems cumbersome.
Deleting an invoice is even worse. To delete an invoice I have to compare the invoices collection from the database with that of the passed in entity. If I cannot find a database version of the invoice in the entity.Invoices collection, then I know it should be deleted.
I have the feeling that I must not be approaching this correctly.
I'm trying to expose an observable sequence that gives observers all existing records in a database table plus any future items. For the sake of argument, lets say it's log entries. Therefore, I'd have something like this:
public class LogService
{
private readonly Subject<LogEntry> entries;
public LogService()
{
this.entries = new Subject<LogEntry>();
this.entries
.Buffer(...)
.Subscribe(async x => WriteLogEntriesToDatabaseAsync(x));
}
public IObservable<LogEntry> Entries
{
get { return this.entries; }
}
public IObservable<LogEntry> AllLogEntries
{
get
{
// how the heck?
}
}
public void Log(string message)
{
this.entries.OnNext(new LogEntry(message));
}
private async Task<IEnumerable<LogEntry>> GetLogEntriesAsync()
{
// reads existing entries from DB table and returns them
}
private async Task WriteLogEntriesToDatabaseAsync(IList<LogEntry> entries)
{
// writes entries to the database
}
}
My initial thought for the implementation of AllLogEntries was something like this:
return Observable.Create<LogEntry>(
async observer =>
{
var existingEntries = await this.GetLogEntriesAsync();
foreach (var existingEntry in existingEntries)
{
observer.OnNext(existingEntry);
}
return this.entries.Subscribe(observer);
});
But the problem with this is that there could log entries that have been buffered and not yet written to the database. Hence, those entries will be missed because they are not in the database and have already passed through the entries observable.
My next thought was to separate the buffered entries from the non-buffered and use the buffered when implementing AllLogEntries:
return Observable.Create<LogEntry>(
async observer =>
{
var existingEntries = await this.GetLogEntriesAsync();
foreach (var existingEntry in existingEntries)
{
observer.OnNext(existingEntry);
}
return this.bufferedEntries
.SelectMany(x => x)
.Subscribe(observer);
});
There are two problems with this:
It means clients of AllLogEntries also have to wait for the buffer timespan to pass before they receive their log entries. I want them to see log entries instantaneously.
There is still a race condition in that log entries could be written to the database between the point at which I finish reading the existing ones and the point at which I return the future entries.
So my question is: how would I actually go about achieving my requirements here with no possibility of race conditions, and avoiding any major performance penalties?
To do this via the client code, you will probably have to implement a solution using polling and then look for differences between calls. Possibly combining a solution with
Observable.Interval() : http://rxwiki.wikidot.com/101samples#toc28 , and
Observable.DistinctUntilChanged()
will give you sufficient solution.
Alternatively, I'd suggest you try to find a solution where the clients are notified when the DB/table is updated. In a web application, you could use something like SignalR to do this.
For example: http://techbrij.com/database-change-notifications-asp-net-signalr-sqldependency
If its not a web-application, a similar update mechanism via sockets may work.
See these links (these came from the accepted answer of SignalR polling database for updates):
http://xsockets.net/api/net-c#snippet61
https://github.com/codeplanner/XSocketsPollingLegacyDB
The Scenario
Currently I have a c# silverlight business application. The application is hosted in Asp.net using the ADO.Net entity framework and a domain service class to read/write to/from my sql server database.
The Setup
Client UI
In my silverlight client interface I have an autocomplete box which uses a query to get a list of items relating to the search. The query is in my domain service class and looks as follows:
public IQueryable<Status> GetStatus()
{
var q = (from job in Context.Job
select job.Status).Distinct()
.Select(deliveryState => new Status
{
DeliveryState = deliveryState,
Count = Context.Job.Count
(job => job.Status.Trim() == deliveryState.Trim())
});
q = q.Where(job => job.DeliveryState != null);
return q;
}
Code Behind
Then in my code behind for the silverlight client interface xaml page, I load the query in the contructor using the following code:
var context = dds.DomainContext as InmZenDomainContext;
statusFilterBox.ItemsSource = context.Status;
context.Load(context.GetStatusQuery(), (lo) =>
{
//just to show you how to load..
//new ErrorWindow("Loaded.." + lo.Entities.Count()).Show();
}, null);
The Issue
The issue I have is what happens now if I add another autocomplete box to my client interface, and another Get query to perform a different type of search:
public IQueryable<ShortCode> GetShortCode()
{
var q = (from job in Context.Job
select job.ShortCode).Distinct()
.Select(name => new ShortCode
{
Name = name,
Count = Context.Job.Count
(job => job.ShortCode.Trim() == name.Trim())
});
q = q.Where(job => job.Name != null);
return q;
}
The issue arises due to the fact that the "Context.Load()" function does not support loading multiple queries. Also If I try to declare to separate "Context.Load()" - Only one of them works........
How can I get around this!?!
This is kind of an indirect answer to you question so take it or leave it.
I have multiple AutoCompleteBoxes on one screen but I am not using a DDS. I gave up the DDS a long time ago when I kept running into problems. The main thing that I dont like about the dds is that so much of the logic is in the view.
What I do is bind my ACB directly to the entity and use a custom filter to do my complex searches. The list that it searches is an entity list that I populate using the get command
public EntityList<Person> Person
{
get { return _DomainContext.Persons; }
}
I could go into more detail but if you are lock into using the DDS then I will leave it at that.
There is one entity type per load, but there shouldn't be any problem having multiple loads running at the same time. Do you have Fiddler installed to watch what is actually going across the wire?