Redis key partitioning practices with linked items - c#

I'm using a Redis database and ServiceStack client for it. I have a class called "Post" which has a property GroupId. Now when I'm storing this class the key is "urn:post:2:groupid:123". Now if I want to find all posts related to one group i need to use SearchKeys("urn:*groupid:123") method to retrieve all posts related to one group. Is this best practice to use Redis DB or should I convert my post key into form of "urn:groupid:123"post:2" ? If so how I can achieve this?
Post class:
public class Post
{
public const string POST_INCREMENT_KEY = "POST_INCREMENT";
public string Id { get; set; }
public string Message { get; set; }
public string GroupId { get; set; }
public void BuildId(long uniqueId)
{
Id = uniqueId + ":groupid:" + GroupId;
}
}
Code for storing post:
var post = new Post
{
GroupId = groupId,
Message = Request.Form["message"]
};
post.BuildId(_client.Increment(Post.POST_INCREMENT_KEY, 1));
_client.Store(post);

The best practice in redis is to maintain an index of the relationship you want to query.
Manually maintaining an index in Redis
An index is just a redis SET containing the related Ids you want to maintain, given that you want to "retrieve all posts related to one group" I would maintain the following index:
const string GroupPostIndex = "idx:group>post:{0}";
So that everytime you store a post, you also want to update the index, e.g:
client.Store(post);
client.AddItemToSet(GroupPostIndex.Fmt(groupId), post.Id);
Note: Redis SET operations are idempotent in that adding an item/id multiple times to a SET will always result in there being only one occurrence of that item in the SET, so its always safe to add an item to the set whenever storing a POST without needing to check if it already exists.
Now when I want to retrieve all posts in a group I just need to get all the ids from the SET with:
var postIds = client.GetAllItemsFromSet(GroupPostIndex.Fmt(groupId));
Then fetch all the posts with those ids:
var posts = redis.As<Post>().GetByIds(postIds);
Using ServiceStack.Redis Related Entities API's
The above shows what's required to maintain an index in Redis yourself, but as this is a common use-case, ServiceStack.Redis also offers a high-level typed API that you can use instead.
Which lets you store related entities with:
client.As<Group>().StoreRelatedEntities(groupId, post);
Note: this also takes care of storing the Post
and retrieve them with:
var posts = client.As<Group>().GetRelatedEntities<Post>(groupId);
It also offers other convenience API's like quickly finding out how many posts there are within a given group:
var postsCount = client.As<Group>().GetRelatedEntitiesCount<Post>(groupId);
and deleting either 1 or all entities in the group:
client.As<Group>().DeleteRelatedEntity<Post>(groupId, postId);
client.As<Group>().DeleteRelatedEntities<Post>(groupId); //all group posts

Related

Realm + Xamarin Forms Update Query

I'm using Realm + Xamarin Forms to do what I figure is about the most basic operation possible: a list view shows a collection of items, with a search bar filtering the results as the user types.
I have a get only collection property used as the list view's items source, initially populated from a Realm query, and this gets updated automatically with any changes to data, but I can't figure out how to update the search text without adding a set and literally replacing the entire collection.
This is very inefficient--I assume this is triggering re-registration of a bunch of notify-changed event listeners for the collection and each item in it and generally causing mass chaos with each letter tapped.
In the past I've created my own wrapping observable collection with a search method to handle this and I suppose that is an option here as well, but is there any way to do this with Realm? That is, to update the query without recreating the entire collection, some way to re-run the original query?
Update: This technique not longer works.
https://github.com/realm/realm-dotnet/issues/1569
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...it also differs from the behavior of LINQ to Objects, where every iteration will reevaluate expressions, meaning that changes to both sides of a condition will affect the result. A Realm query will evaluate the right-hand sides of the conditions on the first run.
When you construct a query that contains Where parameters that are based upon non-Realm based conditions the query results do not update when those variable/parameters are changed unless you update/execute the query again.
Realm queries are live, in the sense that they will continue to represent the current state of the database.
So what I do is to create a filter class (RealmObject), then if you instance a "filter object", save it to Realm, you can base your Linq's Where parameters upon one or more of the "filter" properties. Updating this RealmObject filter via Realm.Add(filterObject, true) your queries based upon that object are also updated
Realm queries are live, in the sense that they will continue to represent the current state of the database.
The results are lighting fast filtering that works great in any UI Search routine.
Example Model:
public class ARealmClass : RealmObject
{
public int Key { get; set; }
public string KeyString { get; set; }
}
public class ARealmClassFilter : RealmObject
{
[PrimaryKey]
public int Key { get; set; }
public int FilterKeyBy { get; set; }
}
Populate a Realm with some test data
var realm = Realm.GetInstance();
var all = realm.All<ARealmClass>();
if (all.Count() == 0)
{
realm.Write(() =>
{
for (int i = 0; i < 1000; i++)
{
var obj = new ARealmClass { Key = i, KeyString = i.ToString() };
realm.Add(obj);
}
});
}
Dynamic Live Query Example:
var realm = Realm.GetInstance();
var all = realm.All<ARealmClass>();
Console.WriteLine(all.Count());
var filterItem = new ARealmClassFilter { Key = 1, FilterKeyBy = 500 };
realm.Write(() =>
{
realm.Add(filterItem);
});
var filtered = all.Where(_ => _.Key > filterItem.FilterKeyBy);
Console.WriteLine(filtered.Count());
realm.Write(() =>
{
filterItem.FilterKeyBy = 750;
realm.Add(filterItem, true);
});
Console.WriteLine(filtered.Count());
Output:
2017-04-24 11:53:20.376 ios_build_foo[24496:3239020] 1000
2017-04-24 11:53:20.423 ios_build_foo[24496:3239020] 499
2017-04-24 11:53:20.425 ios_build_foo[24496:3239020] 249
Note: Quoted text # https://realm.io/docs/xamarin/latest/api/linqsupport.html

using string.split in entity to traverse tree depth

i have the following self-referencing table
public partial class products_category
{
public long id { get; set; }
public string category_name { get; set; }
public string category_description { get; set; }
//self referencing to table id
public Nullable<long> Parent_Id { get; set; }
public string navPath {get; set; }
}
here string navpath contains all the leading parents for a child categroy, say:
"Clothes" = 1 Parent_id=null, navpath=""
"Silk" = 2 Parent_id=1 navpath="1"
"Silk Suit"=3 parent_id=2 navpath="1-2"
"Saree" =4 parent_id=3 navpath="1-2-3"
"Dress Material"=5 parent_id=1 navpath="1" and so on....
now as per this scenario i want to access the flattend tree for frther processing for a certain depth only say to level 2 or until level 4 depth of children associated with navpath.
my idea regarding this issue was to approach using linq to ef this way:
var catTrees = db.products_category.Where(pc => pc.navpath.Split('-').Length < 4).ToList();
i am using the following link to do further traversing and tree generation:
https://bitlush.com/blog/recursive-hierarchical-joins-in-c-sharp-and-linq
and it is doing a great work so far, the only issue is i dont want to pre select whole table for processing. i want to achieve paging and a certain level of depth for first iteration, so i can maintain performance in case of thousand of records. [think of this as a category hierarchy or blog/youtube comments hierarchy].
but using the above ef linq command is giving the following error:
The LINQ expression node type 'ArrayLength' is not supported in LINQ to Entities.
i checked with ef docs and other places in SO to know that string.split doesn't work with EF implicitly. but can we apply it using extension methods or can this tree selection have alternate approach without using string.split and hitting DB only ones?
please advice.
This looks like an issues with building SQL code out of your LINQ mpre specifically SQL which takes a string splits it on dash and counts the elements.
if you dont hate the idea of loading into memory then you can force anything :)
var catTrees = db.products_category.ToList().Where(pc => pc.navpath.Split('-').Length < 4).ToList();
The trick here is to force the execution of the SQL by adding the .ToList() when we want the data from the database. This is called realizing the data.
Even with that realization trick the count is faster
var catTrees = db.products_category.ToList().Where(pc => pc.navpath.Count(a => a == '-') < 3).ToList();
these solutions are essentially the same as
List<Result> filter() {
List<Result> r = new List<Result>();
foreach(var a in db.products_category) {
if(a.navpath.Count(a => a == '-') < 3) {
r.add(a);
}
}
return r;
}
When thinking about it the filter method is somewhat less memory intensive as it reads one and one and never stores everything in memory. (in theory at least, only a few really knows what the .NET compiler does in the shadows)
I would advice you against using the navpath for checking depth.
If you can change your model, you could add an additional numeric Depth field for each category and populate it according its navpath, then you could select them from your code in this way:
var catTrees = db.products_category.Where(pc => pc.Depth < 3).ToList();
There are many ways to populate that new column, but the bottom line is that you will have to do it just once (given that you keep track of it every time you modify the navpath of a category).
One possible way of populating it would be looping through all categories, something like:
var allCategories = db.products_category.ToList();
foreach(var category in allCategories)
{
var depth = category.navpath == "" ? 0 : category.navpath.Split('-').Length + 1;
category.Depth = depth;
}
allCategories.SubmitChanges();

Update and edit many to many tables Entity Framework

I have four tables (Locations, Events, MainTags, EventTags) and between the Events table and each one of the other three tables is a many-to-many relationship.
Now I want to edit and display data from those tables on the same page but I'm not sure what is the right way to do it.
For now I'm doing it like this taking the id from each individual entry in the Events table and then searching for the same id in each of the other table, saving those results in a list and later displaying it on the page.
(adc is my connection to the database)
List<Event> allEvents = adc.Events.ToList();
List<Location> testTHis = new List<Location>();
foreach (Event eve1 in allEvents)
{
var query_test = from a in adc.Locations where a.Id == eve1.id select a;
testTHis = query_test.ToList();
}
But this seems to be rather an intensive work and it doesn't look nice so I'm wondering if there is a much better way to do this especially later when a database has a lot of data in it this doesn't seem like a good idea cause it will take a lot of time to complete. Anyone can help me to figure out a better solution?
Your Event entity should expose 3 navigational properties:
public virtual ICollection<Location> Locations { get; set; }
public virtual ICollection<MainTag> MainTags { get; set; }
public virtual ICollection<EventTag> EventTags { get; set; }
Then when you query for events you tell EF to include all tags/event tags and locations.
context.Events
.Include(x => x.Locations)
.Include(x => x.MainTags)
.Include(x => x.EventTags)
.ToList()
This will retrieve events as well as their Locations, MainTags and EventTags all together in one query.
One word of warning with include is, each event will have its own Tag/Location, even if its referenced multiple times.
E.g. lets say you have 2 Events,
CodePerfDotNetEvent { Tag={ID=1, Name="Blog"} }
MichalCiechanEvent { Tag={ID=1, Name="Blog"} }
When you use include you will get 2 different Tag objects even though they represent the same value in the database.
Otherwise you leave LazyLoading and ProxyGeneration on, and let EF make multiple calls to the database each time you access one of the Locations/EventTags/MainTags virtual properties.

How to add or remove a many-to-many relationship in Entity Framework?

There's a many-to-many UserFeed table that stands between User and Feed and denotes a twitter-like follow relationship.
It only has two fields, which form a composite key: UserID and FeedID.
I need to write a method that will subscribe or unsubscribe a user from a feed based on a boolean flag.
public void SetSubscriptionFlag (int userId, int storeId, bool subscribe)
{
}
I'm new to Entity Framework so I'm trying to find and follow an "EF-ish" way to accomplish this.
My initial thoughts are:
Instead of working with the middle UserFeed class, I should create a many-to-many Subscriptions property (EDIT: hit limitations here);
After I've done so, I'll need to fetch a User instance by ID, check whether it has given Feed in its Subscriptions and add/delete it depending on the flag and current existence;
Figure out how to avoid racing conflicts when there is a time interval before the check and adding/deleting and user manages to submit two adding or deletion requests;
Optimize my code as to avoid unneccessary SELECTs, if any occur, because all I really want to do is a single SELECT and single INSERT/DELETE.
A relevant code snippet and comment on my points is highly appreciated.
Thanks!
You can use dummy objects - it definitely works for insert and I hope it can be used for delete as well:
Create new relation:
var user = new User() { Id = userId };
context.Users.Attach(user);
var store = new Store() { Id = storeId };
context.Stores.Attach(store);
// Now context trackes both entities as "existing"
// and you can build a new relation
user.Subscriptions.Add(store);
context.SaveChanges();
Remove existing relation:
var user = new User() { Id = userId };
var store = new Store() { Id = storeId };
user.Subscriptions.Add(store);
context.Users.Attach(user);
// Now context trackes both entities as "existing"
// with "existing" relation so you can try to remove it
user.Subscriptions.Remove(store);
context.SaveChanges();

What is the correct way of using Entity Framework?

I have a DB like this that I generated from EF:
Now I'd like to add a "fielduserinput" entity so I write the following code:
public bool AddValueToField(string field, string value, string userId)
{
//adds a value to the db
var context = new DBonlyFieldsContainer();
var fieldSet = (from fields in context.fieldSet
where fields.fieldName.Equals(field)
select fields).SingleOrDefault();
var userSet = (from users in context.users
where users.id.Equals(userId)
select users).SingleOrDefault();
var inputField = new fielduserinput { userInput = value, field = fieldSet, user = userSet };
return false;
}
Obviously it's not finished but I think it conveys what I'm doing.
Is this really the right way of doing this? My goal is to add a row to fielduserinput that contains the value and references to user and field. It seems a bit tedious to do it this way. I'm imagining something like:
public bool AddValueToField(string userId, string value, string fieldId)
{
var context = new db();
var newField = { field.fieldId = idField, userInput = value, user.id = userId }
//Add and save changes
}
For older versions of EF, I think you're doing more or less what needs to be done. It's one of the many reasons I didn't feel EF was ready until recently. I'm going to lay out the scenario we have to give you another option.
We use the code first approach in EF 4 CTP. If this change is important enough, read on, wait for other answers (because Flying Speghetti Monster knows I could be wrong) and then decide if you want to upgrade. Keep in mind it's a CTP not an RC, so considerable changes could be coming. But if you're starting to write a new application, I highly recommend reading some about it before getting too far.
With the code first approach, it is possible to create models that contain properties for a reference to another model and a property for the id of the other model (User & UserId). When configured correctly setting a value for either the reference or the id will set the id correctly in the database.
Take the following class ...
public class FieldUserInput{
public int UserId {get;set;}
public int FieldId {get;set;}
public virtual User User {get;set;}
public virtual Field Field {get;set;}
}
... and configuration
public class FieldUserInputConfiguration{
public FieldUserInputConfiguration(){
MapSingleType(fli => new {
userid = fli.UserId,
fieldid = fli.FieldId
};
HasRequired(fli => fli.User).HasConstraint((fli, u)=>fli.UserId == u.Id);
HasRequired(fli => fli.Field).HasConstraint((fli, f)=>fli.FieldId == f.Id);
}
}
You can write the code...
public void CreateField(User user, int fieldId){
var context = new MyContext();
var fieldUserInput = new FieldUserInput{ User = user, FieldId = fieldId };
context.FieldUserInputs.Add(fieldUserInput);
context.SaveChanges();
}
... or vice versa with the properties and everything will work out fine in the database. Here's a great post on full configuration of EF.
Another point to remember is that this level of configuration is not necessary. Code first is possible to use without any configuration at all if you stick to the standards specified in the first set of posts referenced. It doesn't create the prettiest names in the database, but it works.
Not a great answer, but figured I'd share.

Categories