I am new to Google Cloud Datastore (NoSQL db). I am developing a web application in C# for which I have successfully configured authentication for google Datastore API by setting GOOGLE_APPLICATION_CREDENTIALS.
I am also able to insert data (entities) in the Datastore kind (I think "kind" refers to tables in Google). However, I am unable to retrieve entities based on retrieval, for example, I want to retrieve entities from Datastore, from a kind named "Task" where a column named "word" has the word "hello" in it.
I would also like to know whether I can keep a column which has a datatype of "string" as key column (like a primary key column in SQL database tables). I also need help as to how I can update a column/property for a particular entity.
string projectId = "xyz";
DatastoreDb db = DatastoreDb.Create(projectId);
Query query = new Query("Task")
{
Filter = Filter.Equal("word", "hello")
};
DatastoreQueryResults dres = db.RunQuery(query);
First of all, a kind in Cloud Datastore is indeed the equivalent of a table in SQL. Apart from that, each entity has a unique identifier which includes a key name string and a numeric ID (integer).
If you want to retrieve a particular entity, you can use the Lookup function based on the key of that entity, like this:
Entity task = _db.Lookup(_sampleTask.Key);
This also applies if you want to retrieve multiple entities:
var keys = new Key[] { _keyFactory.CreateKey(1), _keyFactory.CreateKey(2) };
var tasks = _db.Lookup(keys[0], keys[1]);
For updating a particular entity, you can modify its properties and then store it using the same key it had before:
_sampleTask["priority"] = 5;
_db.Update(_sampleTask);
This link can be quite useful to understand how to use Cloud Datastore with .NET and C#. It includes a tutorial for building a bookshelf based in Datastore and deploying it in App Engine.
As for what to use as a key, Datastore uses an ancestor structure, just like a file system directory. One entity can be set as the child of another, that becomes its parent, and this can continue until a full structure is created with ancestors and children.
This is an example of a key from an entity which has ancestors:
[User:alice, TaskList:default, Task:sampleTask]
This is the key of a root entity, which has no parents:
[User:alice]
For more info about the ancestor paths, check this link.
In GCP Web Console, you can run queries directly in sql, from there you can do a quick test if the query is returning expected results.
Datastore requires indexes to be crated before you can query on them, from web console when you run the query, you will know more about it.
If you see results on web console, but not able to retrieve when queries programmatically, you can check GCP logs for any exceptions to debug the issue.
Related
This is a bit of a puzzle I'm trying to figure out.
I am working on a system where we have a number of company records saved in the database. Some of these records are duplicates and are no longer wanted/required.
However, several external systems are still mapping to these invalid records. If we were to delete them entirely it would cause errors to the systems still wanting to get the detail of that company.
The ideal workflow I would like would be;
The external system looks up Company ID X.
The current system has a table which has a record of all the remapped records, so when the request comes in, the table specifies to redirect Company ID X to Company ID Y.
There are a number of endpoints that could be altered one-by-one to do this - but it would be time-consuming, resulting in lots of repetition too.
My question is, using Entity Framework and .Net - is there a smart way of achieving this workflow?
My initial thoughts were to do something with the constructor for the company object, which repopulates the object from EF if a 'redirect' exists, but I don't know if this will play nice with navigation properties.
Would anyone have an idea?
Thanks very much.
You can create a column with foreign key for the same table to express the single unique valid company.
For example, you can add DuplicateOf column:
ALTER TABLE [Company]
ADD COLUMN [DuplicateOf] bigint NULL,
FOREIGN KEY [DuplicateOf] REFERENCES [Company] ([Id]);
and express this relation in your code:
public class Company
{
// ...
public Company DuplicateOf { get; set; }
// May be useful, hides check for duplicate logic:
public bool IsDuplicate => DuplicateOf != null;
// May be useful as well,
// returns the non-duplicate uniue company, not a duplicate, either linked or current:
public Company EffectiveCompany => DuplicateOf ?? this;
}
You will have to address EffectiveCompany when you want to work with non-duplicate and maintain this column to always point to the correct record. It will also result into additional query, if eager-loaded.
Another idea is to have a stored procedure GetCompany(bigint id) which will return the effective record - if DuplicateOf exists, or record itself otherwise. It will be good for your external systems and will let you hide all this stuff behind abstraction layer of stored procedure. If you decide to change it in future, then you can easily update it without breaking external systems.
However, for you it isn't always convenient to work with stored procedures with EF.
These are just ideas and not the best solutions, anyway.
In my opinion, the best solution would be to get rid of duplicates, update data everywhere and forget forever about this mess of duplicated data.
I am looking at the following sample code to include referenced documents and avoid round trip.
var order = session.Query<Order>()
.Customize(x => x.Include<Order>(o=>o.CustomerId)) // Load also the costumer
.First();
var customer = session.Load<Customer>(order.CustomerId);
My question is how does Raven know that this o=>o.CustomerId implies Customer document/collection? At no time was the entity Customer supplied in the query to get the Order entity. Yet Raven claims that the 2nd query to get Customer can be done against the cache, w/o any network trip.
If it's by naming convention, which seems like a very poor/fragile/brittle convention to adopt, what happens when I need to include more than 1 documents?
Eg. a car was purchased under 2 names, so I want to link back to 2 customers, the primary and secondary customer/driver. They're both stored in the Customer collection.
var sale = session.Query<Sale>()
.Customize(x => x.Include<Sale>(o=>o.PrimaryCustomerId).Include<Sale>(o=>o.SecondaryCustomerId)) // Load also the costumer
.First();
var primaryCustomer = session.Load<Customer>(order.PrimaryCustomerId);
var secondaryCustomer = session.Load<Customer>(order.SecondaryCustomerId);
How can I do the above in 1 network trip? How would Raven even knows that this o=>o.PrimaryCustomerId and o=>o.SecondaryCustomerId are references to the one and same table Customer since obviously the property name and collection name don't line up?
Raven doesn't have the concept of "tables". It does know about "collections", but they are just a convenience mechanism. Behind the scenes, all documents are stored in one big database. The only thing that makes a "collection" is that each document has a Raven-Entity-Name metadata value.
Both the examples you showed will result in one round trip (each). Your code looks just fine to me.
My question is how does Raven know that this o=>o.CustomerId implies Customer document/collection? At no time was the entity Customer supplied in the query to get the Order entity.
It doesn't need to be supplied in the query. As long as the data stored in the CustomerId field of the Sale document is a full document key, then that document will be returned to the client and loaded into session.
Yet Raven claims that the 2nd query to get Customer can be done against the cache, w/o any network trip.
That's correct. The session container tracks all documents returned - not just the ones from the query results. So later when you call session.Load using the same document key, it already has it in session so it doesn't need to go back to the server.
Regardless of whether you query, load, or include - the document doesn't get deserialized into a static type until you pull it out of the session. That's why you specify the Customer type in the session.Load<Customer> call.
If it's by naming convention, which seems like a very poor/fragile/brittle convention to adopt ...
Nope, it's by the value stored in the property which is a document key such as "customers/123". Every document is addressable by its document key, with or without knowing the static type of the class.
what happens when I need to include more than 1 documents?
The exact same thing. There isn't a limit on how many documents can be included or loaded into session. However, you should be sure to open the session in a using statement so it is disposed properly. The session is a "Unit of Work container".
How would Raven even knows that this o=>o.PrimaryCustomerId and o=>o.SecondaryCustomerId are references to the one and same table Customer since obviously the property name and collection name don't line up?
Again, it doesn't matter what the names of the fields are. It matters that the data in those fields contains a document id, such as "customers/123". If you aren't storing the full string identifier, then you will need to build the document key inside the lambda expression. In other words, if Sale.CustomerId contains just the number 123, then you would need to include it with .Include<Sale>(o=> "customers/" + o.CustomerId).
I am adding users to an engagement plan programmaticaly, using the following code:
VisitorManager.AddVisitor("salesforce\\fred.smith","{67F395B9-2C29-4B73-9382-69E0FCB6A546}");
This works fine, but I also need to store a custom value against the visit. Any ideas on how to do this?
I can't set vistor.CurrentVisit.Profiles as there is no setter available and there is no method available to add to it.
I am able to add tags with the following code
vistor.Tags.Add("opportunityId", "006M0000004xnLh");
However, when I try to retrieve them from within a custom automation action, the values returned are null. I guess because it was set programmaticaly and is being retrieved from a different session? The code I used to retrieve the tags is:
var opportunityTagRow = visitorTags.Find("opportunityId");
and also:
var allOpportunityRows = visitorTags.GetAll("opportunityId");
Any ideas on how best to persist data for a visit?
I think you're confusing the concepts of "visitors" and "users" in Sitecore. In the Sitecore DMS a "Visitor" is essentially a cookie on a machine which maps to an entry in the Visitors table of the analytics database. A named User is someone who is logged into Sitecore and is stored in the .NET membership table in the core database. When you add a tag to a visitor this adds an entry to the VisitorTags table which is basically just a name-value pair which is mapped to the visitor entry in the Visitors table. In your case "salesforce\fred.smith" is not a Visitor but a User. You can retrieve the current visitor using Sitecore.Analytics.Tracker.Visitor or you can retrieve a different visitor by ID using the VisitorFactory. If alternatively you do want to persist data against a user, you can use the Users profile, which is again different to DMS and visitor profiles. You will find the relevant classes for Sitecore users under Sitecore.Security.
Edit: I would do what you want in the following way:
//Enroll user in plan
VisitorManager.AddVisitor("salesforce\\fred.smith", new ID("{67F395B9-2C29-4B73-9382-69E0FCB6A546}"));
//Add data to USER profile not visitor
Sitecore.Security.Accounts.User user = Sitecore.Security.Accounts.User.FromName("salesforce\\fred.smith", true);
user.Profile["opportunityId"] = "006M0000004xnLh";
//Get user from automation action parameter and get data from profile
Sitecore.Security.Accounts.User user = Sitecore.Security.Accounts.User.FromName(automationStatesRow.UserName, true);
string data = user.Profile["opportunityId"];
All visit data is stored in the Sitecore Analytics databases - See the visitor tags table. A visitor tag has an VisitorId that can be used to lookup the visitor. You have two options to get this data (along with another other Analytics data) - use the Sitecore Analytics API (recommended) or to retrieve the data you want by constructing custom sql calls (can get messy). Here's how I done something similar in an engagement action using the analytics API:
public override AutomationActionResult Execute(VisitorDataSet.AutomationStatesRow automationStatesRow, Item action, bool isBackgroundThread)
{
Visitor visitor = VisitorFactory.GetVisitor();
string tagValue = visitor.Tags["myTag"]
//Do stuff
}
I've got a SQL Server database that I'm trying to build a RESTful API for.
I'm using ADO.Net and Linq to retrieve a single row from a table like this:
[HttpGet]
public tTrip getTripById(Guid id)
{
var _trip = (from trips in db.tTrip
where trips.ID == id
select trips).FirstOrDefault();
return _trip;
}
When I debug the code the correct object is retrieved. If I keep running however, there will be no response. I'm guessing that's because for every foreign key present in the returned row, ADO does another lookup through the other mapped tables which slows down everything by a lot.
If I only select a single column that doesn't contain any FKCs everything works fine.
Any ideas how I can turn off the FKC lookup for that fetched object?
Thank you!
I found the problem - In the ObjectContext class (that's where the 'db' variable comes from btw), I had the ContextOptions.LazyLoadingEnabled variable set to true.
Set it to false and the application returns only the Guid for every entry instead of loading the entry details from the database.
I am developing an HRM application to import and export xml data from database. The application receives exported xml data for the employee entry. I imported the xml file using linq to xml, where I converted the xml into respective objects. Then I want to attach (update) the employee objects.
I tried to use
//linqoper class for importing xml data and converts into IEnumerable employees object.
var emp = linqoper.importxml(filename.xml);
Using (EmployeedataContext db = new EmployeedatContext){
db.attachAllonSubmit(emp);
db.submitchange();
}
But I got error
“An entity can only be attached as modified without original state if it declares as version member or doesn't have an update check policy”.
I have also an option to retrieve each employee, and assign value to the new employee from xml data using this format.
//import IEnumerable of Employee objects
var employees = = linqoper.importxml(filename.xml)
using(Employeedatacontext db = new Employeedatacontext){
foreach(var empobj in employees)
{
Employee emp = db.Employee.where(m=>m.id==empobj.Id);
emp.FirstName=empobj.FirstName;
emp.BirthDate=empobj.BirthDate;
//….continue
}
db.submitChanges();
}
But the problem with the above is I have to iterate through the whole employee objects, which is very tiresome.
So is there any other way, I could attach (update) the employee entity in the database using LINQ to SQL.
I have seen some similar links on SO, but none of them seems to help.
https://stackoverflow.com/questions/898267/linq-to-sql-attach-refresh-entity-object
When linq-to-sql saves the changes to the database, it has to know properties of the object has been changed. It also checks if a potentially conflicting update to the database have been done during the update (optimistic concurrency).
To handle those cases LINQ-to-SQL needs two copies of the object when attaching. One with the original values (as present in the DB) and one with the new, changed values. There is also a more advanced mechanism involving a version member which is mapped to a rowversion column.
The linq-to-sql way to update a set of data is to first read all data from the database, then update the objects retrieved form the database and finally call SubmitChanges(). That would be my first approach in your situation.
If you experience performance problems, then it's time to go outside of linq-to-sql's toolbox. A solution with better performance is to load the new data into a separate staging table (for best performance, use bulk insert). Then run a SQL command or Stored Procedure that does the actual merging of data. The SQL Merge clause is excellent for this kind of updates.
LINQ to SQL is proper ORM, but if you want to take control of create/update/delete in your hand; than you can try some simple ORMs which just provide ways to do CRUD operations. I can recommend one http://crystalmapper.codeplex.com, it is simple yet powerful.
Why CrystalMapper?
I built this for large financial transaction system with lots of insert and update operations. What I need is speed and control of insert/update serving complex business scenarios ... hitting multiple tables just for one transaction.
When I put this to use in social text processing platform, it serves very well there too.