I have a wcf data service that is being consumed by my applications within my organization. It contains the following entities:
Client
User
Membership
The data is set-up in as multi-tenant and currently requires what we call a client string, or the string representation of the clients unique id, to be passed-in as part of the path and I extract that and add that in a query interceptor as a filter. This has worked beautifully up until recently when we had a new client come on-board that has a parent clientstring with many children. So the result is a query that includes a list of clientids passed-in for all queries as an "in (.....)". Not too bad but when this client now attempts to query memberships they have 9k+ clientids passed-in. The resultant query seems to run fairly quickly when executed alone but the entire db transaction is taking >30 seconds consistently.
I would ultimately like to find a way to bypass this filter for such queries if a specific userid is provided. The URL looks like this:
http://service.com/clienta/Users(1234)/memberships
If a specific id is given for users or memberships I can avoid the filter as the effectively achieves the same end-result. What we can't have is someone doing something like this:
http://service.com/clienta/memberships
Without any filter.
I realize this may be scattered and require much more detail as this was a bit rushed. Please ask any questions. Any help or guidance is much appreciated.
Can you use a query interceptor to intercept the request then if your condition is met to avoid the filtering, simply ignore it?
You could also move to a Service Method and pass in your arguments and do just about anything you want to do.
Related
Is there a way to send a correlation ID from C# code to SQL Server at the command level?
For instance, using x-correlation-id is an accepted way to track a request down to all parts of the system. we are looking for a way to pass this string value to stored procedure calls in SQL Server.
I spent sometime reading thru documents and posts but I was not able to find anything useful.
Can someone please let me know if there is a way to do this? The goal is to be able to track a specific call thru all services (which we can now) and DB calls (which we cannot and looking for a solution.)
I know the answer here is one year later. But in case, somebody has the same question.
Since EF core 2.2, MS provides a new method called "TagWith()" which you could pass your own annotation with the EF query into SQL server. In this way, you could easily track the SQL query with the same correlation id generated in your C# code.
https://learn.microsoft.com/en-us/ef/core/querying/tags
Unfortunately, this new feature is not available in EF 6. But it is not only us in this situation. If you just need a simple solution, you could check the thread here and MS documents.
If you need a more stable solution, you could check this NuGet plugin for EF 6 as well.
To pass your correlation id to SQL Server you have two options:
explicitly pass it as a parameter to your queries & stored procedures.
This is annoying as it requires work to change all your db calls to have a parameter like #correlationId, and often doesn't make sense having that parameter for simple data-retrieval queries. Perhaps you decide to only pass it for data-modification operations.
But on the positive side it's really obvious where the correlation info comes from (i.e. nobody reading the code will be confused) and doesn't require any additional db calls.
If all your data-modification is done using stored procs I think this is a good way to go.
use SQL Server's SESSION_CONTEXT(), which is a way you can set session state on a connection that can be retrieved from within stored procs etc.
You can find a way to inject it into your db layer (e.g. this) so the session context is always set on a connection before executing your real db calls. Then within your procs/queries you get the correlation id from the SESSION_CONTEXT and write to wherever you want to store it (e.g. some log table or as a column on tables being modified)
This can be good as you don't need to change each of your queries or procs to have the #correlationId parameter.
But it's often not so transparent how the session context is magically set. Also you need to be sure it's always set correctly which can be difficult with ORMs and connection pooling and other architectural complexities.
If you're not already using stored procs for all data modification, and you can get this working with your db access layer, and you don't mind the cost of the extra db calls this is a good option.
I wish this was easier.
Another option is to not pass it to SQL Server, but instead log all your SQL calls from the tier that makes the call and include the correlation id in those logs. That's how Application Insights & .NET seems to do it by default: logging SQL calls as a dependency along with the SQL statement and the correlation id.
I have an application with several Web API controllers and I now I have a requirement which is to be able to filter GET results by the object properties. I've been looking at using OData but I'm not sure if it's a good fit for a couple reasons:
The Web API controller does not have direct access to the DataContext, instead it gets data from our database through our "domain" layer so it has no visibility into our Entity Framework models.
Tying into the first item, the Web API deals with lightweight DTO model objects which are produced in the domain layer. This is effectively what hides the EF models. The issue here is I want these queries to be executed in our database but by the time the Web API method gets a collection from the domain layer all of the objects in the collection have been mapped to these DTO objects, so I don't see how the OData filter could possibly do it's job when the objects are once-removed from EF in this way.
This item may be the most important one: We don't really want to allow arbitrary querying against our Web API/Database. We just sort of want to leverage this OData library to avoid writing our own filters, and filter parsers/builders for every type of object that could be returned by one of our Web API endpoints.
Am I on the wrong track based on #3? If not, would we be able to use this OData library without significant refactoring to how our Web API and our EF interact?
I haven't had experience with OData, but from what I can see it's designed to be fed a Context and manages the interaction and returning of those models. I am definitely not a fan of returning Entities in any form to a client.
It's an ugly situation to be in, but when faced with this, my first course of action is to push back to the clients to justify their searching needs. The default request is almost always "Well, it would be nice to be able to search against everything." My answer to that is that I don't want to know what you want, I want to know what you need because I don't want to give you a loaded gun to shoot your own foot off with and then have you blame me because the system came grinding to a halt. Searching is a huge performance killer if it's too open-ended. It's hard to test for accuracy/relevance, and efficiently index for 100% of possible search cases when users only need 25% of those scenarios. If the client cannot tell you what searching they will need, and just want everything because they might need it, then they don't need it yet.
Personally I stick to specific search DTOs and translate those into the linq expressions.
If I was faced with a hard requirement to implement something like that, I would:
Try to push for these searches/reports to be done off a reporting replica that is synchronized with the live database. (To minimize the bleeding when some idiot managers fire up some wacky non-indexed search criteria so that it doesn't tie up the production DB where people are trying to do work.)
Create a new bounded DbContext specific for searching with separate entity definitions that only expose the minimum # of properties to represent search criteria and IDs.
Hook this bounded context into the API and OData. It will return "search results". When a user selects a search result, use the ID(s) against the API to load the applicable domain, or initiate an action, etc.
no. 1. is optional, a nice to have provided they can live with searches not "seeing" updated criteria until replicated. (I.e. a few seconds to minutes depending on replication strategy/size) Normally these searches are used for reporting-type queries so I'd push to keep these separate from the normal day-to-day searching options that users use. (I.e. an advanced search option or the like.)
I currently access a Web API endpoint serving up hierarchical objects (complex deals) using JSON/BSON. The objects are translated from entity framework objects stored as standard normalised data in a SQL Server database. This all works well.
However, as the number of these objects grows it becomes increasingly inefficient to serialise/deserialise them across the wire before filtering out those required at the client. Having methods for all objects or object-by-id is fine, but often there are more complex criteria for filtering which would require a myriad of different method signatures to fully capture. In an ideal world it would be possible to send Func<Deal,bool> to the Deals endpoint and this would provide the filtering mechanism from the client side to be enacted server-side. The premise being that different users will be interested in deals based on varying facets.
This may be mad, but is there any way that something along these lines can be achieved?
I do this by passing a "SearchCriteria" object to the search endpoint and then performing filtering at the server based on the values set in the various criteria properties. However, we do have a fairly well defined list of criteria and performing the filtering isn't too bad.
Alternatively, I've not used OData, but from what I understand this might be what you are looking for. If I was pondering this again I would investigate this.
https://learn.microsoft.com/en-us/aspnet/web-api/overview/odata-support-in-aspnet-web-api/odata-v4/create-an-odata-v4-endpoint
I'm using ArcGIS.PCL with C# to query information from an Arcgis server and REST web service. I know how to query a specific layer to see all the fields and information in general about it. But how can I query the server to return the list of layers?
I can use this URL for a specific layer (id=0): http://server/arcgis/rest/services/myassets/assets/MapServer/0
but if I don't know the ID of the layer, what can I do to iterate through all of them?
I know I can use this URL: http://server/arcgis/rest/services/myassets/assets/MapServer/ and the server returns all the information, but I don't know which method to use from this ArcGIS.PCL library to map the results to classes.
Also, if I query data from a specific layer and its fields, what are the parameters to use to return all the info of all the fields? At the moment I use "*" for outFields and "1=1" for the Where clause, but feels a bit hackish.
Anyone's got experience with this library?
Thanks!
There isn't a operation for that defined as of yet though there is still a way to do it. The test project has an example which will just map the result to a dictionary though you can just define your own type to do it too if you prefer.
If you want to get a collection of services for the site you can use DescribeSite.
Using * for outFields is correct if you want all the fields returned, otherwise you need to list the ones you want. Any where clause is needed as otherwise ArcGIS Server will throw an error so using 1=1 is the simplest way to get all data.
I am setting up a web service and i'm unsure about the most efficient design..
The web service is going to be called to populate a remote website with a list of people and their associated information. So, in the database there will be a "Person" table, then other tables for nicknames, addresses, phone numbers, etc. There may be 50 "People" returned for the page.
I would think I'd want to keep the # of web service calls to a minimum.. So is the best way to just have one call, GetPeople(), which returns all People and their information in one xml? Or would I want to maybe have GetPeople() return just a list of People, then have a GetPeopleInfo() call for each person?
Thanks in advance
Will you have an opportinuty to talk a bit more with the folks using the web service? The best answer would be depend on a number of factors, including how they are using it.
The simplest soution would be to create one web service method GetPeople() that executes a query joining the Person table with the rest of the tables and returnes all of the data in one flat table. If the client is just going to display all this information in a single list/table this would be the best option all around.
If, on the other hand, the client is going to generate a list of people and then have a click through to get more detailed information on a separate detail page, they might want a GetPeople() method that just returns the names/ids and a GetPeopleInfo() that pulls back the detail. If this isn't going to cause a performance problem on your system, this should be relatively straight forward.
I would probably build both - create a GetPeople() method that brings back all the data {as long as there isn't so much data that transportation becomes and issue} and a GetPersonInfo() method that allows they to pull back details on a specific person. This offers your client the greatest flexibility with not too much additional effort on your part.
You need to think about how your service is going to be used.
Will the users require all of that information most/all of the time? If so then it's appropriate for GetAllPeoplesDetails() to return everything
If they're going to only rarely need all the info then GetPeople() should return just a list of People.
If you're not sure then you could provide both GetAllPeoplesDetails() and GetPeople() and let the end user make that decision
Giving your users options especially when they are disperssed is your best option in my opinion. Having options such as:
GetPeopleList() : return list of people names
GetPeopleAll() : Get list of all people and all info
GetPersonInfo() : Get info of 1 person
Just my opinion but by giving your people a choice your making your application more usefull to them.
I think your GetPeople() call is appropriate. You can return a very lean result set and the end user can GetPeopleInfo() if they need it. If the result set is large, you may want to provide a paging option and result count in your method that allows the caller to pull back a certain amount of rows at a time.
XML is fine.
JSON is better.
Dave Ward at Encosia uses DataTransferObjects to serialize his "Person" objects.
I keep linking to this article because it's done so well, and this is a common problem.
Hope this helps, JP