I am using the QueryOperationResponse GetContinuation() method to attempt and page through a very large data set read off an odata wcf feed. I am using some OData helper libraries acquired from microsoft.
The problem is that the paging process seems to get stuck in an infinite loop, and never finishes. For example if I set the page size to 10000 records and there are 80000 records to retrieve, I observe that the loop continues for well over 8 iterations, when it should be finished.
Below is the class that queries the service and implements the paging (at the bottom). I also observed that the 'OriginalString' of the NextLinkUri never changes with each iteration, which I think is wrong? Hopefully I'm just missing something really obvious, I think this is the correct way to page:
private static IList<dynamic> Get(string serviceUri, NameValueCollection queryOptions, IAuthenticationScheme authenticationScheme)
{
string baseUri;
string entitySet;
string entityKey;
string queryString;
ValidateServiceUri(serviceUri, out baseUri, out entitySet, out entityKey, out queryString);
string resource = !string.IsNullOrEmpty(entityKey) ? entitySet + "(" + entityKey + ")" : entitySet;
DataServiceContext context = new DataServiceContext(new Uri(baseUri));
context.IgnoreMissingProperties = true;
DataServiceContextHandler handler = new DataServiceContextHandler(authenticationScheme);
handler.HandleGet(context);
DataServiceQuery<EntryProxyObject> query = context.CreateQuery<EntryProxyObject>(resource);
NameValueCollection options = HttpUtility.ParseQueryString(queryString);
options.Add(queryOptions);
foreach (string key in options.AllKeys)
{
query = query.AddQueryOption(key, options[key]);
}
QueryOperationResponse<EntryProxyObject> response = query.Execute() as QueryOperationResponse<EntryProxyObject>;
IList<dynamic> result;
if (options["$inlinecount"] == "allpages")
{
result = new DynamicEntityCollection(response.TotalCount) as IList<dynamic>;
}
else
{
result = new List<dynamic>();
}
foreach (EntryProxyObject proxy in response)
{
DynamicEntity entity = new DynamicEntity(proxy.Properties);
result.Add(entity);
}
while (response.GetContinuation() != null)
{
DataServiceQueryContinuation<EntryProxyObject> continuation = response.GetContinuation();
QueryOperationResponse<EntryProxyObject> nextResponse = context.Execute<EntryProxyObject>(continuation);
Console.WriteLine("Uri: " + continuation.NextLinkUri.OriginalString);
foreach (EntryProxyObject proxy in nextResponse)
{
DynamicEntity entity = new DynamicEntity(proxy.Properties);
result.Add(entity);
}
}
return result;
}
This is how I call the method:
return OData.Get("http://xxx.x.xxx.xx/MyDataService.svc/MyProducts", "$filter=Index gt 0");
You are always looking at the continuation of the first response. That stays constant (for obvious reasons). You need to look at the continuation of the nextResponse in your code to figure out if there's more data to read and to get the continuation to get the next page. The code above reads the first page and then reads the second page over and over again.
Related
I'm trying to list all items under a path for a determined bucket using Google Cloud Storage APIs. Under this path, there are more than 1000 items - which is the maximum number of items that ListObjects / ListObjectsAsync return.
In order to be able to repeat the call and get the next 1000 items, I need the NextPageToken from the previous response, that's how result pagination gets done! It's easy and reasonable.
However, I never get a NextPageToken within the responses. Am I missing something? This is my code so far. Some key points:
Althought there are more than +1000 items sharing the same prefix, I only get one response while looping through them in the enumerator.
The NextPageToken in the Console.WriteLine statement is always null.
Repeating the same call returns the same first 1000 objects over and over.
async IAsyncEnumerable<string> ListObjectsAsync(
string prefix, [EnumeratorCancellation] CancellationToken cancelToken)
{
var listObjectsOptions = new ListObjectsOptions
{
Fields = "items(name)"
};
var rawResponses = mGoogleClient.ListObjectsAsync(
mBucketName,
prefix,
listObjectsOptions).AsRawResponses();
using (var enumerator = rawResponses.GetEnumerator())
{
while (await enumerator.MoveNext(cancelToken))
{
foreach (var googleObject in enumerator.Current.Items)
yield return googleObject.Name;
}
Console.WriteLine(enumerator.Current.NextPageToken);
}
}
Thank you.
Apparently, I'm missing the nextPageToken field in the ListObjectsOptions used to make the request. Without specifying that field, the service won't return it - because of who knows why!
This code should work:
async IAsyncEnumerable<string> ListObjectsAsync(
string prefix, [EnumeratorCancellation] CancellationToken cancelToken)
{
var listObjectsOptions = new ListObjectsOptions
{
Fields = "items(name),nextPageToken"
};
var rawResponses = mGoogleClient.ListObjectsAsync(
mBucketName,
prefix,
listObjectsOptions).AsRawResponses();
using (var enumerator = rawResponses.GetEnumerator())
{
while (await enumerator.MoveNext(cancelToken))
{
foreach (var googleObject in enumerator.Current.Items)
yield return googleObject.Name;
}
Console.WriteLine(enumerator.Current.NextPageToken);
}
}
The nice thing is that you don't even have to use the NextPageToken explicitly. This is, you don't have to do something like this:
string token = null;
do
{
var options = new ListObjectsOptions
{
Fields = "items(name),nextPageToken",
PageToen = token
};
var rawResponses = mGoogleClient.ListObjectsAsync(
mBucketName,
prefix,
listObjectsOptions).AsRawResponses();
using (var enumerator = rawResponses.GetEnumerator())
{
while (await enumerator.MoveNext(cancelToken))
{
foreach (var googleObject in enumerator.Current.Items)
yield return googleObject.Name;
}
token = enumerator.Current.NextPageToken;
}
}
while (!string.IsNullOrEmpty(token));
...because the enumerator you got in rawResponses.GetEnumerator() will take care of using the token of a response to automatically fetch the next (if needed) while iterating them.
So the first piece of code is valid to iterate over +1000 objects in a single call.
I've been working for a few days on a performance problem.
Before I delve deeper I want to quickly explain how the specific service work.
I have a main Service that get a request and send requests to other micro-services but the single Entry Point for the user is to Main service, I thinks is more simple to understand with this image:
After the Main service get request from API he do some logic, query the Db and then get a list, every item on the list has Id, to get enrichment about every item the main service create request to one of the micro-service.
For example John request main service, main service get from Db a list of 90 items then the main service will create 90 calls to micro service and return to John single response that include 90 items.
Now the question is only about the right way to create async call to micro service.
This how I develop this part:
GetDetailsAsync(Id, result.Items, request.SystemComponentId);
private static void GetDetailsAsync(string Id, List<MainItem> items, int systemId)
{
var getDetailsTasks = new List<Task>();
foreach (MainItem single in items)
{
getDetailsTasks.Add(SetSingleDetailsAsync(Id, single, systemId));
}
Task.WhenAll(getDetailsTasks);
}
private static async Task SetSingleDetailsAsync(string Id, MainItem single, int systemId)
{
single.ActivityExtendedDetails = await ProcessItemDetailsRequest.GetItemDetailsAsync(Id, single.TypeId,
single.ItemId, systemId);
}
public static Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return Task.FromResult(res);
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
This code run and the result is not good enough, the CPU rises very quickly and becomes very high, I think that I has a problem on GetItemDetailsAsync func because I use client.GetStreamAsync(url).Result
when using .Result it's block until the task is completed.
So I do some minor change on GetItemDetailsAsync to try to be really async:
public static async Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = await client.GetStreamAsync(url))
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return res;
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
But now I get null where I supposed to get the data that come from Async function.
I try to debugging and I noticed something weird, everything happen likes as I would expect: the methods was called, request to micro-service was executed and get response but the response from the End-Point(which is found on main-service) return before the async method return from micro-service, that cause that I get null instead of my expected data.
I thinks that maybe I don't use correctly async\await and would be happy if anyone could explain how this behavior happens
I made a request to get a specific single value property from all events in a calendar with the Graph-SDK. To achieve this i used a filter according to the Graph APIs documentation (https://developer.microsoft.com/en-us/graph/docs/api-reference/v1.0/api/singlevaluelegacyextendedproperty_get). The filter i used was " id eq 'Boolean {00062002-0000-0000-C000-000000000046} Id 0x8223' ". Below is the code i used for this request.
public static async Task<Tuple<List<Event>, List<ICalendarEventsCollectionRequest>>> GetEventsSingleValuePropertyAsync(GraphServiceClient graphClient, String userId, String calendarId, String filterQuery, int top, String select)
{
List<ICalendarEventsCollectionRequest> requestList = new List<ICalendarEventsCollectionRequest>();
// filterQuery = "id eq 'Boolean {00062002-0000-0000-C000-000000000046} Id 0x8223'"
String filterSingleVP = "singleValueExtendedProperties($filter=" + filterQuery + ")";
List<Event> eventList = new List<Event>();
ICalendarEventsCollectionPage result = null;
ICalendarEventsCollectionRequest request = null;
if (calendarId == "")
request = graphClient.Users[userId].Calendar.Events.Request().Expand(filterSingleVP).Top(top).Select(select);
else
request = graphClient.Users[userId].Calendars[calendarId].Events.Request().Expand(filterSingleVP).Top(top).Select(select);
try
{
if (request != null)
{
result = await request.GetAsync();
requestList.Add(request);
eventList.AddRange(result);
}
if (result != null)
{
var nextPage = result;
while (nextPage.NextPageRequest != null)
{
var nextPageRequest = nextPage.NextPageRequest;
nextPage = await nextPageRequest.GetAsync();
if (nextPage != null)
{
requestList.Add(nextPageRequest);
eventList.AddRange(nextPage);
}
}
}
}
catch
{
throw;
}
return new Tuple<List<Event>, List<ICalendarEventsCollectionRequest>>(eventList, requestList);
}
I get all events and every event that matched the query gets expanded with the SingleValueLegacyExtendedProperty. The only thing that bothers me is that it looks like this:
"singleValueExtendedProperties":
[
{
"value":"true",
"id":"Boolean {00000000-0000-0000-0000-000000000000} Id 0x8223"
}
],
As you can see the value is present but the id now has an empty GUID.
I tested some other properties but i always had the same result.
I thought my filter compares the given "filterQuery" with the "id" in the answer.
Did i misunderstand something or is my request implementation just wrong?
That just looks like a bug on our side. Seems like the Guid prop might not be getting set or serialized correctly. I will bring it up to the team - thanks for the report.
-edit-
Yep, in fact we already have a fix for this that should be checked in in a few days.
-edit, edit-
Just for education's sake, the reason that it behaves this way is that the GUID that you are using is one of the "well known guids". In that case, our code is setting the well-known GUID field internally instead of the normal propertySetId guid field and REST always uses the propertySetId when rendering responses. The fix of course on our side is to use the well known guid field if the propertySetId is Guid.Empty.
-edit,edit,edit-
A fix was checked in for this and will begin normal rollout. It should reach WW saturation in a few weeks.
With ASP.NET WebApi, when I send GET api/question?page=0&name=qwerty&other=params and API should give result within pagination envelop.
For that, I'd like to put result and change given page querystring to other values.
I tried as below code but I got a bad feeling about this.
protected HttpResponseMessage CreateResponse(HttpStatusCode httpStatusCode, IEnumerable<Question> entityToEmbed)
// get QueryString and modify page property
var dic = new HttpRouteValueDictionary(Request.GetQueryNameValuePairs());
if (dic.ContainsKey("page"))
dic["page"] = (page + 1).ToString();
else
dic.Add("page", (page + 1).ToString());
var urlHelper = new UrlHelper(Request);
var nextLink= page > 0 ? urlHelper.Link("DefaultApi", dic) : null;
// put it in the envelope
var pageEnvelope = new PageEnvelope<Question>
{
NextPageLink = nextLink,
Results = entityToEmbed
};
HttpResponseMessage response = Request.CreateResponse<PageEnvelope<Question>>(httpStatusCode, pageEnvelope, this.Configuration.Formatters.JsonFormatter);
return response;
}
The NextPageLink gives a lot complex result.:
http://localhost/api/Question?Length=1&LongLength=1&Rank=1&SyncRoot=System.Collections.Generic.KeyValuePair%602%5BSystem.String%2CSystem.String%5D%5B%5D&IsReadOnly=False&IsFixedSize=True&IsSynchronized=False&page=1
My question is,
My page handling with Dictionary approach seems dirty and wrong. Is there better way to address my problem?
I don't know why urlHelper.Link(routeName, dic) gives such a verbose ToString result. How to get rid of unusable Dictionary-related properties?
The key issue probably in your code is the conversion to the HttpRouteValueDictionary. New it up instead and add in a loop all key value pairs.
The approach can be simplified quite a lot, and you should also probably want to consider using an HttpActionResult (so that you can more easily test your actions.
You should also avoid using the httproutevaluedictionary and instead write your UrlHelper like
urlHelper.Link("DefaultApi", new { page = pageNo }) // assuming you just want the page no, or go with the copying approach otherwise.
Where just pre calculate your page no (and avoid ToString);
Write it all in an IHttpActionResult that exposes an int property with the page No. so you can easily test the action result independently of how you figure out the pagination.
So something like:
public class QuestionsResult : IHttpActionResult
{
public QuestionResult(IEnumerable<Question> questions>, int? nextPage)
{
/// set the properties here
}
public IEnumerable<Question> Questions { get; private set; }
public int? NextPage { get; private set; }
/// ... execution goes here
}
To just get the page no, do something like:
Source - http://www.asp.net/web-api/overview/releases/whats-new-in-aspnet-web-api-21
string page = request.Uri.ParseQueryString()["page"];
or
you can use this extension method below (from Rick Strahl)
public static string GetQueryString(this HttpRequestMessage request, string key)
{
// IEnumerable<KeyValuePair<string,string>> - right!
var queryStrings = request.GetQueryNameValuePairs();
if (queryStrings == null)
return null;
var match = queryStrings.FirstOrDefault(kv => string.Compare(kv.Key, key, true) == 0);
if (string.IsNullOrEmpty(match.Value))
return null;
return match.Value;
}
I suspect I have a deadlock issue, but it's an odd one that I can't rationalize. I have an API that needs to verify a few things in order to process the call. As part of the business logic, I might have to make more of those same calls as well. In this case, if a particular piece of data associated with an entity is not found, we attempt to use a backup (if one is configured), which requires checking other entities. Eventually, the code will hang.
Let's just dive into the code (comments highlight the calls in question).
API Controller:
public async Task<HttpResponseMessage> Get(int entityID, string content, bool? useBackUp = true)
{
//Some look-ups here, no issues at all
//This works, but it's this method that has an issue later in the process.
SystemEntity entityObj =
await BusinessLayer.GetSystemEntityAsync(SystemEntityID);
if (entityObj == null)
{
return new HttpResponseMessage
{
StatusCode = System.Net.HttpStatusCode.BadRequest,
Content = new StringContent("Entity is unavailable.")
};
}
string text = BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp).Result.FirstOrDefault().Value;
if (text == null)
{
return new HttpResponseMessage {StatusCode = System.Net.HttpStatusCode.NoContent};
}
return new HttpResponseMessage
{
StatusCode = System.Net.HttpStatusCode.OK,
Content = new StringContent(text)
};
}
Business Layer:
public async Task<Dictionary<string, string>> GetContentTextAsync(int systemEntityID, List<string> contentNames, bool useBackUp)
{
Dictionary<string, string> records = new Dictionary<string, string>();
//We iterate for caching purposes
foreach (string name in contentNames)
{
string nameCopy = name;
string record = Cache.GetData(
string.Format("{0}_{1}_{2}", CONTENT, systemEntityID, name), () =>
DataLayer.GetCotnent(systemEntityID, nameCopy));
if (record == null && useBackUp)
{
List<int> entityIDs = new List<int> {systemEntityID};
int currentEntityID = systemEntityID;
//Here's that method again. This call seems to work.
SystemEntity currentEntity = await GetSystemEntityAsync(systemEntityID);
if (currentEntity != null && currentEntity.BackUpID.HasValue)
{
currentEntityID = (int) currentEntity.BackUpID;
}
while (!entityIDs.Contains(currentEntityID))
{
int id = currentEntityID;
record = Cache.GetData(
string.Format("{0}_{1}_{2}", CONTENT, systemEntityID, name), () =>
DataLayer.GetCotnent(id, nameCopy));
if (record != null) break;
entityIDs.Add(currentEntityID);
//This call seems to cause the deadlock
currentEntity = await GetSystemEntityAsync(currentEntityID);
if (currentEntity != null && currentEntity.BackUpID.HasValue)
{
currentEntityID = (int) currentEntity.UseBackupID;
}
}
}
if (record != null)
{
records.Add(name, record);
}
}
return records;
}
public async Task<SystemEntity> GetSystemEntityAsync(int systemEntityID)
{
SystemEntity systemEntity = await DataLayer.GetSystemEntity(
scc => scc.SystemEntityID == systemEntityID);
return systemEntity;
}
Data Layer:
public async Task<SystemEntity> GetSystemEntity(Expression<Func<SystemEntity, bool>> whereExpression)
{
using (EntityContext dbContext = createDbInstance())
{
//This is the last line that the debugger in VS 2013 brings me to. Stepping into this returns to whatever called the API method, waiting endlessly.
return await
dbContext.SystemEntity.Include(sc => sc.OtherEntity).Where(whereExpression).FirstOrDefaultAsync();
}
}
To recap: I call GetSystemEntityAsync three times. The first two times, it completes successfully. The third time, it hangs. If I comment out the first two calls so they don't run at all, the third one still hangs. If I remove the await and use just a normal FirstOrDefault in the return statement of the data layer method, then everything completes just fine.
Note: I have to keep the GetSystemEntityAsync method asynchronous. I cannot alter it to be synchronous.
What are the possible sources of the deadlock I'm encountering? I'm out of ideas on how to solve it.
Which one of these async calls is not like the other?
This one, I suspect:
string text = BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp).Result.FirstOrDefault().Value;
Try changing it to this:
string text = (await BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp)).FirstOrDefault().Value;
The possible source of the deadlock is described by Stephen Cleary in his "Don't Block on Async Code" blog post.