I have a document db collection that contains about 1500 docs. My eventual goal is to build an api with the above as my persistence layer. I read the docs (https://azure.microsoft.com/en-us/blog/documentdb-paging-support-with-top-and-more-query-improvements/) that mentions to use request continuation token via feed options for paging.
Have the following method on a console app to understand the same: The problem or the issue here is that the first time the query is executed the continuation token=null but the second time the query is executed the continuation token = "some value"
So my question is shouldn't the continuation token be set the first time itself? or am I missing something obvious
//field
private static readonly FeedOptions DefaultOptions = new FeedOptions { EnableCrossPartitionQuery = true,MaxItemCount = 1};
//method
private static async Task<bool> QueryItemDocuments(string collectionLink)
{
IDocumentQuery<Item> query = client.CreateDocumentQuery<Item>(collectionLink, new SqlQuerySpec()
{
QueryText = "SELECT * FROM items",
},DefaultOptions).AsDocumentQuery();
var loopNumber = 0;
while (query.HasMoreResults)
{
loopNumber++;
var results=(await query.ExecuteNextAsync<Item>()).ToList();
Console.WriteLine("Count {0}",results.Count);
Console.WriteLine("Loopnumber :{0} Token {1}", loopNumber, DefaultOptions.RequestContinuation ?? "Null");
Console.WriteLine("Id: {0},Name: {1}", results[0].Id, results[0].Name);
}
return query.HasMoreResults;
}
sample results from console :
Count 1
Loopnumber :1 Token Null //why is this null?
Id: 1dbaf1d0-0125-11e0-88a8-005056957023,Name:Energy Drink
Count 1
Loopnumber :2 Token - RID:jq4gAIZqMwACBBBBBBB==#RT:1#TRC:1#PKRID:0
Id: 244a8230-0231-11e0-8c8b-005056957023,Name: Gluten Free Dish
Edits: Ideally, I would send back the continuation token on the response headers allowing the client to fetch more if needed (on my web api) - so my method would like
public Get(int? page=1,int? size=20)
so when I send the response back with the first 20 I need to send back the continuation token for the client to request more: so if the continuation token is not set the first time - meaning when I retrieve the 1st batch of 20, how do I do this?
You are on the right track. Two thoughts:
You don't need to manually process the continuation token when using HasMoreResults and ExecuteNext. HasMoreResults checks if there is a continuation token and when you call ExecuteNext, it's automatically added.
I suspect that the reason you don't see a continuation on the first round is that it's the outgoing DefaultOptions and it's not updated until it goes out again on the second ExecuteNext. I'm thinking that your output on the second loop is actually the first continuation token.
okay, so i figured that it might be a bug in the SDK or by design - do not know, however when I query for the result using Document Db's rest end points - the continuation token is indeed set on the first batch's response. Here is the sample code that used:
private static async Task QueryItemDocumentsUsingRestApi()
{
//About rest end points to query documents here https://msdn.microsoft.com/en-us/library/azure/mt670897.aspx
IEnumerable<string> continuationTokens = null;
var continToken = string.Empty;
var verb = "POST";
var resourceType = "docs";
var resourceLink = string.Format("dbs/{0}/colls/{1}/docs", DatabaseName, CollectionName);
var resourceId = string.Format("dbs/{0}/colls/{1}", DatabaseName, CollectionName)
var authHeader = GenerateAuthSignature(verb, resourceId, resourceType, authorizationKey, "master", "1.0"); // look here for how this is generated https://github.com/Azure/azure-documentdb-dotnet/blob/master/samples/rest-from-.net/Program.cs
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("x-ms-date", _utcDate);
client.DefaultRequestHeaders.Add("x-ms-version", "2015-08-06");
client.DefaultRequestHeaders.Remove("authorization");
client.DefaultRequestHeaders.Add("authorization", authHeader);
client.DefaultRequestHeaders.Add("x-ms-documentdb-isquery", "True");
client.DefaultRequestHeaders.Add("x-ms-max-item-count", "5");
client.DefaultRequestHeaders.Add("x-ms-continuation", string.Empty);
var qry = new RestSqlQuery { query = "SELECT * FROM items" };
var result = client.PostWithNoCharSetAsync(new Uri(new Uri(endpointUrl), resourceLink), qry).Result;
var content = await result.Content.ReadAsStringAsync();
var pagedList = JsonConvert.DeserializeObject<PagedList>(content);
if (result.Headers.TryGetValues("x-ms-continuation", out continuationTokens))
{
continToken = continuationTokens.FirstOrDefault();
}
}
}
Related
In a custom C# Winforms app, I'm using the Azure DevOps REST API Update Comments call to update work item comments using async/await
My call to UpdateComment_Async is in a tight loop designed to submit all update comment requests for a single work item and then process each comment update as it completes.
Following is a mix of p-code and C# for (almost) working code. In the case where there are 2 (or more) comments to update, the first update returns HttpResponseMessage.Status 200 (success), but the second update submitted returns HttpResponseMessage.Status 409 "Conflict". I assume that ADS has the work item locked for the first update, and so the 2nd update fails with the 409. I think I proved this by sleeping the thread for 5 seconds after the call to UpdateComment_Async. With the sleep in place, both updates work.
Is there a way to manage the series of calls to UpdateComment_Async so that subsequent calls aren't done until the previous one is complete?
CODE
// get comment(s) for one work item .......
foreach (comment in workItem)
{
newCommentText = "new comment text blah-blah-blah";
Task<Tuple<string, string, HttpResponseMessage>> updateCommentTask = UpdateComment_Async(projectUrl, workItem.Id.ToString(), comment.Id.ToString(), newCommentText);
// I put a thread.sleep(5000) right here
updateCommentTaskList.Add(updateCommentTask);
}
// Process update comments tasks as they complete
while (updateCommentTaskList.Count > 0)
{
Task<Tuple<string, string, HttpResponseMessage>> finishedUpdateCommentTask = await Task.WhenAny(updateCommentTaskList);
// Get Results
Tuple<string, string, HttpResponseMessage> updateCommentTaskResult = finishedUpdateCommentTask.Result;
// process updateCommentTaskResult
// etc
// etc
// etc
updateCommentTaskList.Remove(finishedUpdateCommentTask);
}
//*******************************
public async Task<Tuple<string, string, HttpResponseMessage>> UpdateComment_Async(string projectUrl, string workItemId, string commentId, string commentNumber, string newCommentText)
{
HttpResponseMessage responseResult = null;
#region MAKE JSON REQUEST BODY
IList<ClsUpdateComment> updateFieldJsonList = new List<ClsUpdateComment>();
updateFieldJsonList.Clear();
// Note: This code works but is not in compliance with MS docs on 2 counts.
//
// 1) The MS docs on comment update say that the body should look like fig 1.
// The only I could do this was to create a new ClsUpdateComment and then add it to
// List<ClsUpdateComment>. Adding the ClsUpdateComment object to a list causes the [ ]
// to be created when the list is serialized. To make this work, I had to serialize
// just the ClsUpdateComment object so that when serialzed, it ends up looking like Fig 2 (no brackets)
//
// 2) application/json-patch+json causes error 415 - unsupported media type to occur. application/json
// works.
/*
Fig 1
[
{
"text": "Moving to the right area path - Fabrikam-Git"
}
]
Fig 2
{
"text": "Moving to the right area path - Fabrikam-Git"
}
*/
ClsUpdateComment updateFieldJson = new ClsUpdateComment
{
Text = $"{newCommentText}"
};
updateFieldJsonList.Add(updateFieldJson);
#endregion MAKE JSON REQUEST BODY
#region SUBMIT UPDATE REQUEST
string request = $"{projectUrl}/_apis/wit/workitems/{workItemId}/comments/{commentId}?api-version=5.1-preview.3";
JsonSerializerSettings jsonSerializerSettings = new JsonSerializerSettings
{
StringEscapeHandling = StringEscapeHandling.EscapeHtml,
};
string updateFieldJsonSerialized = JsonConvert.SerializeObject(updateFieldJson, Formatting.None, jsonSerializerSettings);
using (HttpClient client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true, ClientCertificateOptions = ClientCertificateOption.Manual }))
{
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
HttpRequestMessage httpRequestMessage = new HttpRequestMessage(new HttpMethod("PATCH"), request)
{
Content = new StringContent(updateFieldJsonSerialized, Encoding.UTF8, "application/json")
};
using (responseResult = await client.SendAsync(httpRequestMessage))
{
// don't need Content, just the HttpResponseMessage
// //string content = await responseResult.Content.ReadAsStringAsync();
}
}
#endregion SUBMIT UPDATE REQUEST
return Tuple.Create(workItemId, commentNumber, responseResult);
}
It's not an answer, but after trying to do mass updates of comments using the REST API in various ways, I've come to the conclusion that it simply doesn't support mass updates of comments
below is the code that I do pagination in Azure Cosmos. In that function I return the ContinuationToken of the FeedResponse. The first request to get the first page is fine and it return the Continuation Token. However if I used that token in the next request then the API return error 500.
I also notice that the ContinuationToken return from FeedRespone seem like in Json format like that. I have tried to get the token section only, or even copy the whole json but no cigar though
"nextToken": "[{"token":"+RID:~UVURALkfIb4FAAAAAAAAAA==#RT:1#TRC:3#RTD:hCgamV5sp6dv/pVR3z0oBTMxMzIuMTQuNDFVMTY7MjY7NDIvOTk3MzIxMlsA#ISV:2#IEO:65567#QCF:1#FPC:AQEAAAAAAAAACAAAAAAAAAA=","range":{"min":"","max":"FF"}}]"
Response from the First Page with Token return
Enter Return Token to next request and error 500
Function Code
public virtual async Task<(IEnumerable<TDomain>, string token)> ListAsync(List<ISpecification<TEntity>> specifications, PageOptions pageOptions, CancellationToken cancellationToken)
{
var container = await GetContainer(cancellationToken);
string token = null;
var result = new List<TDomain>();
QueryRequestOptions options = new QueryRequestOptions()
{
MaxItemCount = pageOptions.MaxResults
};
options.MaxItemCount = pageOptions.MaxResults;
try
{
var query = container
.GetItemLinqQueryable<TEntity>(false, pageOptions.NextToken, options)
.Specify(specifications);
var iterator = _cosmosLinqQuery.GetFeedIterator(query);
var response = await iterator.ReadNextAsync(cancellationToken);
token = response.ContinuationToken; // return a token
foreach (var item in response)
{
var mapped = _mapper.ToDomain(item);
result.Add(mapped);
}
}
catch (Exception ex)
{
var exception = new DataAccessException("Unexpected error while listing items", ex);
exception.Data["ContainerName"] = ContainerName;
throw exception;
}
return (result,token);
}
Your second screenshot is showing that you are passing a token that starts with +RID... which is not how the previous token starts (previous token starts with [{"token").
Could you be dropping the JSON wrapping attributes that are part of the token?
The second call should be passing exactly [{"token":"+RID:~UVURALkfIb4FAAAAAAAAAA==#RT:1#TRC:3#RTD:hCgamV5sp6dv/pVR3z0oBTMxMzIuMTQuNDFVMTY7MjY7NDIvOTk3MzIxMlsA#ISV:2#IEO:65567#QCF:1#FPC:AQEAAAAAAAAACAAAAAAAAAA=","range":{"min":"","max":"FF"}}].
Keep in mind that you are also sending it in the URL, so there might be character escaping there too.
I have a method that deletes multiple events. Currently the code is as following:
public async Task DeleteEvents(IEnumerable<string> eventExternalIds)
{
foreach(var eventExternalId in eventExternalIds)
{
await DeleteEvent(eventExternalId);
}
}
public async Task DeleteEvent(string eventExternalId)
{
await GraphClient
.Users[Username]
.Calendars[CalendarName]
.Events[eventExternalId]
.Request()
.DeleteAsync();
}
I would imagine it won't perform well with any significant number of id's to delete. Is there a way to delete them all in a batch(es) instead of each individually?
msgraph-sdk-dotnet v1.15.0 or above
For msgraph-sdk-dotnet version 1.15.0 or above the support for Batch request has been introduced via BatchRequestContent class
Example
//1. construct a Batch request
var batchRequestContent = new BatchRequestContent();
var step = 1;
foreach (var eventId in eventIds)
{
var requestUrl = graphClient
.Me
.Events[eventId]
.Request().RequestUrl;
var request = new HttpRequestMessage(HttpMethod.Delete, requestUrl);
var requestStep = new BatchRequestStep(step.ToString(), request, null);
batchRequestContent.AddBatchRequestStep(requestStep);
step++;
}
//2. Submit request
var batchRequest = new HttpRequestMessage(HttpMethod.Post, "https://graph.microsoft.com/v1.0/$batch");
batchRequest.Content = batchRequestContent;
await graphClient.AuthenticationProvider.AuthenticateRequestAsync(batchRequest);
var httpClient = new HttpClient();
var batchResponse = await httpClient.SendAsync(batchRequest);
//3. Process response
var batchResponseContent = new BatchResponseContent(batchResponse);
var responses = await batchResponseContent.GetResponsesAsync();
foreach (var response in responses)
{
if (response.Value.IsSuccessStatusCode)
{
//...
}
}
Issues
while targeting NetCore 2.1 or above or .NET Framework
NullReferenceException exception might occur, to address this issue
you could switch to 1.16.0-preview.1 (details)
Limitations
Note: A batch cannot hold more that 20 requests
msgraph-sdk-dotnet v1.14.0 or older
For previous versions, the following example demonstrates how to implement a support for Batch request:
var batchRequest = new BatchRequest();
foreach (var eventId in eventIds)
{
var request = graphClient.Me.Events[eventId].Request();
batchRequest.AddQuery(request,HttpMethod.Delete);
}
var responses = await graphClient.SendBatchAsync(batchRequest);
where BatchRequest is a custom class which adds support for JSON Batching
I have an azure function app that I call from a slack slash command.
Sometimes the function takes a little while to return the data requested, so I made that function return a "Calculating..." message to slack immediately, and run the actual processing on a Task.Run (the request contains a webhook that I post back to when I finally get the data) :
Task.Run(() => opsData.GenerateQuoteCheckMessage(incomingData, context.FunctionAppDirectory, log));
This works mostly fine, except every now and then when people are calling the function from slack, it will return the data twice. So it will show one "Calculating..." message and then 2 results returned from the above function.
BTW, Azure functions start with :
public static async Task
Thanks!
UPDATE : here is the code for the function:
[FunctionName("QuoteCheck")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous)]HttpRequestMessage req, TraceWriter log, ExecutionContext context)
{
var opsHelper = new OpsHelper();
string bodyContent = await req.Content.ReadAsStringAsync();
var parsedBody = HttpUtility.ParseQueryString(bodyContent);
var commandName = parsedBody["command"];
var incomingBrandId = parsedBody["text"];
int.TryParse(incomingBrandId, out var brandId);
var responseUrl = parsedBody["response_url"];
var incomingData = new IncomingSlackRequestModel
{
UserName = parsedBody["user_name"],
ChannelName = parsedBody["channel_name"],
CommandName = commandName,
ResponseUri = new Uri(responseUrl),
BrandId = brandId
};
var opsData = OpsDataFactory.GetOpsData(context.FunctionAppDirectory, environment);
Task.Run(() => opsData.GenerateQuoteCheckMessage(incomingData, context.FunctionAppDirectory, log));
// Generate a "Calculating" response message based on the correct parameters being passed
var calculatingMessage = opsHelper.GenerateCalculatingMessage(incomingData);
// Return calculating message
return req.CreateResponse(HttpStatusCode.OK, calculatingMessage, JsonMediaTypeFormatter.DefaultMediaType);
}
}
And then the GenerateQuoteCheckMessage calculates some data and eventually posts back to slack (Using Rest Sharp) :
var client = new RestClient(responseUri);
var request = new RestRequest(Method.POST);
request.AddParameter("application/json; charset=utf-8", JsonConvert.SerializeObject(outgoingMessage), ParameterType.RequestBody);
client.Execute(request);
Using Kzrystof's suggestion, I added a service bus call in the function that posts to a queue, and added another function that reads off that queue and processes the request, responding to the webhook that slack gives me :
public void DeferProcessingToServiceBus(IncomingSlackRequestModel incomingSlackRequestModel)
{
var serializedModel = JsonConvert.SerializeObject(incomingSlackRequestModel);
var sbConnectionString = ConfigurationManager.AppSettings.Get("SERVICE_BUS_CONNECTION_STRING");
var sbQueueName = ConfigurationManager.AppSettings.Get("OpsNotificationsQueueName");
var client = QueueClient.CreateFromConnectionString(sbConnectionString, sbQueueName);
var brokeredMessage = new BrokeredMessage(serializedModel);
client.Send(brokeredMessage);
}
I have an interesting issue... For my windows phone 8.1 universal app, I call a method (CommAuthState), defined in my class, from my code:
Here is truncated code which is calling this method.
public async void ShowInitialPosts(object sender, RoutedEventArgs e)
{
#if WINDOWS_PHONE_APP
Windows.Phone.UI.Input.HardwareButtons.BackPressed += HardwareButtons_BackPressed;
#endif
//Get reference to the App setting container...
Windows.Storage.ApplicationDataContainer appRoamingSettings = Windows.Storage.ApplicationData.Current.RoamingSettings;
//Get the value for the logged in state and remembering option....
string commLoggedInValue = (string)appRoamingSettings.Values["CommLoggedIn"];
//If not logged in, redirect to the Community sign in page...
if (commLoggedInValue != "Yes")
{
this.Frame.Navigate(typeof(CommSignIn));
}
else if (commLoggedInValue == "Yes")
{
await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
//Check the access token and its validity...
forum_Hepler.CommAuthState(errorMessage, pgbar, pgText, ServerNetworkError);
});
//Get the access token value...
string myAccessTokenValue = (string)appRoamingSettings.Values["MyAccessToken"];
//Set the access token to the page variable...
accesstoken = myAccessTokenValue;
// Show the progress bar
mycontrols.progressbarShow(pgbar, pgText);
//Set the initial forum URL and page index..
downloadedPostsIndex = 0;
forumUrl = forumUrl + downloadedPostsIndex;
// Make a REST API call with Oauth token........
string telligentResult = await forum_Hepler.MyForumPostsOauth(forumUrl, accesstoken, errorMessage);
.....
.......
}
Here is my class method:
public async void CommAuthState(string errormessage, ProgressBar myprogressbar, TextBlock mytextblock, TextBlock myservernetworkerror) {
// Start showing the progress bar...
mycontrols.progressbarShow(myprogressbar, mytextblock);
//Get the access token value...
string myAccessTokenValue = (string) appRoamingSettings.Values["MyAccessToken"];
//Get the refresh token value...
string myRefreshTokenValue = (string) appRoamingSettings.Values["MyRefreshToken"];
//Get the original access token obtain time....
long myAccessTokenObtainedTimeValue = (long) appRoamingSettings.Values["MyAccessTokenObtainedTime"];
//Convertig date/time back to DateTime object....
origAccessTokenObtainedTime = DateTime.FromBinary(myAccessTokenObtainedTimeValue);
currentDateTime = DateTime.Now;
//Check to see if access token has expired....
accessTokenTimeElasped = currentDateTime - origAccessTokenObtainedTime;
accessTokenTimeElapsedSecs = accessTokenTimeElasped.TotalSeconds;
//Get the value of the access token expiration....
int MyAccessTokenExpiresValue = (int) appRoamingSettings.Values["MyAccessTokenExpires"];
if (accessTokenTimeElapsedSecs <= MyAccessTokenExpiresValue) {
//Get the long GUID value to be used for the GetOauthStuff function below...
string myLongGuidValue = (string) appRoamingSettings.Values["MyLongGuid"];
//Make a GET call to the OAUTH endpoint to get another Access Token by sending the refresh token ....
// string telligentOauthResult = await GetOauthStuff(oauthEndPoint, MyLongGuidValue, keyName, keySecret, errormessage);
string telligentOauthResult = await GetRefreshedOauthStuff(oauthEndPoint, myLongGuidValue, myRefreshTokenValue, keyName, keySecret, errormessage);
if (telligentOauthResult != null) {
// Creating a list out of the JSON object returned by the Telligent API endpoint...
ForumHelper.OAuthObject mytelligentResult = JsonConvert.DeserializeObject < ForumHelper.OAuthObject > (telligentOauthResult);
var accessToken = mytelligentResult.OAuthToken.access_token;
// Get the access token and the time and save it to settings
accessTokenObtainedTime = DateTime.Now;
var accessTokenError = mytelligentResult.OAuthToken.error;
var accessTokenTime = mytelligentResult.OAuthToken.expires_in;
var refreshToken = mytelligentResult.OAuthToken.refresh_token;
//Save access token to the app settings...
appRoamingSettings.Values["MyAccessToken"] = accessToken;
//Converting to binary format before saving since app settings cannot save in DateTime format and also we can convert it back later to the DateTime object
appRoamingSettings.Values["MyAccessTokenObtainedTime"] = accessTokenObtainedTime.ToBinary();
} else {
// Stop showing the progress bar...
mycontrols.progressbarNoShow(myprogressbar, mytextblock);
//Show the error message...
myservernetworkerror.Visibility = Windows.UI.Xaml.Visibility.Visible;
}
}
}
All works fine and I can step through my code to see how it is functioning till I get into the GetRefreshedOauthStuff method. here is the GetRefreshedOauthStuff method for reference:
public async Task < string > GetRefreshedOauthStuff(string url, string MyGUID, string MyResfreshToken, string KeyName, string KeySecret, string ErrorMessage) {
//Setting up Httpclient to send to the Telligent API call.....
var client = new System.Net.Http.HttpClient();
var adminKey = String.Format("{0}:{1}", KeySecret, KeyName);
var adminKeyBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(adminKey));
// Send customized headers with the token and impersonation request
client.DefaultRequestHeaders.Add("Rest-User-Token", adminKeyBase64);
client.DefaultRequestHeaders.Add("Cache-Control", "no-cache");
//Adding the long GUID to the OAUTH endpoint URL...
url = url + MyGUID + "&refresh_token=" + MyResfreshToken;
//For getting the list of posts....
System.Net.Http.HttpResponseMessage telligentResponse = await client.GetAsync(url);
if (telligentResponse.StatusCode == System.Net.HttpStatusCode.OK || telligentResponse.StatusCode == System.Net.HttpStatusCode.Forbidden) {
// Read the HTTP response content....
HttpContent responseContent = telligentResponse.Content;
// Read the response content as string.....
return await responseContent.ReadAsStringAsync();
} else {
throw new Exception("Error connecting to " + url + " ! Status: " + telligentResponse.StatusCode);
//Pop up the server or network issue message...
//mycontrols.popupMessages(ErrorMessage, "Network or Server error!");
////Navigate back to the main page....
//this.rootFrame.Navigate(typeof(MainPage));
//return null;
}
}
As soon as my code executes this line:
System.Net.Http.HttpResponseMessage telligentResponse = await client.GetAsync(url);
it steps out of this method instead of fully completing it and goes to this line in my starting method ShowInitialPosts:
string telligentResult = await forum_Hepler.MyForumPostsOauth(forumUrl, accesstoken, errorMessage);
This obviously creates problems and fails my logic. what am I doing wrong here? How do I fix this?
Any help or pointers are much appreciated
Thanks
That's exactly what await is supposed to do. It doesn't block the thread from executing - it tells the method to return control to whoever called it and then resume execution when the async method has finished.
Eric Lippert has a great blog post about this.
http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/asynchronous-programming-in-c-5-0-part-two-whence-await.aspx