We Keep Coding

Azure Cosmos Pagination: FeedResponse.ContinuationToken not working in the next request

below is the code that I do pagination in Azure Cosmos. In that function I return the ContinuationToken of the FeedResponse. The first request to get the first page is fine and it return the Continuation Token. However if I used that token in the next request then the API return error 500.
I also notice that the ContinuationToken return from FeedRespone seem like in Json format like that. I have tried to get the token section only, or even copy the whole json but no cigar though
"nextToken": "[{"token":"+RID:~UVURALkfIb4FAAAAAAAAAA==#RT:1#TRC:3#RTD:hCgamV5sp6dv/pVR3z0oBTMxMzIuMTQuNDFVMTY7MjY7NDIvOTk3MzIxMlsA#ISV:2#IEO:65567#QCF:1#FPC:AQEAAAAAAAAACAAAAAAAAAA=","range":{"min":"","max":"FF"}}]"
Response from the First Page with Token return
Enter Return Token to next request and error 500
Function Code
public virtual async Task<(IEnumerable<TDomain>, string token)> ListAsync(List<ISpecification<TEntity>> specifications, PageOptions pageOptions, CancellationToken cancellationToken)
{
var container = await GetContainer(cancellationToken);
string token = null;
var result = new List<TDomain>();
QueryRequestOptions options = new QueryRequestOptions()
{
MaxItemCount = pageOptions.MaxResults
};
options.MaxItemCount = pageOptions.MaxResults;
try
{
var query = container
.GetItemLinqQueryable<TEntity>(false, pageOptions.NextToken, options)
.Specify(specifications);
var iterator = _cosmosLinqQuery.GetFeedIterator(query);
var response = await iterator.ReadNextAsync(cancellationToken);
token = response.ContinuationToken; // return a token
foreach (var item in response)
{
var mapped = _mapper.ToDomain(item);
result.Add(mapped);
}
}
catch (Exception ex)
{
var exception = new DataAccessException("Unexpected error while listing items", ex);
exception.Data["ContainerName"] = ContainerName;
throw exception;
}
return (result,token);
}

Your second screenshot is showing that you are passing a token that starts with +RID... which is not how the previous token starts (previous token starts with [{"token").
Could you be dropping the JSON wrapping attributes that are part of the token?
The second call should be passing exactly [{"token":"+RID:~UVURALkfIb4FAAAAAAAAAA==#RT:1#TRC:3#RTD:hCgamV5sp6dv/pVR3z0oBTMxMzIuMTQuNDFVMTY7MjY7NDIvOTk3MzIxMlsA#ISV:2#IEO:65567#QCF:1#FPC:AQEAAAAAAAAACAAAAAAAAAA=","range":{"min":"","max":"FF"}}].
Keep in mind that you are also sending it in the URL, so there might be character escaping there too.

Authorized endpoint returns empty 401 xml response

I have a controller method which looks like this
[HttpGet]
[Authorize(Roles = "Administrator,Employee")]
public async Task<ActionResult<ItemsDTO<Reservation>>> GetReservations([FromQuery] SliceDTO slice)
{
var role = this.User.FindFirst(ClaimTypes.Role).Value;
var userName = User.FindFirstValue(ClaimTypes.NameIdentifier);
var reservations = await _reservationService.GetReservations(userName, role, slice.Page, slice.RowsPerPage, (slice.SearchTerm != null) ? slice.SearchTerm : "");
return Ok(reservations);
}
When the user is authorized it works perfectly fine. When the user is not authorized, I get a 401 response as expected, however, for some reason the content type is set to xml. This does not really break anything, but creates console errors when trying to parse an empty xml response body. How can I fix this?

Azure Maps Batch Geocoding Status 202 12+ Hours Later

I'm using RestSharp to programmatically build/make/deserialize a call to the Azure Maps API for batch geocoding. Currently testing the process with a 5 address batch--something I expected to go quickly after all the posts about "10k addresses in minutes." But a request I make successfully yesterday is still not available, only showing the "Accepted 202" status that the documentation says means it's still processing...and there's no outage showing at the status page.
I've replicated these calls and results using Postman, so I'm not sure there's a code problem per se...but it wouldn't be the first time I got tunnel vision and overlooked something obvious.
My POST call is generated with the code below, and returns an OK status with the necessary Location header with what looks like a valid link.
public RestRequest CreateBatchRequest()
{
var request = new RestRequest($"{_batchAddressEndpoint}subscription-key={_apiToken}", Method.POST);
request.AddQueryParameter("api-version", _version);
var batchRequestBody = GenerateQueryBatch();
var requestBodyJson = JsonConvert.SerializeObject(batchRequestBody);
request.AddHeader("Content-Type", "application/json");
request.AddParameter("undefined", requestBodyJson, ParameterType.RequestBody);
return request;
}
protected AzureBatchRequest GenerateQueryBatch()
{
var requestBody = new AzureBatchRequest();
foreach (var address in Addresses)
{
var addressString = $"{address.Address}, {address.City}, {address.State}";
if (!string.IsNullOrEmpty(_country))
addressString = $"{address.Address}, {address.City}, {address.State}, {_country.ToUpper()}";
requestBody.Queries.Add($"?query={addressString}&limit={_resultLimit}");
}
return requestBody;
}
This gives me a body parameter for the request that appears to match the documentation (actual addresses hidden for privacy reason, but they've been successfully geocoded with other services)...
{
undefined={"queries":[
"?query=123 MAIN ST, LOS ANGELES, CA&limit=3",
"?query=123 MAIN ST, PLEASANTVILLE, CA&limit=3",
"?query=123 MAIN ST, STOCKTON, CA&limit=3",
"?query=123 MAIN ST, SAN DIEGO, CA&limit=3",
"?query=123 MAIN ST, REDDING, CA&limit=3"
]}
}
I get the Location header value and make the GET call with it using the code below...
public List<Coordinate> DeserializeBatchResponse(RestResponse response)
{
var batchLink = response.Headers.Where(header => header.Name.Equals("Location")).FirstOrDefault();
var request = new RestRequest(batchLink.Value.ToString(), Method.GET);
var batch = SendRequest(request);
if (batch.StatusCode == System.Net.HttpStatusCode.Accepted)
{
var isProcessing = true;
while (isProcessing)
{
Thread.Sleep(TimeSpan.FromSeconds(60));
request = new RestRequest(batchLink.Value.ToString(), Method.GET);
batch = SendRequest(request);
if (batch.StatusCode != System.Net.HttpStatusCode.Accepted)
isProcessing = false;
}
}
}
And it never leaves that loop. When I hardcode the URL returned from yesterday's POST request, it has the same behavior--as ditto when tried in Postman to isolate from the rest of my code.
Does anyone have any insight?
UPDATE
We discovered that after creating a new plan at a higher tier (the S1 rather than S0 tier) there was no noticeable delay on the batch calls. Still not a solution, per se, because that prices us out of the product for production purposes, but possibly a fix for others until the updates mentioned in the accepted answer come to fruition.

This is to be expected at the moment while in preview. This will be moving out of preview soon and will be significantly faster.

Document Db Request Continuation Token Not set on the 1st iteration

I have a document db collection that contains about 1500 docs. My eventual goal is to build an api with the above as my persistence layer. I read the docs (https://azure.microsoft.com/en-us/blog/documentdb-paging-support-with-top-and-more-query-improvements/) that mentions to use request continuation token via feed options for paging.
Have the following method on a console app to understand the same: The problem or the issue here is that the first time the query is executed the continuation token=null but the second time the query is executed the continuation token = "some value"
So my question is shouldn't the continuation token be set the first time itself? or am I missing something obvious
//field
private static readonly FeedOptions DefaultOptions = new FeedOptions { EnableCrossPartitionQuery = true,MaxItemCount = 1};
//method
private static async Task<bool> QueryItemDocuments(string collectionLink)
{
IDocumentQuery<Item> query = client.CreateDocumentQuery<Item>(collectionLink, new SqlQuerySpec()
{
QueryText = "SELECT * FROM items",
},DefaultOptions).AsDocumentQuery();
var loopNumber = 0;
while (query.HasMoreResults)
{
loopNumber++;
var results=(await query.ExecuteNextAsync<Item>()).ToList();
Console.WriteLine("Count {0}",results.Count);
Console.WriteLine("Loopnumber :{0} Token {1}", loopNumber, DefaultOptions.RequestContinuation ?? "Null");
Console.WriteLine("Id: {0},Name: {1}", results[0].Id, results[0].Name);
}
return query.HasMoreResults;
}
sample results from console :
Count 1
Loopnumber :1 Token Null //why is this null?
Id: 1dbaf1d0-0125-11e0-88a8-005056957023,Name:Energy Drink
Count 1
Loopnumber :2 Token - RID:jq4gAIZqMwACBBBBBBB==#RT:1#TRC:1#PKRID:0
Id: 244a8230-0231-11e0-8c8b-005056957023,Name: Gluten Free Dish
Edits: Ideally, I would send back the continuation token on the response headers allowing the client to fetch more if needed (on my web api) - so my method would like
public Get(int? page=1,int? size=20)
so when I send the response back with the first 20 I need to send back the continuation token for the client to request more: so if the continuation token is not set the first time - meaning when I retrieve the 1st batch of 20, how do I do this?

You are on the right track. Two thoughts:
You don't need to manually process the continuation token when using HasMoreResults and ExecuteNext. HasMoreResults checks if there is a continuation token and when you call ExecuteNext, it's automatically added.
I suspect that the reason you don't see a continuation on the first round is that it's the outgoing DefaultOptions and it's not updated until it goes out again on the second ExecuteNext. I'm thinking that your output on the second loop is actually the first continuation token.

okay, so i figured that it might be a bug in the SDK or by design - do not know, however when I query for the result using Document Db's rest end points - the continuation token is indeed set on the first batch's response. Here is the sample code that used:
private static async Task QueryItemDocumentsUsingRestApi()
{
//About rest end points to query documents here https://msdn.microsoft.com/en-us/library/azure/mt670897.aspx
IEnumerable<string> continuationTokens = null;
var continToken = string.Empty;
var verb = "POST";
var resourceType = "docs";
var resourceLink = string.Format("dbs/{0}/colls/{1}/docs", DatabaseName, CollectionName);
var resourceId = string.Format("dbs/{0}/colls/{1}", DatabaseName, CollectionName)
var authHeader = GenerateAuthSignature(verb, resourceId, resourceType, authorizationKey, "master", "1.0"); // look here for how this is generated https://github.com/Azure/azure-documentdb-dotnet/blob/master/samples/rest-from-.net/Program.cs
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("x-ms-date", _utcDate);
client.DefaultRequestHeaders.Add("x-ms-version", "2015-08-06");
client.DefaultRequestHeaders.Remove("authorization");
client.DefaultRequestHeaders.Add("authorization", authHeader);
client.DefaultRequestHeaders.Add("x-ms-documentdb-isquery", "True");
client.DefaultRequestHeaders.Add("x-ms-max-item-count", "5");
client.DefaultRequestHeaders.Add("x-ms-continuation", string.Empty);
var qry = new RestSqlQuery { query = "SELECT * FROM items" };
var result = client.PostWithNoCharSetAsync(new Uri(new Uri(endpointUrl), resourceLink), qry).Result;
var content = await result.Content.ReadAsStringAsync();
var pagedList = JsonConvert.DeserializeObject<PagedList>(content);
if (result.Headers.TryGetValues("x-ms-continuation", out continuationTokens))
{
continToken = continuationTokens.FirstOrDefault();
}
}
}

Missing Authorization header in Basic authentication

I used following code to implement Basic Authentication filter in my ASP.Net MVC app. everything is working good in local machine while it's not working in production server and it keeps prompting login box because Request.Headers["Authorization"] is null.
I used fiddler to get headers for this request and Authorization header was there with expected values. I have no idea why Request.Headers["Authorization"] is always null :|
I also created a new project only with this filter and one controller and published in server, guess what !? it's working...
public class RequireBasicAuthenticationAttribute : ActionFilterAttribute
{
public string BasicRealm { get; set; }
protected string Username { get; set; }
protected string Password { get; set; }
public RequireBasicAuthenticationAttribute()
{
this.Username = System.Configuration.ConfigurationManager.AppSettings["ProtectedUsername"];
this.Password = System.Configuration.ConfigurationManager.AppSettings["ProtectedPassword"];
}
public override void OnActionExecuting(ActionExecutingContext filterContext)
{
var req = filterContext.HttpContext.Request;
var auth = req.Headers["Authorization"];
auth.LogText();
if (!string.IsNullOrEmpty(auth))
{
var cred = System.Text.ASCIIEncoding.ASCII.GetString(Convert.FromBase64String(auth.Substring(6))).Split(':');
var user = new { Name = cred[0], Pass = cred[1] };
if (Username.Equals(user.Name, StringComparison.InvariantCultureIgnoreCase) && Password.Equals(user.Pass)) return;
}
var res = filterContext.HttpContext.Response;
res.StatusCode = 401;
res.AddHeader("WWW-Authenticate", String.Format("Basic realm=\"{0}\"", BasicRealm ?? "bimeh-takmili"));
res.End();
}
}

Just looking at your code I don't see how it runs at all, production or otherwise.
I would suggest it's throwing an error that your code is swallowing since the code below closes the response and then tries to call the base method.
public override void ExecuteResult(ControllerContext context)
{
if (context == null) throw new ArgumentNullException("context");
// this is really the key to bringing up the basic authentication login prompt.
// this header is what tells the client we need basic authentication
var res = context.HttpContext.Response;
res.StatusCode = 401;
res.AddHeader("WWW-Authenticate", "Basic");
res.End();
base.ExecuteResult(context);
}
You cant do this, the code will throw an error:
Server cannot set status after HTTP headers have been sent.
And since it's throwing an error (i think) and being bounced around, it might not be output a 401 status response. The "WWW-Authenticate" header is still being sent however which is why your getting a dialog.
The credentials dialog is popped a when "WWW-Authenticate" is detected but it will only send back an Authorization header in the request if it received a 401 status from the last response.
So if you drop:
base.ExecuteResult(context);
from your code, what happens?
Edit:
Actually dropping
res.End();
would be the way to go. Duh.