I'm trying to list all items under a path for a determined bucket using Google Cloud Storage APIs. Under this path, there are more than 1000 items - which is the maximum number of items that ListObjects / ListObjectsAsync return.
In order to be able to repeat the call and get the next 1000 items, I need the NextPageToken from the previous response, that's how result pagination gets done! It's easy and reasonable.
However, I never get a NextPageToken within the responses. Am I missing something? This is my code so far. Some key points:
Althought there are more than +1000 items sharing the same prefix, I only get one response while looping through them in the enumerator.
The NextPageToken in the Console.WriteLine statement is always null.
Repeating the same call returns the same first 1000 objects over and over.
async IAsyncEnumerable<string> ListObjectsAsync(
string prefix, [EnumeratorCancellation] CancellationToken cancelToken)
{
var listObjectsOptions = new ListObjectsOptions
{
Fields = "items(name)"
};
var rawResponses = mGoogleClient.ListObjectsAsync(
mBucketName,
prefix,
listObjectsOptions).AsRawResponses();
using (var enumerator = rawResponses.GetEnumerator())
{
while (await enumerator.MoveNext(cancelToken))
{
foreach (var googleObject in enumerator.Current.Items)
yield return googleObject.Name;
}
Console.WriteLine(enumerator.Current.NextPageToken);
}
}
Thank you.
Apparently, I'm missing the nextPageToken field in the ListObjectsOptions used to make the request. Without specifying that field, the service won't return it - because of who knows why!
This code should work:
async IAsyncEnumerable<string> ListObjectsAsync(
string prefix, [EnumeratorCancellation] CancellationToken cancelToken)
{
var listObjectsOptions = new ListObjectsOptions
{
Fields = "items(name),nextPageToken"
};
var rawResponses = mGoogleClient.ListObjectsAsync(
mBucketName,
prefix,
listObjectsOptions).AsRawResponses();
using (var enumerator = rawResponses.GetEnumerator())
{
while (await enumerator.MoveNext(cancelToken))
{
foreach (var googleObject in enumerator.Current.Items)
yield return googleObject.Name;
}
Console.WriteLine(enumerator.Current.NextPageToken);
}
}
The nice thing is that you don't even have to use the NextPageToken explicitly. This is, you don't have to do something like this:
string token = null;
do
{
var options = new ListObjectsOptions
{
Fields = "items(name),nextPageToken",
PageToen = token
};
var rawResponses = mGoogleClient.ListObjectsAsync(
mBucketName,
prefix,
listObjectsOptions).AsRawResponses();
using (var enumerator = rawResponses.GetEnumerator())
{
while (await enumerator.MoveNext(cancelToken))
{
foreach (var googleObject in enumerator.Current.Items)
yield return googleObject.Name;
}
token = enumerator.Current.NextPageToken;
}
}
while (!string.IsNullOrEmpty(token));
...because the enumerator you got in rawResponses.GetEnumerator() will take care of using the token of a response to automatically fetch the next (if needed) while iterating them.
So the first piece of code is valid to iterate over +1000 objects in a single call.
Related
Im trying to get an item from azure cosmos db in azure functions .net 5.0
but im getting this error because im using async method
Feedlterator <rsakeys> Container.GetitemQuerylterator <rsakeys> (QueryDefinition queryDefinition, [string continuation Token = null], [QueryRequestOptions requestOptions = null]) (+ 1 overload) This method creates a query for items under a container in an Azure Cosmos database using a SQL statement with parameterized values. It returns a Feedlterator. For more information on preparing SQL statements with parameterized values, please see QueryDefinition.
Returns:
An iterator to go through the items. CS8411: Asynchronous foreach statement cannot operate on variables of type 'Feediterator <rsakeys>' because 'Feedlterator <rsakeys>' does not contain a suitable public instance or extension definition for 'GetAsyncEnumerator
CS8411: Asynchronous foreach statement cannot operate on variables of type 'Feediterator <rsakeys>' because 'Feedlterator <rsakeys>' does not contain a suitable public instance or extension definition for 'GetAsyncEnumerator
This is my code, i have not found any good documentation on this error at all, im pretty much lost.
namespace projectA
{
class projectA
{
[Function("projectA")]
async Task<HttpResponseData> Run([HttpTrigger(AuthorizationLevel.Anonymous, "")] HttpRequestData req,
FunctionContext context)
{
var logger = context.GetLogger("projectA");
logger.LogInformation("C# HTTP trigger function processed a request.");
response.Headers.Add("Content-Type", "text/plain; charset=utf-8");
/// The Azure Cosmos DB endpoint.
string EndpointUrl = "https://name.documents.azure.com:443/";
/// The primary key for the Azure DocumentDB account.
string PrimaryKey = "primarykey";
// The Cosmos client instance
CosmosClient cosmosClient = new CosmosClient(EndpointUrl, PrimaryKey);
Database database;
Container container;
string databaseId = "database1";
string containerId = "container1";
var sqlQueryText = "SELECT * FROM c WHERE c.key = '123'";
Console.WriteLine("Running query: {0}\n", sqlQueryText);
container = cosmosClient.GetContainer(databaseId, containerId);
QueryDefinition queryDefinition = new QueryDefinition(sqlQueryText);
List<rsakeys> rsakeyss = new List<rsakeys>();
//In this await foreach i get the error...
await foreach (rsakeys RsaKeys in container.GetItemQueryIterator<rsakeys>(queryDefinition))
{
rsakeyss.Add(RsaKeys);
Console.WriteLine("\tRead {0}\n", RsaKeys);
}
}
}
}
The pattern you would use for a Feed Iterator would be
while (feedIterator.HasMoreResults)
foreach (var item in await feedIterator.ReadNextAsync())
Console.Write(item.Cost);
If you wanted to use an awaited foreach (AKA IAsynEnumerable), you could roll your own IAsynEnumerable to yield
private static async IAsyncEnumerable<T> SomeMethod<T>(
Container container,
QueryDefinition queryDefinition,
[EnumeratorCancellation] CancellationToken cancellationToken = default)
{
using var feedIterator = container.GetItemQueryIterator<T>(queryDefinition);
while (feedIterator.HasMoreResults)
foreach (var item in await feedIterator
.ReadNextAsync(cancellationToken)
.ConfigureAwait(false))
yield return item;
}
Usage
await foreach(var item in SomeMethod<SomeType>(container, queryDefinition)
Console.Write(item);
Bonus fact
It's also worth noting that ReadNextAsync returns Task<FeedResponse<T>>
public abstract Task<FeedResponse<T>> ReadNextAsync(CancellationToken cancellationToken = default);
FeedResponse is inherited from Response
FeedResponse<T> : Response<IEnumerable<T>>, IEnumerable<T>
However, for convenience Response has an implicit operator, so it can be implicitly converted to its generic Enumerbale type IEnumerable<T>
public static implicit operator T(Response<T> response)
Ergo, both of the following statements are valid
using var feedIterator = container.GetItemQueryIterator<Bob>(queryDefinition);
FeedResponse<Bob> bobResponse = await feedIterator.ReadNextAsync()
IEnumerable<Bob> bobs = await feedIterator.ReadNextAsync()
Update From comments
After doing using
var feedIterator = container.GetItemQueryIterator<rsakeys>(queryDefinition); var x =
await feedIterator.ReadNextAsync();
Console.WriteLine(x.GetType().ToString());
I get this console output :
Microsoft.Azure.Cosmos.QueryResponse`1[PolarisRSAFunction1NET5.rsakeys]
how do i access the query value?
Calling ReadNextAsync would return an FeedResponse<rsakeys> from there you can just call x.Resource to return an IEnumerable<rsakeys> or just directly assign it
I.e
var response = await feedIterator.ReadNextAsync();
IEnumerable<rsakeys> items = response.Resource;
I have this function that can get up to 10 items as an input list
public async Task<KeyValuePair<string, bool>[]> PayCallSendSMS(List<SmsRequest> ListSms)
{
List<Task<KeyValuePair<string, bool>>> tasks = new List<Task<KeyValuePair<string, bool>>>();
foreach (SmsRequest sms in ListSms)
{
tasks.Add(Task.Run(() => SendSMS(sms)));
}
var result = await Task.WhenAll(tasks);
return result;
}
and in this function, i await for some JSON to be downloaded and after it's done in deserialize it.
public async Task<KeyValuePair<string, bool>> SendSMS(SmsRequest sms)
{
//some code
using (WebResponse response = webRequest.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
StreamReader rdr = new StreamReader(responseStream, Encoding.UTF8);
string Json = await rdr.ReadToEndAsync();
deserializedJsonDictionary = (Dictionary<string, object>)jsonSerializer.DeserializeObject(Json);
}
}
//some code
return GetResult(sms.recipient);
}
public KeyValuePair<string, bool> GetResult(string recipient)
{
if (deserializedJsonDictionary[STATUS].ToString().ToLower().Equals("true"))
{
return new KeyValuePair<string, bool>(recipient, true);
}
else // deserializedJsonDictionary[STATUS] == "False"
{
return new KeyValuePair<string, bool>(recipient, false);
}
}
My problem is in the return GetResult(); part in which deserializedJsonDictionary is null(and ofc it is becuase the json havent done downloading).
but I don't know how to solve it
I tried to use ContinueWith but it doesn't work for me.
I'm willing to accept any change to my original code and/or the design of the solution
Unrelated tip: Don't abuse KeyValuePair<>, use C# 7 value-tuples instead (not least because they're much easier to read).
Using a foreach loop to build a List<Task> is fine - though it can be more succint to use .Select() instead. I use this approach in my answer.
But don't use Task.Run with the ancient WebRequest (HttpWebRequest) type. Instead use HttpClient which has full support for async IO.
Also, you should conform to the .NET naming-convention:
All methods that are async should have Async has a method-name suffix (e.g. PayCallSendSMS should be named PayCallSendSmsAsync).
Acronyms and initialisms longer than 2 characters should be in PascalCase, not CAPS, so use Sms instead of SMS.
Use camelCase, not PascalCase for parameters and locals - and List is a redundant prefix. A better name for ListSms would be smsRequests as its type is List<SmsRequest>).
Generally speaking, parameters should be declared using the least-specific type required - especially collection parameters, consider typing them as IEnumerable<T> or IReadOnlyCollection<T> instead of T[], List<T>, and so on).
You need to first check that the response from the remote server actually is a JSON response (instead of a HTML error message or XML response) and has the expected status code - otherwise you'll be trying to deserialize something that is not JSON.
Consider supporting CancellationToken too (this is not included in my answer as it adds too much visual noise).
Always use Dictionary.TryGetValue instead of blindly assuming the dictionary indexer will match.
public async Task< IReadOnlyList<(String recipient, Boolean ok)> > PayCallSendSmsAsync( IEnumerable<SmsRequest> smsRequests )
{
using( HttpClient httpClient = this.httpClientFactory.Create() )
{
var tasks = smsRequests
.Select(r => SendSmsAsync(httpClient, r))
.ToList(); // <-- The call to ToList is important as it materializes the list and triggers all of the Tasks.
(String recipient, Boolean ok)[] results = await Task.WhenAll(tasks);
return results;
}
}
private static async Task<(String recipient, Boolean ok)> SendSmsAsync(HttpClient httpClient, SmsRequest smsRequest)
{
using (HttpRequestMessage request = new HttpRequestMessage( ... ) )
using (HttpResponseMessage response = await httpClient.SendAsync(request).ConfigureAwait(false))
{
String responseType = response.Content.Headers.ContentType?.MediaType ?? "";
if (responseType != "application/json" || response.StatusCode != HttpStatusCode.OK)
{
throw new InvalidOperationException("Expected HTTP 200 JSON response but encountered an HTTP " + response.StatusCode + " " + responseType + " response instead." );
}
String jsonText = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
Dictionary<String,Object> dict = JsonConvert.DeserializeObject< Dictionary<String,Object> >(jsonText);
if(
dict != null &&
dict.TryGetValue(STATUS, out Object statusValue) &&
statusValue is String statusStr &&
"true".Equals( statusStr, StringComparison.OrdinalIgnoreCase )
)
{
return ( smsRequest.Recipient, ok: true );
}
else
{
return ( smsRequest.Recipient, ok: false );
}
}
}
My Code
public async Task Invoke(HttpContext httpContext, ISecretKeyModel SecretKeyModel)
{
if (httpContext.Request.Path.StartsWithSegments("/api"))
{
if (httpContext.Request.Headers.TryGetValue("SecretKey", out StringValues SecretKey))
{
SecretKeyModel.SecretKey = SecretKey.SingleOrDefault();
if (SecretKeyModel.SecretKey.Equals(_config.Value.ClientSecretKey)) SecretKeyModel.IsValid = true;
}
}
await _next(httpContext);
}
In the above code, I have checked whether the path string segment start with "/api" or not.
But, I want to check more than one or group of such paths, I can do like this
if(httpContext.Request.Path.StartsWithSegments("/api") || httpContext.Request.Path.StartsWithSegments("/app"))
But i want another methods to do this
You could put all values that you need to check in a list or array then use linq Any() method:
var list = new List<string> { "/api", "/app" };
if (list.Any(s => httpContext.Request.Path.StartsWithSegments(s)))
{
...
}
I've been working for a few days on a performance problem.
Before I delve deeper I want to quickly explain how the specific service work.
I have a main Service that get a request and send requests to other micro-services but the single Entry Point for the user is to Main service, I thinks is more simple to understand with this image:
After the Main service get request from API he do some logic, query the Db and then get a list, every item on the list has Id, to get enrichment about every item the main service create request to one of the micro-service.
For example John request main service, main service get from Db a list of 90 items then the main service will create 90 calls to micro service and return to John single response that include 90 items.
Now the question is only about the right way to create async call to micro service.
This how I develop this part:
GetDetailsAsync(Id, result.Items, request.SystemComponentId);
private static void GetDetailsAsync(string Id, List<MainItem> items, int systemId)
{
var getDetailsTasks = new List<Task>();
foreach (MainItem single in items)
{
getDetailsTasks.Add(SetSingleDetailsAsync(Id, single, systemId));
}
Task.WhenAll(getDetailsTasks);
}
private static async Task SetSingleDetailsAsync(string Id, MainItem single, int systemId)
{
single.ActivityExtendedDetails = await ProcessItemDetailsRequest.GetItemDetailsAsync(Id, single.TypeId,
single.ItemId, systemId);
}
public static Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return Task.FromResult(res);
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
This code run and the result is not good enough, the CPU rises very quickly and becomes very high, I think that I has a problem on GetItemDetailsAsync func because I use client.GetStreamAsync(url).Result
when using .Result it's block until the task is completed.
So I do some minor change on GetItemDetailsAsync to try to be really async:
public static async Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = await client.GetStreamAsync(url))
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return res;
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
But now I get null where I supposed to get the data that come from Async function.
I try to debugging and I noticed something weird, everything happen likes as I would expect: the methods was called, request to micro-service was executed and get response but the response from the End-Point(which is found on main-service) return before the async method return from micro-service, that cause that I get null instead of my expected data.
I thinks that maybe I don't use correctly async\await and would be happy if anyone could explain how this behavior happens
I am using the QueryOperationResponse GetContinuation() method to attempt and page through a very large data set read off an odata wcf feed. I am using some OData helper libraries acquired from microsoft.
The problem is that the paging process seems to get stuck in an infinite loop, and never finishes. For example if I set the page size to 10000 records and there are 80000 records to retrieve, I observe that the loop continues for well over 8 iterations, when it should be finished.
Below is the class that queries the service and implements the paging (at the bottom). I also observed that the 'OriginalString' of the NextLinkUri never changes with each iteration, which I think is wrong? Hopefully I'm just missing something really obvious, I think this is the correct way to page:
private static IList<dynamic> Get(string serviceUri, NameValueCollection queryOptions, IAuthenticationScheme authenticationScheme)
{
string baseUri;
string entitySet;
string entityKey;
string queryString;
ValidateServiceUri(serviceUri, out baseUri, out entitySet, out entityKey, out queryString);
string resource = !string.IsNullOrEmpty(entityKey) ? entitySet + "(" + entityKey + ")" : entitySet;
DataServiceContext context = new DataServiceContext(new Uri(baseUri));
context.IgnoreMissingProperties = true;
DataServiceContextHandler handler = new DataServiceContextHandler(authenticationScheme);
handler.HandleGet(context);
DataServiceQuery<EntryProxyObject> query = context.CreateQuery<EntryProxyObject>(resource);
NameValueCollection options = HttpUtility.ParseQueryString(queryString);
options.Add(queryOptions);
foreach (string key in options.AllKeys)
{
query = query.AddQueryOption(key, options[key]);
}
QueryOperationResponse<EntryProxyObject> response = query.Execute() as QueryOperationResponse<EntryProxyObject>;
IList<dynamic> result;
if (options["$inlinecount"] == "allpages")
{
result = new DynamicEntityCollection(response.TotalCount) as IList<dynamic>;
}
else
{
result = new List<dynamic>();
}
foreach (EntryProxyObject proxy in response)
{
DynamicEntity entity = new DynamicEntity(proxy.Properties);
result.Add(entity);
}
while (response.GetContinuation() != null)
{
DataServiceQueryContinuation<EntryProxyObject> continuation = response.GetContinuation();
QueryOperationResponse<EntryProxyObject> nextResponse = context.Execute<EntryProxyObject>(continuation);
Console.WriteLine("Uri: " + continuation.NextLinkUri.OriginalString);
foreach (EntryProxyObject proxy in nextResponse)
{
DynamicEntity entity = new DynamicEntity(proxy.Properties);
result.Add(entity);
}
}
return result;
}
This is how I call the method:
return OData.Get("http://xxx.x.xxx.xx/MyDataService.svc/MyProducts", "$filter=Index gt 0");
You are always looking at the continuation of the first response. That stays constant (for obvious reasons). You need to look at the continuation of the nextResponse in your code to figure out if there's more data to read and to get the continuation to get the next page. The code above reads the first page and then reads the second page over and over again.