I have a below code which reads all the S3 files from a particular S3 bucket and then it adds all those files in a objectList variable.
private static async Task<List<String>> ListObjects()
{
var objectList = new List<String>();
var firstRequest = new ListObjectsV2Request()
{
BucketName = _firstS3BucketName,
MaxKeys = _configurationStateManager.GetState().MaxReferenceDataKeys
};
ListObjectsV2Response getListObjectResponse;
do
{
getListObjectResponse = await GetListObjectResponse(firstRequest);
if (getListObjectResponse != null && getListObjectResponse.S3Objects != null && getListObjectResponse.S3Objects.Count > 0)
{
foreach (var s3Object in getListObjectResponse.S3Objects)
{
// here s3Object.Key has the file name as it's prefix
objectList.Add(s3Object.Key);
}
}
firstRequest.ContinuationToken = getListObjectResponse.NextContinuationToken;
}
while (getListObjectResponse.IsTruncated);
if (getListObjectResponse == null || getListObjectResponse.S3Objects == null ||
getListObjectResponse.S3Objects.Count == 0)
{
return null;
}
return objectList;
}
private static Task<ListObjectsV2Response> GetListObjectResponse(ListObjectsV2Request _request)
{
return _adapter.ReferenceDataAdapter.ListObjectsV2Async(_request);
}
s3Object.Key has the file name in it. Format for the file name is like this - SomeFileName~MM-DD-YYYY.txt
Problem Statement
I am trying to modify the above code in such a way so that it can read some files from one S3 bucket and few other files from second S3 bucket but add all those files in the same objectList variable. Here is what I need to do -
I don't want to read CustomerOutput~MM-DD-YYYY.txt and AccountManager~MM-DD-YYYY.txt from first S3 bucket but read all other files.
But I only want to read CustomerOutput~MM-DD-YYYY.txt and AccountManager~MM-DD-YYYY.txt from second S3 bucket and avoid all other files.
What is the best and generic way to do above problem? I came up with below code which does the job but it isn't the nice way to do it as I am repeating lot of things which can be made generic.
private static async Task<List<String>> ListObjects()
{
var objectList = new List<String>();
var firstRequest = new ListObjectsV2Request()
{
BucketName = _firstS3BucketName,
MaxKeys = _configurationStateManager.GetState().MaxReferenceDataKeys
};
var secondRequest = new ListObjectsV2Request()
{
BucketName = _secondS3BucketName,
MaxKeys = _configurationStateManager.GetState().MaxReferenceDataKeys
};
ListObjectsV2Response firstResponse;
ListObjectsV2Response secondResponse;
do
{
firstResponse = await GetListObjectResponse(firstRequest);
secondResponse = await GetListObjectResponse(secondRequest);
if (firstResponse != null && firstResponse.S3Objects != null && firstResponse.S3Objects.Count > 0
&& secondResponse != null && secondResponse.S3Objects != null && secondResponse.S3Objects.Count > 0)
{
foreach (var s3Object in firstResponse.S3Objects)
{
var objectDelimiterIndex = s3Object.Key.IndexOf("~");
if (objectDelimiterIndex < 0) continue;
// do not read CustomerOutput and AccountManager from first s3 bucket but read all other files
if (!s3Object.Key.Substring(0, objectDelimiterIndex).Equals(ReferenceFiles.CustomerOutput.ToString(), StringComparison.OrdinalIgnoreCase)
|| !s3Object.Key.Substring(0, objectDelimiterIndex).Equals(ReferenceFiles.AccountManager.ToString(), StringComparison.OrdinalIgnoreCase))
{
objectList.Add(s3Object.Key);
}
}
foreach (var s3Object in secondResponse.S3Objects)
{
var objectDelimiterIndex = s3Object.Key.IndexOf("~");
if (objectDelimiterIndex < 0) continue;
// Only read CustomerOutput and AccountManager from second s3 bucket and avoid all other files
if (s3Object.Key.Substring(0, objectDelimiterIndex).Equals(ReferenceFiles.CustomerOutput.ToString(), StringComparison.OrdinalIgnoreCase)
|| s3Object.Key.Substring(0, objectDelimiterIndex).Equals(ReferenceFiles.AccountManager.ToString(), StringComparison.OrdinalIgnoreCase))
{
objectList.Add(s3Object.Key);
}
}
}
firstRequest.ContinuationToken = firstResponse.NextContinuationToken;
secondRequest.ContinuationToken = secondResponse.NextContinuationToken;
}
while (firstResponse.IsTruncated && secondResponse.IsTruncated);
if (firstResponse == null || firstResponse.S3Objects == null ||
firstResponse.S3Objects.Count == 0 || secondResponse == null || secondResponse.S3Objects == null ||
secondResponse.S3Objects.Count == 0)
{
return null;
}
return objectList;
}
Related
We have a recursive function we are using to bulk insert records into Cosmos.
While running a long migration to insert many records it appears that some records failed to insert but we have been unable to see why from the logging we have.
My assumption is a certain type of status code is not being retried when it should be or something is failing silently.
Can anyone advise what might be causing records to fail without being logged/being retried?
AllowBulkExecution = true is set when creating the container that is passed in to the code below.
public async Task<bool> TryBulkInsertAsync<T>(List<T> audits, int retryAttempts, TimeSpan retryDelay, int currentAttempt = 0, Container container = null) where T : BaseCosmosModel
{
if (currentAttempt > retryAttempts)
{
_logger.LogError($"Failed number of max retries ${retryAttempts}");
return false;
}
if (container == null)
{
container = _cosmosContainerFactory.BuildCosmosContainer<T>();
}
var attemptAudits = audits.Select(a => new CosmosInsertAttempt<T>
{
Audit = a,
RunningTask = container.CreateItemAsync(a)
}).ToList();
try
{
await Task.WhenAll(attemptAudits.Select(a => a.RunningTask));
var failedAudits = attemptAudits.Where(a =>
a.RunningTask.Result.StatusCode == HttpStatusCode.TooManyRequests ||
a.RunningTask.Result.StatusCode == HttpStatusCode.PreconditionFailed ||
a.RunningTask.Result.StatusCode == HttpStatusCode.RequestTimeout ||
a.RunningTask.Result.StatusCode == HttpStatusCode.ServiceUnavailable
).Select(a => a.Audit).ToList();
var nonRetryableAudits = attemptAudits.Where(a =>
!failedAudits.Contains(a.Audit) && (a.RunningTask.Result.StatusCode < (HttpStatusCode) 200 ||
a.RunningTask.Result.StatusCode > (HttpStatusCode) 299));
foreach (var audit in nonRetryableAudits)
{
_logger.LogError("Audit failed to bulk insert with non-retryable status code {cosmosAuditInsert}", audit.Audit);
}
if (failedAudits.Count > 0)
{
_logger.LogError("Retrying bulk insert from incorrect status code but no error, count: {retrySize}", failedAudits.Count);
return await TryBulkInsertAsync(failedAudits, retryAttempts, retryDelay, currentAttempt + 1, container);
}
return true;
}
catch (Exception)
{
await Task.Delay(retryDelay);
var failedAuditsWithException = attemptAudits
.Where(a => a.RunningTask.Exception != null)
.Select(a => a.Audit).ToList();
var failedAuditsWithBadStatusCode = attemptAudits.Where(a =>
(
a.RunningTask.Exception == null && a.RunningTask.Result != null &&
(a.RunningTask.Result.StatusCode == HttpStatusCode.TooManyRequests ||
a.RunningTask.Result.StatusCode == HttpStatusCode.PreconditionFailed ||
a.RunningTask.Result.StatusCode == HttpStatusCode.RequestTimeout ||
a.RunningTask.Result.StatusCode == HttpStatusCode.ServiceUnavailable)
)).Select(a => a.Audit).ToList();
if (failedAuditsWithBadStatusCode.Any())
{
_logger.LogError("Retrying bulk insert from incorrect status code but no error (some have exceptions), count: {retrySize}", failedAuditsWithBadStatusCode.Count);
}
var failedAudits = failedAuditsWithException.Concat(failedAuditsWithBadStatusCode).ToList();
var nonRetryableAudits = attemptAudits.Where(a =>
!failedAudits.Contains(a.Audit) && a.RunningTask.Exception == null && (a.RunningTask.Result.StatusCode < (HttpStatusCode)200 ||
a.RunningTask.Result.StatusCode > (HttpStatusCode)299));
foreach (var audit in nonRetryableAudits)
{
_logger.LogError("Audit failed to bulk insert with non-retryable status code {cosmosAuditInsert}", audit.Audit);
}
if (failedAudits.Count > 0)
{
_logger.LogError("Retrying bulk insert {failedAuditCount}", failedAudits.Count);
return await TryBulkInsertAsync(failedAudits, retryAttempts, retryDelay, currentAttempt + 1, container);
}
return true;
}
}
I'm currently trying to get my method to return an error message if a condition isn't valid. But I am uncertain how to go about this in a void method.
I have a method that looks like this
[HttpPost]
[ValidateAntiForgeryToken]
[Audit]
public void AddUnits(int so_id, int site_id, int[] addItem_id, int[] addItem_qty, int[] addItem_disc)
{
// Loop however many times is necessary to iterate through the largest array
for (int i = 0; i < Math.Max(Math.Max(addItem_id.Length, addComp_id.Length), addPart_id.Length); i++)
{
foreach (SODetails sod in db.SalesOrders.Find(so_id).SalesDetails)
{
if (i < addItem_id.Length && addItem_qty[i] != 0 && sod.ItemID == addItem_id[i] && addItem_id[i] != 365 && addItem_id[i] != 410)
{
sod.item_qty += addItem_qty[i];
sod.item_discount = addItem_disc[i];
addItem_id[i] = 0;
addItem_qty[i] = 0;
addItem_disc[i] = 0;
}
}
db.SaveChanges();
if(i < addItem_qty.Length && addItem_qty[i] != 0)
{
SODetails sODetails = new SODetails
{
SalesOrderID = so_id,
SiteID = site_id
};
// Only add a unit to the SODetails object if it's not null and has an id and quanitity specified
if(i < addItem_id.Length && addItem_id[i] != 0 && addItem_qty[i] != 0)
{
sODetails.ItemID = addItem_id[i];
sODetails.item_qty = addItem_qty[i];
sODetails.item_discount = addItem_disc[i];
}
if (sODetails.SiteID == 0)
sODetails.SiteID = null;
SalesOrder SO = db.SalesOrders.Find(sODetails.SalesOrderID);
SODetails salesOrderDetails = db.SODetails.Add(sODetails);
salesOrderDetails.SalesOrder = SO;
Item SO_Item = db.Items.Find(sODetails.ItemID);
Component SO_Component = db.Components.Find(sODetails.ComponentID);
Part SO_Part = db.Parts.Find(sODetails.PartID);
if (SO_Item != null)
{
if (SO.OrderType == SOType.OffSiteInventory && sODetails.InventorySite == "Main Inventory" && SO_Item.On_Hand < salesOrderDetails.item_qty)
{
ModelState.AddModelError(string.Empty, "Not enough stock in inventory");
//TempData["SalesOrderMessage"] = SO_Item.SalesOrderMessage;
}
sODetails.item_qty = sODetails.item_qty == null ? 0 : sODetails.item_qty;
int qtyOrdered = sODetails.item_qty == null ? 0 : (int)sODetails.item_qty;
salesOrderDetails.dynamicItem_qty = qtyOrdered;
if (SO_Item.SalesOrderMessage != null)
TempData["SalesOrderMessage"] = SO_Item.SalesOrderMessage;
}
db.SODetails.Add(sODetails);
}
}
db.SaveChanges();
}
Here is the part of the code where I am doing the validation check
if (SO.OrderType == SOType.OffSiteInventory && sODetails.InventorySite == "Main Inventory" && SO_Item.On_Hand < salesOrderDetails.item_qty)
{
ModelState.AddModelError(string.Empty, "Not enough stock in inventory");
//TempData["SalesOrderMessage"] = SO_Item.SalesOrderMessage;
}
If the condition is valid I want it to return to the screen with an error message showing up. But with the method being Void, I don't know how I can make it do this, or I don't know if it is even possible.
You could create a specific exception class that you can throw in your void function. You then handle this exception in the calling function.
class NotEnoughStockException : Exception {}
Then in your method:
If no stock ...
throw new NotEnoughStockException();
In the calling method
try {
call the stock method
} catch NotEnoughStockException {
whatever you want to do
}
Create a wrapper function that will call your stock function. You do try catch in that new function and return an error message. Your Ajax should call the new function.
I'm swapping my values in List Object on some conditions and update List Object value.
Currently, what I'm doing is
- Looping on each object through List
- Check If condition is net
- Swap values
public static void SwapMinMaxIfNull<T>(this IEnumerable<T> rows, string reportfor)
{
if (reportfor.Equals("Comparison"))
{
var row = rows as IEnumerable<RiskBoardDataToExport>;
try
{
if (rows.Any())
{
var Tests = row.Where(min => min.MinGaitSpeed == null && min.MaxGaitSpeed != null).ToList();
if (Tests != null)
{
foreach (RiskBoardDataToExport test in Tests)
{
test.MinGaitSpeed = test.MaxGaitSpeed;
test.MaxGaitSpeed = null;
}
}
// again check for next object
Tests = row.Where(min => min.MinTUGTime == null && min.MaxTUGTime != null).ToList();
if (Tests != null)
{
foreach (RiskBoardDataToExport test in Tests)
{
test.MinTUGTime = test.MaxTUGTime;
test.MaxTUGTime = null;
}
}
// again check for next object
Tests = row.Where(min => min.MinBergScoreSpeed == null && min.MaxBergScoreSpeed != null).ToList();
if (Tests != null)
{
foreach (RiskBoardDataToExport test in Tests)
{
test.MinBergScoreSpeed = test.MaxBergScoreSpeed;
test.MaxBergScoreSpeed = null;
}
}
//.. for brevity
}
}
}
Can I do it in better way? I know about PropertyInfo i.e. Can check property name and get value etc, but, not having any hint to get this done.
Thanks
It's not exactly what you're asking for, but you can combine the clauses in your Where statements and then have a few if statements in the body:
public static void SwapMinMaxIfNull(this IEnumerable<RiskBoardDataToExport> rows,
string reportfor)
{
if (rows = null) return;
if (reportfor.Equals("Comparison", StringComparison.OrdinalIgnoreCase))
{
foreach (var row in rows.Where(r =>
(r.MinGaitSpeed == null && r.MaxGaitSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null)))
{
if (row.MinGaitSpeed == null)
{
row.MinGaitSpeed = row.MaxGaitSpeed;
row.MaxGaitSpeed = null;
}
if (row.MinTUGTime == null)
{
row.MinTUGTime = row.MaxTUGTime;
row.MaxTUGTime = null;
}
if (row.MinBergScoreSpeed == null)
{
row.MinBergScoreSpeed = row.MaxBergScoreSpeed;
row.MaxBergScoreSpeed = null;
}
}
}
}
As this is an operation where order of the items in the list does not matter, you can easily speed this up by parallelization (you can read up on that here).
So, what you should do, is handle this foreach loop in a parallel way and combine it with Rufus L's optimized code for the fastest result.
var rows = rows.Where(r =>
(r.MinGaitSpeed == null && r.MaxGaitSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null))
Parallel.ForEach(rows, (row) => {
{
if (row.MinGaitSpeed == null)
{
row.MinGaitSpeed = row.MaxGaitSpeed;
row.MaxGaitSpeed = null;
}
if (row.MinTUGTime == null)
{
row.MinTUGTime = row.MaxTUGTime;
row.MaxTUGTime = null;
}
if (row.MinBergScoreSpeed == null)
{
row.MinBergScoreSpeed = row.MaxBergScoreSpeed;
row.MaxBergScoreSpeed = null;
}
}
Note that this requires the System.Threading.Tasks namespace, that's where the Parallel class is.
i started with this! and i would like to avoid this
private IQueryable<Customer> FilterResult(string search, List<Customer> dtResult, List<string> columnFilters)
{
IQueryable<Customer> results = dtResult.AsQueryable();
results = results.Where(p => (search == null || (p.Name != null && p.Name.ToLower().Contains(search.ToLower()) || p.City != null && p.City.ToLower().Contains(search.ToLower())
&& (columnFilters[0] == null || (p.Name != null && p.Name.ToLower().Contains(columnFilters[0].ToLower())))
&& (columnFilters[1] == null || (p.City != null && p.City.ToLower().Contains(columnFilters[1].ToLower())))
);
return results;
}
by using reflection maybe.... because imagine i have 100 properties it would be easy to mess up ...so i tried this way i would like to use reflection to loop on all properties instead of making references to each of them(properties)
public IQueryable<Aangifte> FilterColumn<T>()
{
try
{
List<Aangifte> v = AangifteGetData.GetData(StartDate, EndDate);
List<Aangifte> v2 = new List<Aangifte>();
Aangifte Foo = new Aangifte();
List<string> propertEntity = new List<string>();
var search = Request.Form.GetValues("search[value]").FirstOrDefault();
int count = -1;
var results = v.AsQueryable();
List<string> columnName = new List<string>();
foreach (var prop in Foo.GetType().GetProperties())
{
var t = "";
if (!prop.Name.Contains("FORMAT"))
{
TVALUE = prop.GetValue(Foo, null);
t= prop.Name;
propertEntity.Add(t);
count++;
}
if (count < propertEntity.Count())
{
var newt = t;
var Z = Request.Form.GetValues("columns[" + count + "][search][value]").FirstOrDefault();
results = results.Where
(
p => (search == null || (t != null && t.ToLower().Contains(search.ToLower())))
&& (Request.Form.GetValues("columns[" + count + "][search][value]").FirstOrDefault() == null || (t != null && t.ToLower().Contains(Request.Form.GetValues("columns[" + count + "][search][value]").FirstOrDefault().ToLower())))
);
}
}
return results;
}
catch (Exception EX)
{
throw EX;
}
}
I am using the NEST (c#) client to communicate with Elasticsearch. I am using the suggest API to implement autocomplete in my site, here is my query code sample
var completionResult = client.Search(
body => body
.Type("myProject")
.Size(5)
.Fields("title","tags","control","platform")
.SuggestGlobalText(searchText)
.SuggestCompletion("SuggestCompletionResult", sfc => sfc
.OnField("control.suggestControl"))
.SuggestPhrase("SuggestPhraseResult", sfp => sfp
.OnField("control.suggestControl"))
);
i need to get final result as List<SuggestionList> and my suggestion list class is
public class SuggestionList
{
public string Text { get; set; }
}
the operation i performed to convert the result into List<SuggestionList> is
List<SuggestionList> SuggestResultSet = new List<SuggestionList>();
if (completionResult != null)
{
if (completionResult.Suggest != null && completionResult.Suggest.Count > 0)
{
var suggestionList = completionResult.Suggest.ToList();
if (suggestionList != null && suggestionList.Count > 0)
{
if (suggestionList[0].Value.ToList().Count() != 0 && suggestionList[0].Value.ToList()[0].Options.ToList().Count > 0)
{
foreach (var text in suggestionList[0].Value.ToList()[0].Options.ToList())
{
SuggestResultSet.Add(new SuggestionList { Text = text.Text });
}
}
if (suggestionList[1].Value.ToList().Count() != 0 && suggestionList[1].Value.ToList()[0].Options.ToList().Count > 0)
{
foreach (var text in suggestionList[1].Value.ToList()[0].Options.ToList())
{
SuggestResultSet.Add(new SuggestionList { Text = text.Text });
}
}
}
}
}
it take longer time to get final result, is there any way to do it without affecting performance
I think you have over complicated your conversion process, nesting is high. Bearing in mind this isn't compiled and is only pseudo.
List<SuggestionList> SuggestResultSet = new List<SuggestionList>();
if(completionResults != null)
{
foreach(var suggestion in completionResults.Suggest)
{
SuggestsResultSet.Add(new SuggestionList {Text = suggestion.Text });
}
}