We Keep Coding

C# Octokit Update File if it Exists, Otherwise Create

I'm wondering whether there is a clean way to create a new file using Octokit in case the file specified doesn't exist, or otherwise create it? I'm currently attempting to do this via a try-catch block. However, there needs to be a more straightforward way to do this except a try-catch block?
private IRepositoryContentsClient Content => _client.Repository.Content;
public void TransmitLog(string logFilePath)
{
var logContent = LogFileToString(logFilePath);
var githubPath = GenerateGithubPath(logFilePath);
try
{
UpdateLog(githubPath, logContent);
}
catch (NotFoundException)
{
CreateLog(githubPath, logContent);
}
catch (AggregateException)
{
CreateLog(githubPath, logContent);
}
}
private void UpdateLog(string githubPath, string logContent)
{
// MY APP FAILS HERE
var existingFile = Content.GetAllContentsByRef(
_owner, _repo, githubPath, _branch).Result;
// update the file
var relevantSha = existingFile.First().Sha;
var updateRequest = new UpdateFileRequest("Log update" + DateTime.UtcNow, logContent, relevantSha, _branch);
var updateChangeSet = Content.UpdateFile(_owner, _repo, githubPath, updateRequest);
}
private void CreateLog(string githubPath, string logFileContent)
{
// if file is not found, create it
var createRequest =
new CreateFileRequest("Log Creation" + DateTime.UtcNow, logFileContent, _branch);
var createChangeSet = Content.CreateFile(_owner, _repo, githubPath, createRequest);
}
EDIT: I initially had both CreateLog and UpdateLog declared as async void, which led to a whole set of other problems. I edited that out since what I'm really interested in is how to avoid this try/catch structure.

Correct way to use Async/Await

I've been working for a few days on a performance problem.
Before I delve deeper I want to quickly explain how the specific service work.
I have a main Service that get a request and send requests to other micro-services but the single Entry Point for the user is to Main service, I thinks is more simple to understand with this image:
After the Main service get request from API he do some logic, query the Db and then get a list, every item on the list has Id, to get enrichment about every item the main service create request to one of the micro-service.
For example John request main service, main service get from Db a list of 90 items then the main service will create 90 calls to micro service and return to John single response that include 90 items.
Now the question is only about the right way to create async call to micro service.
This how I develop this part:
GetDetailsAsync(Id, result.Items, request.SystemComponentId);
private static void GetDetailsAsync(string Id, List<MainItem> items, int systemId)
{
var getDetailsTasks = new List<Task>();
foreach (MainItem single in items)
{
getDetailsTasks.Add(SetSingleDetailsAsync(Id, single, systemId));
}
Task.WhenAll(getDetailsTasks);
}
private static async Task SetSingleDetailsAsync(string Id, MainItem single, int systemId)
{
single.ActivityExtendedDetails = await ProcessItemDetailsRequest.GetItemDetailsAsync(Id, single.TypeId,
single.ItemId, systemId);
}
public static Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return Task.FromResult(res);
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
This code run and the result is not good enough, the CPU rises very quickly and becomes very high, I think that I has a problem on GetItemDetailsAsync func because I use client.GetStreamAsync(url).Result
when using .Result it's block until the task is completed.
So I do some minor change on GetItemDetailsAsync to try to be really async:
public static async Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = await client.GetStreamAsync(url))
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return res;
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
But now I get null where I supposed to get the data that come from Async function.
I try to debugging and I noticed something weird, everything happen likes as I would expect: the methods was called, request to micro-service was executed and get response but the response from the End-Point(which is found on main-service) return before the async method return from micro-service, that cause that I get null instead of my expected data.
I thinks that maybe I don't use correctly async\await and would be happy if anyone could explain how this behavior happens

EntityState must be set to null, Created (for Create message) or Changed (for Update message)

In my C# console application I am trying to update an account in CRM 2016. IsFaulted keeps returning true.
The error message it returns when I drill down is the following:
EntityState must be set to null, Created (for Create message) or Changed (for Update message).
Also in case it might cause the fault I have pasted my LINQ query at the bottom.
The answers I get from Google states either that I am mixing ServiceContext and ProxyService (which am not, I am not using it in this context). The others says that I am using context.UpdateObject(object) incorrectly, which I am not using either.
Update: Someone just informed me that the above error is caused because I am trying to return all the metadata and not just the updated data. Still I have no idea how to fix the error, but this information should be helpful.
private static void HandleUpdate(IOrganizationService crmService, List<Entity> updateEntities)
{
Console.WriteLine("Updating Entities: " + updateEntities.Count);
if (updateEntities.Count > 0)
{
try
{
var multipleRequest = new ExecuteMultipleRequest()
{
// Assign settings that define execution behavior: continue on error, return responses.
Settings = new ExecuteMultipleSettings()
{
ContinueOnError = true,
ReturnResponses = true
},
// Create an empty organization request collection.
Requests = new OrganizationRequestCollection()
};
foreach (var account in updateEntities)
{
multipleRequest.Requests.Add(
new UpdateRequest()
{
Target = account
});
}
ExecuteMultipleResponse response = (ExecuteMultipleResponse)crmService.Execute(multipleRequest);
if (response.IsFaulted)
{
int failedToUpdateAccount = 0;
foreach (ExecuteMultipleResponseItem singleResp in response.Responses)
{
if (singleResp.Fault != null)
{
string faultMessage = singleResp.Fault.Message;
var account = ((UpdateRequest)multipleRequest.Requests[singleResp.RequestIndex]).Target;
Log.Error($"Error update acc.id: {account.Id}.Error: {singleResp.Fault.Message}.");
failedToUpdateAccount++;
}
}
Log.Debug($"Failed to update {failedToUpdateAccount} accounts.");
}
else
{
Log.Debug("Execute multiple executed without errors");
}
}
catch (Exception ex)
{
Log.Error($"Error while executing Multiplerequest", ex);
}
}
}
// LINQ below
private static List<Account> GetAllActiveCRMAccounts(CRM2011DataContext CRMcontext)
{
Console.WriteLine("Start Getting CRMExistingAccounts ....");
List<Account> CRMExisterendeAccounts = new List<Account>();
try
{
CRMExisterendeAccounts = (from a in CRMcontext.AccountSet
where a.StateCode == AccountState.Active
where a.anotherVariable == 1
select new Account()
{
my_var1 = a.myVar1,
my_var2 = a.myVar2,
AccountId = a.AccountId,
anotherVar = a.alsoThisVar,
}).ToList();
}
catch (FaultException ex)
{
Log.Debug($"GetCRMExistingAccounts Exception { ex.Message}");
Console.WriteLine("GetCRMExistingAccounts Exception " + ex.Message);
throw new Exception(ex.Message);
}
return CRMExisterendeAccounts;
}
And yes, my variables has different names in my system.
The query returns the object just fine with all the correct data.

You can work around this in one of two ways:
1) Create your CRM2011DataContext with the MergeOption set to MergeOption.NoTracking. Entities loaded from a context that is not tracking will have a null EntityState property.
2) You can create a copy of your Entity and save the copy.

Which one of these async calls is not like the other?

I suspect I have a deadlock issue, but it's an odd one that I can't rationalize. I have an API that needs to verify a few things in order to process the call. As part of the business logic, I might have to make more of those same calls as well. In this case, if a particular piece of data associated with an entity is not found, we attempt to use a backup (if one is configured), which requires checking other entities. Eventually, the code will hang.
Let's just dive into the code (comments highlight the calls in question).
API Controller:
public async Task<HttpResponseMessage> Get(int entityID, string content, bool? useBackUp = true)
{
//Some look-ups here, no issues at all
//This works, but it's this method that has an issue later in the process.
SystemEntity entityObj =
await BusinessLayer.GetSystemEntityAsync(SystemEntityID);
if (entityObj == null)
{
return new HttpResponseMessage
{
StatusCode = System.Net.HttpStatusCode.BadRequest,
Content = new StringContent("Entity is unavailable.")
};
}
string text = BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp).Result.FirstOrDefault().Value;
if (text == null)
{
return new HttpResponseMessage {StatusCode = System.Net.HttpStatusCode.NoContent};
}
return new HttpResponseMessage
{
StatusCode = System.Net.HttpStatusCode.OK,
Content = new StringContent(text)
};
}
Business Layer:
public async Task<Dictionary<string, string>> GetContentTextAsync(int systemEntityID, List<string> contentNames, bool useBackUp)
{
Dictionary<string, string> records = new Dictionary<string, string>();
//We iterate for caching purposes
foreach (string name in contentNames)
{
string nameCopy = name;
string record = Cache.GetData(
string.Format("{0}_{1}_{2}", CONTENT, systemEntityID, name), () =>
DataLayer.GetCotnent(systemEntityID, nameCopy));
if (record == null && useBackUp)
{
List<int> entityIDs = new List<int> {systemEntityID};
int currentEntityID = systemEntityID;
//Here's that method again. This call seems to work.
SystemEntity currentEntity = await GetSystemEntityAsync(systemEntityID);
if (currentEntity != null && currentEntity.BackUpID.HasValue)
{
currentEntityID = (int) currentEntity.BackUpID;
}
while (!entityIDs.Contains(currentEntityID))
{
int id = currentEntityID;
record = Cache.GetData(
string.Format("{0}_{1}_{2}", CONTENT, systemEntityID, name), () =>
DataLayer.GetCotnent(id, nameCopy));
if (record != null) break;
entityIDs.Add(currentEntityID);
//This call seems to cause the deadlock
currentEntity = await GetSystemEntityAsync(currentEntityID);
if (currentEntity != null && currentEntity.BackUpID.HasValue)
{
currentEntityID = (int) currentEntity.UseBackupID;
}
}
}
if (record != null)
{
records.Add(name, record);
}
}
return records;
}
public async Task<SystemEntity> GetSystemEntityAsync(int systemEntityID)
{
SystemEntity systemEntity = await DataLayer.GetSystemEntity(
scc => scc.SystemEntityID == systemEntityID);
return systemEntity;
}
Data Layer:
public async Task<SystemEntity> GetSystemEntity(Expression<Func<SystemEntity, bool>> whereExpression)
{
using (EntityContext dbContext = createDbInstance())
{
//This is the last line that the debugger in VS 2013 brings me to. Stepping into this returns to whatever called the API method, waiting endlessly.
return await
dbContext.SystemEntity.Include(sc => sc.OtherEntity).Where(whereExpression).FirstOrDefaultAsync();
}
}
To recap: I call GetSystemEntityAsync three times. The first two times, it completes successfully. The third time, it hangs. If I comment out the first two calls so they don't run at all, the third one still hangs. If I remove the await and use just a normal FirstOrDefault in the return statement of the data layer method, then everything completes just fine.
Note: I have to keep the GetSystemEntityAsync method asynchronous. I cannot alter it to be synchronous.
What are the possible sources of the deadlock I'm encountering? I'm out of ideas on how to solve it.

Which one of these async calls is not like the other?
This one, I suspect:
string text = BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp).Result.FirstOrDefault().Value;
Try changing it to this:
string text = (await BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp)).FirstOrDefault().Value;
The possible source of the deadlock is described by Stephen Cleary in his "Don't Block on Async Code" blog post.

what's best way to check if a S3 object exists?

Currently, I make a GetObjectMetaDataRequest, if the GetObjectMetaDataResponse throw an exception means the object doesn't exist. Is there a better way to check whether the file exists without downloading the file.

you can use S3FileInfo class and Exists method of this class it will hep you to check if file exists without download the file .see the example below I used the AWSSDK 3.1.6 .net(3.5) :
public static bool ExistsFile()
{
BasicAWSCredentials basicCredentials = new BasicAWSCredentials("my access key", "my secretkey");
AmazonS3Config configurationClient = new AmazonS3Config();
configurationClient.RegionEndpoint = RegionEndpoint.EUCentral1;
try
{
using (AmazonS3Client clientConnection = new AmazonS3Client(basicCredentials, configurationClient))
{
S3FileInfo file = new S3FileInfo(clientConnection, "mybucket", "FolderNameUniTest680/FileNameUnitTest680");
return file.Exists;//if the file exists return true, in other case false
}
}
catch(Exception ex)
{
return false;
}
}

If you are in a similar situation as myself and are using .Net Core and don't have access to Amazon.S3.IO (and S3FileInfo method), you can do the following using asynchronous GetObjectMetadataRequest method:
static private AmazonS3Client s3Client = new AmazonS3Client();
public static async Task<bool> FileExistsS3Async(string _bucket, string _key)
{
GetObjectMetadataRequest request = new GetObjectMetadataRequest()
{
BucketName = _bucket,
Key = _key
};
try
{
await s3Client.GetObjectMetadataAsync(request);
return true;
}
catch (AmazonS3Exception exception)
{
return false;
}
}
This function has worked for me when calling within a Unity game. You can also call the above function synchronously using the following:
bool exists = Task.Run(()=>FileExistsS3Async(_bucket, _key)).Result;

Try this solution, it works for me.
AmazonS3Client client = new AmazonS3Client(accessKey, secretKey, regionEndpoint);
S3FileInfo s3FileInfo = new S3FileInfo(client, bucketName, fileName);
return s3FileInfo.Exists;

There is no ListObjectRequest, but instead a ListObjectsRequest where you cannot specify the Key. You then have to go through all the objects to find the one you want. I am currently looking in to it since I seem to get time out errors whilst downloading the file. (If anyone has some idea how to solve that feel free to comment).
You could instead try the List Parts Request if you happen to know the upload id.
Other than that I have no idea. Would like to have a chat with the person who wrote the S3 api...

You're probably going to have to use the REST API yourself, as the method suggested, internally just does exactly the same thing (try...catch on the request)

You can use this code to check whether an object exist in S3 or not:
public class S3CheckFileExists
{
private readonly IAmazonS3 amazonS3;
public S3CheckFileExists(IAmazonS3 amazonS3)
{
this.amazonS3 = amazonS3;
}
public async Task<bool> S3ObjectExists(string bucketName, string keyLocation)
{
var listS3Objects = await this.amazonS3.ListObjectsV2Async(new ListObjectsV2Request
{
BucketName = bucketName,
Prefix = keyLocation, // eg myfolder/myimage.jpg (no / at start)
MaxKeys = 1
});
if (listS3Objects.S3Objects.Any() == false || listS3Objects.S3Objects.All(x => x.Key != keyLocation))
{
// S3 object doesn't exist
return false;
}
// S3 object exists
return true;
}
}
You'll need to register IAmazonS3 in your IoC (aka services) container though:
services.AddAWSService<IAmazonS3>();

Yes.
You can use a ListObjectsRequest. Use the Marker property, and retrieve only 1 element.