I'm outputting the following error that's generated when executing a batch of file permission changes to a file. I'm not sure what to make of this, since it's only a problem for some of the files, and the issue doesn't trigger when trying to run in debug mode:
500 >> Internal Error. User message: "An internal error has occurred which prevented the sharing of these item(s): Example File.DOCX"
I'm using the following code:
var batch = new Google.Apis.Requests.BatchRequest(service);
Google.Apis.Requests.BatchRequest.OnResponse<Permission> callback = delegate (
Permission permission,
Google.Apis.Requests.RequestError error,
int index,
System.Net.Http.HttpResponseMessage message) {
if (error != null) {
// Handle error
Console.WriteLine("File PERMISSION Error: " + error.Code + " >> " + error.Message);
} else {
Console.WriteLine("File Permission ID: " + permission.Id);
}
};
Permission filePermission = new Permission()
{
EmailAddress = "test-email#gmail.com"
, Type = GoogleDriveRoleType
, Role = GoogleDriveRole
};
var permExec = service.Permissions.Create(filePermission, googleDriveObjectId);
permExec.SendNotificationEmail = false;
permExec.Fields = "id";
batch.Queue(permExec, callback);
await batch.ExecuteAsync();
This code is within a method that's public static async Task MyMethod(...).
You may want to make your batch request smaller. You might be experiencing 500 internal error because you are flooding the server with too many request per second. As stated in this related SO post, as the server handles your batch request, it is not smart enough to slow down to avoid the error 500. You can also use exponential backoff and then retry the batch request. As stated in this forum, currently there is no way to know in a batch request what part fail or succeed. You have to create your own implementation regarding this. Hope this helps.
Related
I'm uploading rather a lot of data (30gb+) across thousands of files. The whole process takes a while but I've been finding that consistently after 15 mins of transfers, the upload process fails and I get errors for each file that is currently being transferred (I'm doing it multithreaded so there are multiple uploads at once). The error code I'm getting is "error: Amazon.S3.AmazonS3Exception: The difference between the request time and the current time is too large. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden."
Seeing as its exactly 15 mins from the start of the whole process that this thing crashes, I think its maybe that the client is timing out, however I've set my client's timout to 45 mins I think:
{
var client = new AmazonS3Client(new AmazonS3Config()
{
RegionEndpoint = RegionEndpoint.EUWest2,
UseAccelerateEndpoint = true,
Timeout = TimeSpan.FromMinutes(45),
ReadWriteTimeout = TimeSpan.FromMinutes(45),
RetryMode = RequestRetryMode.Standard,
MaxErrorRetry = 10
});
Parallel.ForEach(srcObjList, async srcObj =>
{
try
{
var putObjectRequest = new PutObjectRequest();
putObjectRequest.BucketName = destBucket;
putObjectRequest.Key = srcObj.Key;
putObjectRequest.FilePath = filePathString;
putObjectRequest.CannedACL = S3CannedACL.PublicRead;
var uploadTask = client.PutObjectAsync(putObjectRequest);
lock (threadLock)
{
syncTasks.Add(uploadTask);
}
await uploadTask;
}
catch (Exception e)
{
Debug.LogError($"Copy task ({srcObj.Key}) failed with error: {e}");
throw;
}
});
try
{
await Task.WhenAll(syncTasks.Where(x => x != null).ToArray());
}
catch (Exception e)
{
Debug.LogError($"Upload encountered an issue: {e}");
}
});
await transferOperations;
Debug.Log("Done!");```
The documentation doesn't specify the maximum timeout value, but given that you're seeing 15 minutes exactly, it stands to reason there is some upper limit to the timeout value, either a hard limit or something in the S3 bucket's settings.
This answer suggests a clock synchronization difference might also be the case, but then I'd wonder why the transfer starts at all.
I have a simple Rest Web service that raises an exception randomly (Internal Server Error).
I noticed it when I have stressed this Web api in a console program in a 50.000 iterations loop.
What I get in this client :
{StatusCode: 500, ReasonPhrase: 'Internal Server Error', Version: 1.1,
What I want is to get this Error 500 detail.
But I cannot catch it in my controller, the code doesn't go in the catch.
The controller code :
[HttpPost]
[Route("{IdLog}/message")]
public string CreateLogMsg(long IdLog, [FromBody] TlogLineDTO oLogLine)
{
long lTmp = 0;
try
{
if (ModelState.IsValid)
{
string sTmp = _oExploitBll.InsertNewLogLine(oLogLine).ToString();
if (!long.TryParse(sTmp, out lTmp))
{
// Create a new file
Random rnd = new Random();
using (System.IO.StreamWriter sw = System.IO.File.AppendText(#"E:\Temp\20200707\REF_" + rnd.Next().ToString() + ".txt"))
{
sw.WriteLine(sTmp);
}
}
return sTmp;
}
else
{
// Create a new file where to put the error message
Random rnd = new Random();
using (System.IO.StreamWriter sw = System.IO.File.AppendText(#"E:\Temp\20200707\REF_" + rnd.Next().ToString() + ".txt"))
{
sw.WriteLine("Ko");
}
return "Ko";
}
}
catch (Exception ex)
{
string sMsgErr = ex.Message + "-" + ex.StackTrace;
// Create a new file
Random rnd = new Random();
using (System.IO.StreamWriter sw = System.IO.File.AppendText(#"E:\Temp\20200707\REF_" + rnd.Next().ToString() + ".txt"))
{
sw.WriteLine(sMsgErr);
}
return "0";
}
}
How can I make it go in the catch ?
Does someone have an idea ?
Thanks a lot in advance.
Eric
If, as you say, the error happens because the server is too stressed the requests which get the error never reach the controller. Actually the server cannot even enqueue them. Sso you can't handle them unless you add a side-car server which receives requests from the client, forwards them to your server and then
if response is ok forwards response to client
else handles the error
In real case scenarios you use a load balancer to divide the request load between more servers
In fact, I have corrected the code and added a loop of 2 retries for the http requests and it seems to work : when the http request fails, I retry it once (or the number of retries I want) and generally the http request success in the second retry.
But I still don't know why the http requests fail sometimes.
I'm attempting to connect from a timed Azure function to a 3rd party SFTP server that I have access to, but do not control. My function runs successfully locally when using the azure functions emulator, however I receive an exception ("Socket read operation has timed out after 30000 milliseconds.") when attempting to run in Azure.
Is there anything from a networking perspective I need to do to allow/set up outbound SFTP connections, or does anyone see anything wrong with my code below?
var ftpHost = Environment.GetEnvironmentVariable("SFTP:Server");
var ftpUser = Environment.GetEnvironmentVariable("SFTP:User");
var ftpPass = Environment.GetEnvironmentVariable("SFTP:Password");
var ftpDirectory = Environment.GetEnvironmentVariable("SFTP:WorkingDirectory");
log.Info($"Connecting to {ftpHost}"); //This outputs the correct values I would expect from my app settings
using (var sftp = new SftpClient(ftpHost, ftpUser, ftpPass))
{
sftp.Connect(); //This throws the exception
log.Info("Connected");
var files = sftp.ListDirectory(ftpDirectory);
log.Info("Directory listing successful");
var exceptions = new List<Exception>();
foreach (var file in files.Where(f => f.IsRegularFile))
{
try
{
log.Info($"{file.FullName} - {file.LastWriteTimeUtc}");
var records = Process(sftp, file);
log.Info($"Parsed {records.Count} records");
sftp.DeleteFile(file.FullName);
log.Info($"Deleted {file.FullName}");
}
catch (Exception ex)
{
exceptions.Add(ex);
}
}
if (exceptions.Any())
{
throw new AggregateException(exceptions);
}
}
Edit
I did leave my failing code out there and the failures appear to be intermittent. Running every 15 minutes, I have a roughly 50% success rate. In the last 20 attempts, 10 have succeeded.
I am trying to programmatically get my site status from IIS to see if it's stopped, but I kept getting the following error,
The object identifier does not represent a valid object. (Exception from HRESULT: 0x800710D8)
The application is using ServerManager Site class to access the site status. Here is the code,
//This is fine, gets back the site
var serverManager = new Microsoft.Web.Administration.ServerManager(ConfigPath);
var site = serverManager.Sites.FirstOrDefault(x => x.Id == 5);
if (site == null) return;
var appPoolName = site.Applications["/"].ApplicationPoolName;
//error!
var state = site.State;
I've test with static site to isolate the issue, making sure that the site is up and running, all configuration are valid, point to the valid application pool...etc.
Let me know if you need more details. Is it the COM thing?
I figured out where the problem is. Basically, there are two parts to the Server manager, the first part of the server manager allows you to read site details from configuration file, which is what I've been doing above. The problem with that is you will only able get the information that's in file and site state is not part of it.
The second part of the Server Manager allows you to connect to the IIS directly and it does this by interacting with the COM element. So what I should be doing is this:
ServerManager manager= ServerManager.OpenRemote("testserver");
var site = manager.Sites.First();
var status = site.State.ToString() ;
I had a similar problem but mine was caused by the delay needed to activate the changes from the call to CommitChanges on the ServerManager object. I found the answer I needed here:
ServerManager CommitChanges makes changes with a slight delay
It seems like polling is required to get consistent results. Something similar to this solved my problem (I got the exception when accessing a newly added application pool):
...
create new application pool
...
sman.CommitChanges();
int i = 0;
const int max = 10;
do
{
i++;
try
{
if (ObjectState.Stopped == pool.State)
{
write_log("Pool was stopped, starting: " + pool.Name);
pool.Start();
}
sman.CommitChanges();
break;
}
catch (System.Runtime.InteropServices.COMException e)
{
if (i < max)
{
write_log("Waiting for IIS to activate new config...");
Thread.Sleep(1000);
}
else
{
throw new Exception(
"CommitChanges timed out efter " + max + " attempts.",
e);
}
}
} while (true);
...
In C# ASP.NET 3.5 web application running on Windows Server 2003, I get the following error once in a while:
"Object reference not set to an instance of an object.: at System.Messaging.Interop.MessagePropertyVariants.Unlock()
at System.Messaging.Message.Unlock()
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageEnumerator.get_Current()
at System.Messaging.MessageQueue.GetAllMessages()".
The line of code that throws this error is:
Message[] msgs = Global.getOutputQueue(mode).GetAllMessages();
where Global.getOutputQueue(mode) gives the messagequeue I want to get messages from.
Update:
Global.getPool(mode).WaitOne();
commonClass.log(-1, "Acquired pool: " + mode, "Report ID: " + unique_report_id);
............../* some code /
..............
lock(getLock(mode))
{
bool yet_to_get = true;
int num_retry = 0;
do
{
try
{
msgs = Global.getOutputQueue(mode).GetAllMessages();
yet_to_get = false;
}
catch
{
Global.setOutputQueue(mode);
msgs = Global.getOutputQueue(mode).GetAllMessages();
yet_to_get = false;
}
++num_retry;
}
while (yet_to_get && num_retry < 2);
}
... / some code*/
....
finally
{
commonClass.log(-1, "Released pool: " + mode, "Report ID: " + unique_report_id);
Global.getPool(mode).Release();
}
Your description and this thread suggests a timing issue. I would create the MessageQueue object infrequently (maybe only once) and have Global.getOutputQueue(mode) return a cached version, seems likely to get around this.
EDIT: Further details suggest you have the opposite problem. I suggest encapsulating access to the message queue, catching this exception and recreating the queue if that exception occurs. So, replace the call to Global.getOutputQueue(mode).GetAllMessages() with something like this:
public void getAllOutputQueueMessages()
{
try
{
return queue_.GetAllMessages();
}
catch (Exception)
{
queue_ = OpenQueue();
return queue_.GetAllMessages();
}
}
You'll notice I did not preserve your mode functionality, but you get the idea. Of course, you have to duplicate this pattern for other calls you make to the queue, but only for the ones you make (not the whole queue interface).
This is an old thread, but google brought me here so I shall add my findings.
I agree with user: tallseth that this is a timing issue.
After the message queue is created it is not instantly available.
try
{
return _queue.GetAllMessages().Length;
}
catch (Exception)
{
System.Threading.Thread.Sleep(4000);
return _queue.GetAllMessages().Length;
}
try adding a pause if you catch an exception when accessing a queue which you know has been created.
On a related note
_logQueuePath = logQueuePath.StartsWith(#".\") ? logQueuePath : #".\" + logQueuePath;
_queue = new MessageQueue(_logQueuePath);
MessageQueue.Create(_logQueuePath);
bool exists = MessageQueue.Exists(_logQueuePath);
running the MessageQueue.Exists(string nameofQ); method immediately after creating the queue will return false. So be careful when calling code such as:
public void CreateQueue()
{
if (!MessageQueue.Exists(_logQueuePath))
{
MessageQueue.Create(_logQueuePath);
}
}
As it is likely to throw an exception stating that the queue you are trying to create already exists.
-edit: (Sorry I don't have the relevant link for this new info)
I read that a newly created MessageQueue will return false on MessageQueue.Exists(QueuePath)until it has received at least one message.
Keeping this and the earlier points i mentioned in mind has gotten my code running reliably.