We Keep Coding

Azure Function SSH.Net Socket read operation has timed out

I'm attempting to connect from a timed Azure function to a 3rd party SFTP server that I have access to, but do not control. My function runs successfully locally when using the azure functions emulator, however I receive an exception ("Socket read operation has timed out after 30000 milliseconds.") when attempting to run in Azure.
Is there anything from a networking perspective I need to do to allow/set up outbound SFTP connections, or does anyone see anything wrong with my code below?
var ftpHost = Environment.GetEnvironmentVariable("SFTP:Server");
var ftpUser = Environment.GetEnvironmentVariable("SFTP:User");
var ftpPass = Environment.GetEnvironmentVariable("SFTP:Password");
var ftpDirectory = Environment.GetEnvironmentVariable("SFTP:WorkingDirectory");
log.Info($"Connecting to {ftpHost}"); //This outputs the correct values I would expect from my app settings
using (var sftp = new SftpClient(ftpHost, ftpUser, ftpPass))
{
sftp.Connect(); //This throws the exception
log.Info("Connected");
var files = sftp.ListDirectory(ftpDirectory);
log.Info("Directory listing successful");
var exceptions = new List<Exception>();
foreach (var file in files.Where(f => f.IsRegularFile))
{
try
{
log.Info($"{file.FullName} - {file.LastWriteTimeUtc}");
var records = Process(sftp, file);
log.Info($"Parsed {records.Count} records");
sftp.DeleteFile(file.FullName);
log.Info($"Deleted {file.FullName}");
}
catch (Exception ex)
{
exceptions.Add(ex);
}
}
if (exceptions.Any())
{
throw new AggregateException(exceptions);
}
}
Edit
I did leave my failing code out there and the failures appear to be intermittent. Running every 15 minutes, I have a roughly 50% success rate. In the last 20 attempts, 10 have succeeded.

Setting file permissions in batch

I'm outputting the following error that's generated when executing a batch of file permission changes to a file. I'm not sure what to make of this, since it's only a problem for some of the files, and the issue doesn't trigger when trying to run in debug mode:
500 >> Internal Error. User message: "An internal error has occurred which prevented the sharing of these item(s): Example File.DOCX"
I'm using the following code:
var batch = new Google.Apis.Requests.BatchRequest(service);
Google.Apis.Requests.BatchRequest.OnResponse<Permission> callback = delegate (
Permission permission,
Google.Apis.Requests.RequestError error,
int index,
System.Net.Http.HttpResponseMessage message) {
if (error != null) {
// Handle error
Console.WriteLine("File PERMISSION Error: " + error.Code + " >> " + error.Message);
} else {
Console.WriteLine("File Permission ID: " + permission.Id);
}
};
Permission filePermission = new Permission()
{
EmailAddress = "test-email#gmail.com"
, Type = GoogleDriveRoleType
, Role = GoogleDriveRole
};
var permExec = service.Permissions.Create(filePermission, googleDriveObjectId);
permExec.SendNotificationEmail = false;
permExec.Fields = "id";
batch.Queue(permExec, callback);
await batch.ExecuteAsync();
This code is within a method that's public static async Task MyMethod(...).

You may want to make your batch request smaller. You might be experiencing 500 internal error because you are flooding the server with too many request per second. As stated in this related SO post, as the server handles your batch request, it is not smart enough to slow down to avoid the error 500. You can also use exponential backoff and then retry the batch request. As stated in this forum, currently there is no way to know in a batch request what part fail or succeed. You have to create your own implementation regarding this. Hope this helps.

Checking HTTP Status of Many Pages on IIS Express: Crashes IIS Express

I am writing a simple C# console application, whose main job is, when given a set of URLs, to ping those URLs and report whether or not an HTTP 200 OK result was returned. The real life data set is in the area of 20,000 URLs to test (to verify that an en-masse edit did not ruin any of the pages).
Currently, the code that checks the response looks like this:
public UrlTestResult TestUrl(string url)
{
if (string.IsNullOrWhiteSpace(url))
{
throw new ArgumentNullException("url");
}
using (var client = new HttpClient())
{
try
{
Task<HttpResponseMessage> message = client.GetAsync(url);
if (message == null || message.Result == null)
{
return new FailedUrlTestResult(url, "No response was returned.");
}
if (message.Result.StatusCode == HttpStatusCode.OK)
{
return new SuccessfulUrlTestResult(url);
}
return new FailedUrlTestResult(url, "{0}: {1}".Format((int)message.Result.StatusCode, message.Result.ReasonPhrase));
}
catch (Exception ex)
{
return new FailedUrlTestResult(url, "An exception occurred: " + ex);
}
}
}
This code does work for smaller sets of data. Even if I iterate over the collection of URLs using Parallel.ForEach instead of a normal foreach, it behaves fine. After running for a few minutes or so, however, when parsing the 20,000 dataset, IIS Express (hosting localhost) will crash.
I'm guessing that my options are:
Run out of IIS and see if that works
Throttle the number of requests to give IIS Express a chance to breathe (trick here is how much to throttle)
Test the URLs in smaller chunks (similar to the second option)
What I am wondering is:
Is there a "cheaper" way to ping a URL and get its HTTP response back than HttpClient?
Are there any configuration options for IIS or IIS Express that I should be taking into consideration?
EDIT: I'm finding that IIS Express seems to simply be running out of memory. Pictured is the instance where the crash occurs:
Which means that IIS Express is holding on to memory that it obviously doesn't need to be (because once the request is over, I don't care about it anymore). Don't know if this'll help solve my problem any, though.

I simply changed to running localhost out of IIS instead of IIS Express. The memory usage was about the same, but it never crashed at any point for the ten minutes that the application was running. I also took Gabi's comment/suggestion and made HttpClient only be instantiated one time instead of once per test. The final code looks like this:
public sealed class UrlTester : IUrlTester
{
private readonly HttpClient httpClient = new HttpClient();
public UrlTestResult TestUrl(string url)
{
if (string.IsNullOrWhiteSpace(url))
{
throw new ArgumentNullException("url");
}
try
{
Task<HttpResponseMessage> message = httpClient.GetAsync(url);
if (message == null || message.Result == null)
{
return new FailedUrlTestResult(url, "No response was returned.");
}
if (message.Result.StatusCode == HttpStatusCode.OK)
{
return new SuccessfulUrlTestResult(url);
}
return new FailedUrlTestResult(url, "{0}: {1}".FormatCurrentCulture((int)message.Result.StatusCode, message.Result.ReasonPhrase));
}
catch (Exception ex)
{
return new FailedUrlTestResult(url, "An exception occurred: " + ex);
}
}
public void Dispose()
{
if (httpClient != null)
{
httpClient.Dispose();
}
}
}
And the caller to this class utilizes C#'s using statement to ensure that the HttpClient instance is properly disposed of.

WebService Call Exception: Thread was being aborted

Recently we've been getting System.Threading.ThreadAbortExceptions from an ASP.NET webservice that posts data to a payment server each night.
The webservice is a standard .asmx page that gets called from another client.
Inside the page I have a foreach loop that iterates through many payment rows in a database:
foreach (var obtRow in readyToBeCaptured)
{
try
{
status = dibs.GetPagePost(...);
// handle status here
}
catch (ThreadAbortException tex)
{
SingletonLogger.Instance.Error(string.Format("Transactionhandler - Could not Capture, Thread was aborted. {0}", tex.Message), tex);
}
catch (Exception ex)
{
SingletonLogger.Instance.Error(string.Format("Transactionhandler - Could not Capture, statuscode: {0}, message: {1}.", status, ex.Message), ex);
}
}
The strange thing is I get a log entry from catch(ThreadAbortException tex) block when this error occurs, but then the code breaks out and is caught in another try catch block further up the call tree. Ideally I would have the code inside the foreach loop to continue
This is the GetPagePost method
private bool GetPagePost(string url, NameValueCollection nameValueCollection, string userName, string password)
{
WebClient client = new WebClient();
if (userName != "")
{
client.Credentials = new NetworkCredential(userName, password);
}
foreach (string str in nameValueCollection.AllKeys)
{
this._callCollection.Add(str, nameValueCollection[str]);
}
StringBuilder builder = new StringBuilder();
builder.Append("DIBS.NET/1.2.0 ( ");
builder.Append("OS ").Append(Environment.OSVersion.Platform).Append(" ").Append(Environment.OSVersion.Version).Append("; ");
builder.Append(".NET CLR ").Append(Environment.Version).Append("; ");
builder.Append("User ").Append(Environment.UserName).Append("; ");
builder.Append("Domain ").Append(Environment.UserDomainName).Append("; ");
builder.Append(" ) ");
client.Headers.Add("User-Agent", builder.ToString());
try
{
byte[] bytes = client.UploadValues(url, "POST", nameValueCollection);
this._httpStatus = 200;
this._httpBody = Encoding.UTF8.GetString(bytes);
return true;
}
catch (WebException exception)
{
this._httpBody = exception.Message;
this._httpStatus = (int) exception.Status;
}
catch (UriFormatException exception2)
{
this._httpBody = exception2.Message;
this._httpStatus = -9999;
}
catch (Exception exception3)
{
this._httpBody = exception3.Message;
this._httpStatus = -99999;
}
return false;
}}
Why is this error occuring and how can I prevent the code from breaking out of the foreach loop?
I've been looking at a few other posts here on StackOverflow but they seem to relate to the usage of Reponse.Redirect which I don't use.
Thanks
/Jens

I have a similar issue. I think IIS is terminating your thread for taking too long. From your description "I have a foreach loop that iterates through many payment rows in a database", you may be able to restructure your app to process each row individually. Create a webmethod to send the rows to the client, then have the client iterate the transactions one at a time and process them in a different webmethod. Then you can handle exceptions that occur and keep going. You may even be able to process many at once using Parallel.ForEach

Last days I had a similar problem with a web service and solved it adding to </system.web> of web.config this line: <httpRuntime executionTimeout="600"/>.
600 is the number of seconds till is shut downed by asp.net. The default is 110 seconds.
It explains here better what it does.

After some RND try few things:
try adding custom account in the application pool
try repairing the .net framework by running the NetFxRepairTool exe from official Microsoft website
try adding httpRuntime and targetFramework="4.5" (This works for me targetFramework="4.5")
httpRuntime targetFramework="4.5" maxRequestLength="4000"
executionTimeout="45"
ref: http://dotnetvisio.blogspot.com/2013/07/solution-for-thread-being-aborted-call.html

Get sent mail error

Is there any way to get sent error from the smtp to check if the mail is sent successfully?
var smtpClient = new SmtpClient("SmtpServer");
smtpClient.SendCompleted += new SendCompletedEventHandler(SendCompletedCallback);
smtpClient.SendAsync(mail, userId);
The errors I am looking for are: mail can't be deliver because the mail address not exists, mail box full etc...
Regards,
Meir.

I am not sure what you are trying to achieve but this will helps you.
I assume you're already aware of the DeriveryNotificationOptions property on System.Net.Mail.MailMessage. The only tricky part to using that property is that its enum type represents a bitfield, so you should set it to the sum of the options you want to apply.
For example, if you want delivery notification on delay, failure, or success, you should set the property to
DeliveryNotificationOptions.Delay + DeliveryNotificationOptions.OnFailure + DeliveryNotificationOptions.OnSuccess
Or
this is one method to capture the failure report or any error when the mail has not been sent (failure report)
// Change your Try-Catch to call the new method named 'CheckExceptionAndResend'
// Error handling for sending message
try
{
smtpClient.Send(message);
// Exception contains information on each failed receipient
}
catch (System.Net.Mail.SmtpFailedRecipientsException recExc)
{
// Call method that will analyze exception and attempt to re-send the email
CheckExceptionAndResend(recExc, smtpClient, message);
}
catch (System.Net.Mail.SmtpException smtpExc)
{
// Log error to event log using StatusCode information in
// smtpExc.StatusCode
MsgBox((smtpExc.StatusCode.ToString + " ==>Procedure SmtpException"));
}
catch (Exception Exc)
{
// Log error to event log using StatusCode information in
// smtpExc.StatusCode
MsgBox((Exc.Message + " ==>Procedure Exception"));
}
private void CheckExceptionAndResend(System.Net.Mail.SmtpFailedRecipientsException exObj, System.Net.Mail.SmtpClient smtpClient, MailMessage emailMessage)
{
try
{
for (int recipient = 0; (recipient <= (exObj.InnerExceptions.Length - 1)); recipient++)
{
System.Net.Mail.SmtpStatusCode statusCode;
// Each InnerException is an System.Net.Mail.SmtpFailed RecipientException
statusCode = exObj.InnerExceptions(recipient).StatusCode;
if (((statusCode == Net.Mail.SmtpStatusCode.MailboxBusy)
|| (statusCode == Net.Mail.SmtpStatusCode.MailboxUnavailable)))
{
// Log this to event log: recExc.InnerExceptions(recipient).FailedRecipient
System.Threading.Thread.Sleep(5000);
smtpClient.Send(emailMessage);
}
else
{
// Log error to event log.
// recExc.InnerExceptions(recipient).StatusCode or use statusCode
}
}
MsgBox((exObj.Message + " ==>Procedure SmtpFailedRecipientsException"));
}
catch (Exception ex)
{
// At this point we have an non recoverable issue:
// NOTE: At this point we do not want to re-throw the exception because this method
// was called from a 'Catch' block and we do not want a hard error to display to the client.
// Options: log error, report issue to client via msgbox, etc. This is up to you.
// To display issue as you have before:
MsgBox((exObj.Message + " ==>Email was not sent"));
}
}

Such kind of errors have a asnychronous nature. When sending mail you talk to the local smtp server of your provider. That server afterwards starts to deliver the mail to the target mail system.
So the SmtpClient class can only show you errors occuring while talking to your local smtp server.
Typically when an error like "unknown user" occures on the target system, it will send an email with the failure message to the originator email address.

This post is helpful to me.
By the way if you're using .net 4.0 this one will be the changes on the above code. Sorry for my first post i don't know why it appears that way.
Here's the code:
private void CheckExceptionAndResend(System.Net.Mail.SmtpFailedRecipientsException exObj, System.Net.Mail.SmtpClient smtpClient, MailMessage emailMessage)
{
try
{
for (int recipient = 0; (recipient <= (exObj.InnerExceptions.Length - 1)); recipient++)
{
System.Net.Mail.SmtpStatusCode statusCode;
// Each InnerException is an System.Net.Mail.SmtpFailed RecipientException
//for .net 4.0
//statusCode = exObj.InnerExceptions(recipient).StatusCode;
statusCode = exObj.StatusCode;
//if (((statusCode == Net.Mail.SmtpStatusCode.MailboxBusy) || (statusCode == Net.Mail.SmtpStatusCode.MailboxUnavailable)))
//for .net 4.0
if (((statusCode == System.Net.Mail.SmtpStatusCode.MailboxBusy)
|| (statusCode == System.Net.Mail.SmtpStatusCode.MailboxUnavailable)))
{
// Log this to event log: recExc.InnerExceptions(recipient).FailedRecipient
System.Threading.Thread.Sleep(5000);
smtpClient.Send(emailMessage);
}
else
{
// Log error to event log.
// recExc.InnerExceptions(recipient).StatusCode or use statusCode
}
}
//MsgBox((exObj.Message + " ==>Procedure SmtpFailedRecipientsException"));
}
catch (Exception ex)
{
// At this point we have an non recoverable issue:
// NOTE: At this point we do not want to re-throw the exception because this method
// was called from a 'Catch' block and we do not want a hard error to display to the client.
// Options: log error, report issue to client via msgbox, etc. This is up to you.
// To display issue as you have before:
// MsgBox((exObj.Message + " ==>Email was not sent"));
}
}