We Keep Coding

FluentFTP connection broke but no exception

I use FluentFTP in my code to transfer data internally to a FTP server. If the connection to the FTP server breaks down during the upload, then there is no exception.
But oddly enough, that doesn't happen with all dates! If I take a *.7z file, there is an exception when the connection is broken.
I'm confused!
When transferring a *.7z file, why does it recognize that the connection was interrupted (service stopped) and restart the connection when the service is available again and with a *.opus file does the program stop in an await?
public class FileWatcher
{
public static async Task Main(string[] args)
{
do
{
Console.WriteLine("Und los geht es!");
await UploadFileAsync();
await Task.Delay(15000);
} while (true);
}
static async Task UploadFileAsync()
{
try
{
string[] filePath = Directory.GetFiles(#"C:\temp\ftpupload", "*",
SearchOption.AllDirectories);
var token = new CancellationToken();
using (AsyncFtpClient client = new AsyncFtpClient())
{
client.Host = "192.168.1.100";
client.Port = 21;
client.Credentials.UserName = "test";
client.Credentials.Password = "test123";
client.Config.EncryptionMode = FtpEncryptionMode.None;
client.Config.InternetProtocolVersions = FtpIpVersion.IPv4;
client.Config.ValidateAnyCertificate = true;
client.Config.ConnectTimeout = 10000;
Console.WriteLine("Connecting......");
await client.AutoConnect(token);
Console.WriteLine("Connected!");
foreach (var erg in filePath)
{
Console.WriteLine("File is uploading: " + erg.GetFtpFileName());
await client.UploadFile(erg, "/" + erg.GetFtpFileName(),
FtpRemoteExists.Overwrite, true, token: token);
Console.WriteLine("File successfully uploaded: " +
erg.GetFtpFileName());
System.IO.File.Delete(erg);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
Error while uploading the file to the server. See InnerException for more info.

I think the problem is that you are not catching the exception from the Main method. The code inside the try-catch block will execute correctly, but if an exception occurs outside the try-catch block, the program will terminate without reporting the error.
So to fix it, you should add a try-catch block in the Main method and inside it, call the UploadFileAsync() method with the await keyword.
Another reason may be the size of the file, or the delay you set in the Main method.

How can I build and return a large zip file in a performant manner?

Background
I'm currently working on a .Net Core - C# application that is reliant on various Azure services. I've been tasked with creating an endpoint that allows users to bulk download a varying number of files based on some querying/filtering. The endpoint will be triggered by a download all button on the frontend and should return a .zip of all said files. The total size of this zip could be anywhere from 100KB-100GB depending on the query/filters provided.
Note: Although I'm familiar with Asynchrony, Concurrency, and Streams. Interactions between these and between api layers is something Im still getting my head around. Bear with me.
Question
How can I achieve this in a performant and scalable manner given some architectural constraints? Details provided below.
Architecture
The backend currently consist of 2 main layers. The API Layer consist of Azure Functions which are the first point of contact for any and all request from the frontend. The Service Layer stands in-between the API Layer and other Azure Services. In this particular case the Service Layer interacts with an Azure Blob Storage Container, where the various files are stored.
Current Implementation/Plan
Request:
The request itself is strait forward. The API Layer takes query's and filters and turns that into a list of filenames. That is then sent in the body of a POST request to the Service Layer. The Service Layer loops through the list and retrieves each file individually from the blob storage. As of right now there is no way of bulk downloading attachments. This is where complications start.
Given the potential file size, can't pull all data into memory at one time, may need to be streamed or batched.
Given many files, may need to download files in parallel from blob storage.
Need to build zip file from async parallel task? Which can't be built completely in memory.
Response:
I currently have a working version of this that doesn't worry about memory. The above diagram is meant as an illustration of the limitations/considerations of the task rather than a concept that can be put to code. No one layer can/should hold all of the data at any given time. My original attempt/idea was to use a series of streams that can pipe data down the line in some manor. However, I realized this might be a fools errand and decided to make this post.
Any thoughts on a better high-level work flow to accomplish this task would be greatly appreciated. I would also love to hear completely different solutions to the problem.

Thank you Sha. Posting your suggestions as an answer to help other community members.
POST a list of file paths to a Azure Function (Http trigger)
Create a queue message containing the file paths and put on a storage queue.
Listen to said storage queue with another Azure function (Queue trigger).
Stream each file from Azure Storage -> Add it to a Zip stream -> Stream it back to Azure storage.
Below code will help on creating ZIP file.
public class AzureBlobStorageCreateZipFileCommand : ICreateZipFileCommand
{
private readonly UploadProgressHandler _uploadProgressHandler;
private readonly ILogger<AzureBlobStorageCreateZipFileCommand> _logger;
private readonly string _storageConnectionString;
private readonly string _zipStorageConnectionString;
public AzureBlobStorageCreateZipFileCommand(
IConfiguration configuration,
UploadProgressHandler uploadProgressHandler,
ILogger<AzureBlobStorageCreateZipFileCommand> logger)
{
_uploadProgressHandler = uploadProgressHandler ?? throw new ArgumentNullException(nameof(uploadProgressHandler));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_storageConnectionString = configuration.GetValue<string>("FilesStorageConnectionString") ?? throw new Exception("FilesStorageConnectionString was null");
_zipStorageConnectionString = configuration.GetValue<string>("ZipStorageConnectionString") ?? throw new Exception("ZipStorageConnectionString was null");
}
public async Task Execute(
string containerName,
IReadOnlyCollection<string> filePaths,
CancellationToken cancellationToken)
{
var zipFileName = $"{DateTime.UtcNow:yyyyMMddHHmmss}.{Guid.NewGuid().ToString().Substring(0, 4)}.zip";
var stopwatch = Stopwatch.StartNew();
try
{
using (var zipFileStream = await OpenZipFileStream(zipFileName, cancellationToken))
{
using (var zipFileOutputStream = CreateZipOutputStream(zipFileStream))
{
var level = 0;
_logger.LogInformation("Using Level {Level} compression", level);
zipFileOutputStream.SetLevel(level);
foreach (var filePath in filePaths)
{
var blockBlobClient = new BlockBlobClient(_storageConnectionString, containerName, filePath);
var properties = await blockBlobClient.GetPropertiesAsync(cancellationToken: cancellationToken);
var zipEntry = new ZipEntry(blockBlobClient.Name)
{
Size = properties.Value.ContentLength
};
zipFileOutputStream.PutNextEntry(zipEntry);
await blockBlobClient.DownloadToAsync(zipFileOutputStream, cancellationToken);
zipFileOutputStream.CloseEntry();
}
}
}
stopwatch.Stop();
_logger.LogInformation("[{ZipFileName}] DONE, took {ElapsedTime}",
zipFileName,
stopwatch.Elapsed);
}
catch (TaskCanceledException)
{
var blockBlobClient = new BlockBlobClient(_zipStorageConnectionString, "zips", zipFileName);
await blockBlobClient.DeleteIfExistsAsync();
throw;
}
}
private async Task<Stream> OpenZipFileStream(
string zipFilename,
CancellationToken cancellationToken)
{
var zipBlobClient = new BlockBlobClient(_zipStorageConnectionString, "zips", zipFilename);
return await zipBlobClient.OpenWriteAsync(true, options: new BlockBlobOpenWriteOptions
{
ProgressHandler = _uploadProgressHandler,
HttpHeaders = new BlobHttpHeaders
{
ContentType = "application/zip"
}
}, cancellationToken: cancellationToken);
}
private static ZipOutputStream CreateZipOutputStream(Stream zipFileStream)
{
return new ZipOutputStream(zipFileStream)
{
IsStreamOwner = false
};
}
}
Check Zip File using Azure functions for further information.

FirebaseAdmin FirebaseApp Check if Firebase Instance exists

I have a C# project that subscribes multiple Registrations to a Topic. Because of the nature of the project and the fact that you cant check to see how many people have already subscribed to a Topic I need to make the following Async Calls to the server:
Subscribe Registrations
TopicManagementResponse response = await FirebaseMessaging.DefaultInstance.SubscribeToTopicAsync(registrationTokens, topic);
Send message to Topic
string response = await FirebaseMessaging.DefaultInstance.SendAsync(message);
Unsubscribe Registrations
TopicManagementResponse response = await FirebaseMessaging.DefaultInstance.UnsubscribeFromTopicAsync(registrationTokens, topic);
Because there are three calls I need to Create an Instance of the FirebaseApp using Credentials:
FirebaseApp.Create(new AppOptions()
{
Credential = GoogleCredential.FromFile(path),
});
BUT because the async posts return a "WaitingForActivation" response (yet it does correctly do what it is supposed to do) I cant Delete the Instance to move on to the next function as it throws an error as it cant re-create another FirebaseApp Instance - It fails if I give it a name so I cant use GetInstance(string name).
Am I missing something or is there another way to do this.
Here is an example of a subscribe function:
internal static async Task SubscribeToTopic(string path, string topic, string regID5, string regID)
{
FirebaseApp app = FirebaseApp.Create(new AppOptions()
{
Credential = GoogleCredential.FromFile(path),
});
var registrationTokens = new List<string>()
{
regID5, regID
};
// Subscribe the devices corresponding to the registration tokens to the
// topic
try
{
TopicManagementResponse response = await FirebaseMessaging.DefaultInstance.SubscribeToTopicAsync(registrationTokens, topic);
using (StreamWriter sw = System.IO.File.AppendText(HttpContext.Current.Server.MapPath("/tokens.txt")))
{
sw.WriteLine($"{response.SuccessCount} tokens were subscribed successfully");
}
}
catch (Exception ex)
{
string myerror = ex.Message;
}
}
Any ideas?

you are creating firebase instance every time. so you need to create firebase instance in application start in global.asax file.

How to perform database operation independently?

I have 1 exe which is nothing bit a Windows form which will continuously run in background and will watch my serial port and I have 1 event data receive event which fires as my serial port receive data.
As soon as I receive data in this event I will pass this data to another event handler which saves this data in database through web api method.
But data to my serial port will be coming frequently so I want to save this data to my database independently so that my database insert operation doesn't block my incoming serial port data.
This is my code:
void _serialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)//Fires as my serial port receives data
{
int dataLength = _serialPort.BytesToRead;
byte[] data = new byte[dataLength];
int nbrDataRead = _serialPort.Read(data, 0, dataLength);
if (nbrDataRead == 0)
return;
// Send data to whom ever interested
if (NewSerialDataRecieved != null)
{
NewSerialDataRecieved(this, new SerialDataEventArgs(data)); //pass serial port data to new below event handler.
}
}
void _spManager_NewSerialDataRecieved(object sender, SerialDataEventArgs e) //I want this event handler to run independently so that database save operation doenst block incoming serial port data
{
if (this.InvokeRequired)
{
// Using this.Invoke causes deadlock when closing serial port, and BeginInvoke is good practice anyway.
this.BeginInvoke(new EventHandler<SerialDataEventArgs>(_spManager_NewSerialDataRecieved), new object[] { sender, e });
return;
}
//data is converted to text
string str = Encoding.ASCII.GetString(e.Data);
if (!string.IsNullOrEmpty(str))
{
//This is where i will save data to through my web api method.
RunAsync(str).Wait();
}
}
static async Task RunAsync(string data)
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://localhost:33396/");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var content = new StringContent(data);
var response = await client.PostAsJsonAsync<StringContent>("api/Service/Post", content);//nothing happens after this line.
}
}
Web api controller:
public class MyController : ApiController
{
[HttpPost]
public HttpResponseMessage Post(HttpRequestMessage request)
{
var someText = request.Content.ReadAsStringAsync().Result;
return new HttpResponseMessage() { Content = new StringContent(someText) };
}
}
But here problem is:
var response = await client.PostAsJsonAsync<StringContent>("api/Service/Post", content);
Nothing happens after this line that is operation blocks on this line.
So can anybody guide me with this?

By independently we determined in the SO C# chat room that you really mean "Asynchronously".
Your solution is the code above, saving this data to a WebAPI endpoint so any solution to the problem needs to be in 2 parts ...
PART 1: The Client Part
On the client all we need to do is make the call asynchronously in order to free up the current thread to carry on receiving data on the incoming serial port, we can do that like so ...
// build the api client, you may want to keep this in a higher scope to avoid recreating on each message
var api = new HttpClient();
api.BaseAddress = new Uri(someConfigVariable);
// asynchronously make the call and handle the result
api.PostAsJsonAsync("api/My", str)
.ContinueWith(t => HandleResponseAsync(t.Result))
.Unwrap();
...
PART 2: The Server Part
Since you have web api i'm also going to assume you are using EF too, the common and "clean" way to do this, with all the extras stripped out (like model validation / error handling) might look something like this ...
// in your EF code you will have something like this ...
Public async Task<User> SaveUser(User userModel)
{
try
{
var newUser = await context.Users.AddAsync(userModel);
context.SavechangesAsync();
return newUser;
}
catch(Exception ex) {}
}
// and in your WebAPI controller something like this ...
HttpPost]
public async Task<HttpResponseMessage> Post(User newUser)
{
return Ok(await SaveUser(newUser));
}
...
Disclaimer:
The concepts involved here go much deeper and as I hinted above, much has been left out here like validation, error checking, ect but this is the core to getting your serial port data in to a database using the technologies I believe you are using.
Key things to read up on for anyone wanting to achieve this kind of thing might include: Tasks, Event Handling, WebAPI, EF, Async operations, streaming.

From what you describe it seems like you might want to have a setup like this:
1) your windows form listens for serial port
2) when new stuff comes to port your windows forms app saves it to some kind of a queue (msmq, for example)
3) you should have separate windows service that checks queue and as it finds new messages in a queue it sends request to web api

Best solution for this problem is to use ConcurrentQueue.
Just do search on google and you will get planty of samples.
ConcurrentQueue is thread safe and it support writing and reading from multiple threads.
So the component listening to the searal port can write data to the queue. And you can have 2 or more tasks running parallel which listening to this queue and update db as soon as it receives data.

Not sure if it's the problem, but you shouldn't block on async code. You are doing RunAsync(str).Wait(); and I believe that's the problem. Have a look at this blog post by Stephen Cleary:
http://blog.stephencleary.com/2012/07/dont-block-on-async-code.html

Run stored procedure in a background Web Api

In my Web Api application I need to do something like in this topic: Best way to run a background task in ASP.Net web app and also get feedback? In the application user could upload an excel file and then import its data to tables in the Database. It all works fine, but the import process can take a long time (about 20 minutes, if the excel have a lot of rows) and when process is started the page is blocked and users have to wait all this time. I need this import process run in a background.
I have a controller with this POST method:
[HttpPost]
public async Task<IHttpActionResult> ImportFile(int procId, int fileId)
{
string importErrorMessage = String.Empty;
// Get excel file and read it
string path = FilePath(fileId);
DataTable table = GetTable(path);
// Add record for start process in table Logs
using (var transaction = db.Database.BeginTransaction()))
{
try
{
db.uspAddLog(fileId, "Process Started", "The process is started");
transaction.Commit();
}
catch (Exception e)
{
transaction.Rollback();
importErrorMessage = e.Message;
return BadRequest(importErrorMessage);
}
}
//Using myHelper start a store procedure, which import data from excel file
//The procedure also add record in table Logs when it is finished
using (myHelper helper = new myHelper())
helper.StartImport(procId, fileId, table, ref importErrorMessage);
if (!String.IsNullOrEmpty(importErrorMessage))
return BadRequest(importErrorMessage);
return Ok(true);
}
I also have a GET method, which returns information about file and its process
[HttpGet]
[ResponseType(typeof(FileDTO))]
public IQueryable<FileDTO> GetFiles(int procId)
{
return db.LeadProcessControl.Where(a => a.ProcessID == procId)
.Project().To<FileDTO>();
}
It returns JSON like this:
{
FileName = name.xlsx
FileID = 23
ProcessID = 11
Status = Started
}
This method is for GRID
File name | Status | Button to run import | Button to delete file
This Status is from table Logs and in FileDTO it placed the last value, for example if I upload file the status will be "File uploaded" when I run Import status will be "Started" and when it finished status will be "Complete". But now the page is locked when the import process is running, so the status always will be "Complete".
So I need to run procedure in background and GET method should return new Status if it has been changed. Any suggestions?

Adding async to a method don't make your method call asynchronous. It just indicate that thread that is handling the current request can be reused for processing other requests while waiting for some network/disk IO. When a client call this method it will only get response once method is complete. In other word async is completely sever side thing and nothing to do with the client call. You need to start your long running process in a separate thread as shown below. But best practice is not to use web app for such a long running processing instead do long processing in a separate windows service.
[HttpPost]
public async Task<IHttpActionResult> ImportFile(int procId, int fileId)
{
string importErrorMessage = String.Empty;
// Get excel file and read it
string path = FilePath(fileId);
DataTable table = GetTable(path);
// Add record for start process in table Logs
using (var transaction = db.Database.BeginTransaction()))
{
try
{
db.uspAddLog(fileId, "Process Started", "The process is started");
transaction.Commit();
}
catch (Exception e)
{
transaction.Rollback();
importErrorMessage = e.Message;
return BadRequest(importErrorMessage);
}
}
//Start long running process in new thread
Task.Factory.StartNew(()=>{
using (myHelper helper = new myHelper())
{
helper.StartImport(procId, fileId, table, ref importErrorMessage);
//** As this code is running background thread you cannot return anything here. You just need to store status in database.
//if (!String.IsNullOrEmpty(importErrorMessage))
//return BadRequest(importErrorMessage);
}
});
//You always return ok to indicate background process started
return Ok(true);
}