We Keep Coding

c# parallel writes to Azure Data Lake File

In our Azure Data Lake, we have daily files recording events and coordinates for those events. We need to take these coordinates and lookup what State, County, Township, and Section these coordinates fall into. I've attempted several versions of this code.
I attempted to do this in U-SQL. I even uploaded a custom assembly that implemented Microsoft.SqlServer.Types.SqlGeography methods, only to find ADLA isn't set up to perform row-by-row operations like geocoding.
I pulled all the rows into SQL Server, converted the coordinates into a SQLGeography and built T-SQL code that would perform the State, County, etc. lookups. After much optimization, I got this process down to ~700ms / row. (with 133M rows in the backlog and ~16k rows added daily we're looking at nearly 3 years to catch up. So I parallelized the T-SQL, things got better, but not enough.
I took the T-SQL code, and built the process as a console application, since the SqlGeography library is actually a .Net library, not a native SQL Server product. I was able to get single threaded processing down t0 ~ 500ms. Adding in .Net's parallelism (parallel.ForEach) and throwing 10/20 of the cores of my machine at it does a lot, but still isn't enough.
I attempted to rewrite this code as an Azure Function and processing files in the data lake file-by-file. Most of the files timed out, since they took longer than 10 minutes to process. So I've updated the code to read in the files, and shread the rows into Azure Queue storage. Then I have a second Azure function that fires for each row in the queue. The idea is, Azure Functions can scale out far greater than any single machine can.
And this is where I'm stuck. I can't reliably write rows to files in ADLS. Here is the code as I have it now.
public static void WriteGeocodedOutput(string Contents, String outputFileName, ILogger log) {
AdlsClient client = AdlsClient.CreateClient(ADlSAccountName, adlCreds);
//if the file doesn't exist write the header first
try {
if (!client.CheckExists(outputFileName)) {
using (var stream = client.CreateFile(outputFileName, IfExists.Fail)) {
byte[] headerByteArray = Encoding.UTF8.GetBytes("EventDate, Longitude, Latitude, RadarSiteID, CellID, RangeNauticalMiles, Azimuth, SevereProbability, Probability, MaxSizeinInchesInUS, StateCode, CountyCode, TownshipCode, RangeCode\r\n");
//stream.Write(headerByteArray, 0, headerByteArray.Length);
client.ConcurrentAppend(outputFileName, true, headerByteArray, 0, headerByteArray.Length);
}
}
} catch (Exception e) {
log.LogInformation("multiple attempts to create the file. Ignoring this error, since the file was created.");
}
//the write the data
byte[] textByteArray = Encoding.UTF8.GetBytes(Contents);
for (int attempt = 0; attempt < 5; attempt++) {
try {
log.LogInformation("prior to write, the outputfile size is: " + client.GetDirectoryEntry(outputFileName).Length);
var offset = client.GetDirectoryEntry(outputFileName).Length;
client.ConcurrentAppend(outputFileName, false, textByteArray, 0, textByteArray.Length);
log.LogInformation("AFTER write, the outputfile size is: " + client.GetDirectoryEntry(outputFileName).Length);
//if successful, stop trying to write this row
attempt = 6;
}
catch (Exception e){
log.LogInformation($"exception on adls write: {e}");
}
Random rnd = new Random();
Thread.Sleep(rnd.Next(attempt * 60));
}
}
The file will be created when it needs to be, but I do get several messages in my log that several threads tried to create it. I'm not always getting the header row written.
I also no longer get any data rows only:
"BadRequest ( IllegalArgumentException concurrentappend failed with error 0xffffffff83090a6f
(Bad request. The target file does not support this particular type of append operation.
If the concurrent append operation has been used with this file in the past, you need to append to this file using the concurrent append operation.
If the append operation with offset has been used in the past, you need to append to this file using the append operation with offset.
On the same file, it is not possible to use both of these operations.). []
I feel like I'm missing some fundamental design idea here. The code should try to write a row into a file. If the file doesn't yet exist, create it and put the header row in. Then, put in the row.
What's the best-practice way to accomplish this kind of write scenario?
Any other suggestions of how to handle this kind of parallel-write workload in ADLS?

I am a bit late to this but I guess one of the problems could be due to the use of "Create" and "ConcurrentAppend" on the same file stream?
ADLS documentation mentions that they can't be used on the same file. Maybe, try changing the "Create" command to "ConcurrentAppend" as the latter can be used to create a file if it doesn't exist.
Also, if you found a better way to do it, please do post your solution here.

Corrupt ZIP file if calling Save twice

I am using DotNetZip 1.9.6 in my application which uses a file structure similar to e.g. *.docx: Zip file containing XML files.
Now every module of the application can store such XML files to my custom file management and on "save" they are serialized to streams which are then saved to the Zip file via DotNetZip.
To update the entries I use ZipFile.UpdateEntry(path, stream).
This works fine and the first time I save my file via calling ZipFile.Save() everything works.
But If I do this a second time (first some UpdateEntrycalls then Save) on the same instance the Zip file is corrupted: The file structure and meta-data (e.g. uncompressed size of each file) is still there, but all files are 0 byte in compressed size.
If I create a new instance from the just saved file after saving everything works fine, but shouldn't it be possible to avoid that and "reuse" the same instance?
The following example (also see https://dotnetfiddle.net/mHxEIy) can be used to reproduce the problem:
using System.IO;
using System.Text;
public class Program
{
public static void Main()
{
var zipFile = new Ionic.Zip.ZipFile();
var content1 = new MemoryStream(Encoding.Default.GetBytes("Content 1"));
zipFile.UpdateEntry("test.txt", content1);
zipFile.Save("test.zip"); // here the Zip file is correct
//zipFile = new Ionic.Zip.ZipFile("test.zip"); // uncomment and it works too
var content2 = new MemoryStream(Encoding.Default.GetBytes("Content 2"));
zipFile.UpdateEntry("test.txt", content2);
zipFile.Save(); // after that it is corrupt
}
}
To run this you need to add the "DotNetZip 1.9.6" NuGet package.
After the first save, this is what you get:
and after the second save:

This looks like it's a bug in the library, around removing an entry. If you just remove an entry and then save again, it correctly removes the file.
However, if you remove an entry and then add another one with the same name - which is what UpdateEntry is documented to do if the entry already exists - the old entry appears to be used instead.
The reason you're ending up with an empty file the second time is that the original MemoryStream is being read again - but by now, it's positioned at the end of the data, so there's no data to read. If you reset the position to the start of the stream (content1.Position = 0;) it will rewrite the original data. If you modify the data within content1, you end up with invalid compressed data.
The only workaround I can immediately think of is to keep your own map from filename to MemoryStream, and replace the contents of each MemoryStream when you want to update it... or just load the file each time, as per your existing workaround.
It's definitely worth filing a bug around this though, as it should work as far as I can tell.

As already suspected this was a bug in DotNetZip up to version 1.9.6.
I think I was able to fix this with THIS change which was just released as version 1.9.7 on NuGet. At least for me the problem does not happen anymore.
Some background what happend as far as I found out:
When you call Save the library sets an internal flag which remembers that the ZIP file was just save and on the second Save call instead of "recompressing" all entries in the ZIP file it copies them from the just saved file.
This works fine for adding/removing entries, but breaks when one of the entries was changed as then it "mixes" the old and the new entry and produces the inconsisten ZIP file.
My fix basically disables that "copy from old file" logic if an entry was changed.

dont want to overwrite my ini file

I am using ini file to store my configuration in my c# gui.
but when i start my gui again , and save the configuration, the previous saved configuration gets overwritten.
IS there a way to keep on saving configurations ?

You want to use an app.config file instead of your .ini. You access the settings in it using the ConfigurationManager from the System.Configuration namespace. You can even create custom configuration sections by creating classes that inherit from ConfigurationSection. That will give you intellisense support of your config file, as well.
One example of that (it's using asp.net, but it works for any .net code) is here.
Edit: Re-reading your question, I'm unclear on if you're trying to save application settings (app.config), or if you're trying to save session data to disk (saving records or serializing objects). If the former, look at app.config. You can even have multiple items that set the same "settings" but with different values (such as having multiple SQL Connection strings) and then call them by some parameter you obtain from a user.
If you're trying to save session data/state, then you want to serialize your objects- look into serialization/deserialization (many options available there) and the System.IO namespace for persisting to disk.

The only way to prevent overwriting the same file each time is to make the file name unique e.g.
FileName<TimeStamp>.ini
Or
FileName<Guid>.ini
Or you could even do what windows does with duplicate files and check how many already exist and append a new number onto the end e.g.
FileName.ini
FileName1.ini
FileName2.ini
Personally I would go with the timestamp/GUID approach. Here's some example code
class Program
{
static void Main()
{
for (int i = 0; i <= 10; i++)
{
SaveConfiguration();
Thread.Sleep(500);
}
}
private static void SaveConfiguration()
{
string fileName = System.IO.Path.Combine(#"Config\File\Dir", String.Format("Config{0:yyyyMMddHHmmss}.ini", DateTime.UtcNow));
System.IO.File.WriteAllText(fileName, "File contents");
}
}

How to read file in C# from POST data from web

Basically, I'm building a website that allows user to upload file.
From the front end (JavaScript), the user will browse a file, I can get the site to send POST data (the parameter "UploadInput" and it's value, which the value is the file)
In the backend (C#), I want to make a copy of the file and save it in a specific path.
Below is the way I did it.
var files = Request.Files;
file[0].SaveAs("\temp\\" + file[0].FileName);
The problem I ran into is that I got the error message saying index out of range. I tried Response.Write(files.Count) and it gives me 0 instead of 1.
I'm wondering where I did wrong and how to fix it, or if there's a better way of doing it.
Thanks!
Edit:
I am using HttpFox to debug. From HttpFox, I can see that under POST data, parameter is "UploadInput" and the value is "test.txt"
Edit 2:
So I tried the way Marc provides, and I have a different problem.
I am able to create a new file, however, the content is not copied over. I tried opening the new created file in notepad and all it says is "UploadInput = test.txt"

If they simply posted the file as the body content, then there will be zero "files" involved here, so file[0] will fail. Instead, you need to look at the input-stream, and simply read from that stream. For example:
using(var file = File.Create(somePath)) {
Request.InputStream.CopyTo(file);
}

OPEN a Resource.resx file instead of Creating it which overrides the previous Resource.resx file

I start my application from withint Visual Studio 2010.
I add then some files into my application and each file type`s icon like icon from doc,docx,xls,pdf etc are added as String/Bitmap key/value pair to my IconImages.Resx file via
private void DumpTempResourceToRealResourceFile(IDictionary<String, Bitmap> tempResource)
{
using (ResXResourceWriter writer = new ResXResourceWriter("IconImages.Resx"))
{
foreach (KeyValuePair<String,Bitmap> item in tempResource)
{
writer.AddResource(item.Key, item.Value);
}
writer.Generate();
}
}
When the icons are added to the resource I close the application.
Then I start my application again with VS 2010 and add some files within my document application. The file types are written again to my IconImages.Resx.
Then I close my application and check the IconImages.Resx file under the \bin\ folder and the previous saved images are gone and I have new/different ones now.
Why can I not say OPEN a .resx file and append stuff to it? Everytime I create a ResourceWriter object with the same name "IconImages.Resx" I overwrite the previous added stuff and thats stupid.
How can my IconImages.Resx file stay alive over an application session without being overwritten by other stuff I add?

I haven't used ResXResourceWriter, but usually *Writer classes simply write a data file from scratch.
If you want to "append" new data you would typically have to use a *Reader class to deserialise the existing data into memory, then merge/add in any new data you wish to, and use a *Writer object to then write the resulting data back out. Take a look at ResXResourceReader to see if it supports what you need to do this.

I am having now a lookup table "FiletypeImage" with the filetype ".docx" and the raw binary data aka blob. This table gets retrieved in my documentService and cached in a static variable. with a Get and Add method which are called by my DocumentListViewModel. Its very fast thx to sqlite :)