We Keep Coding

"File being used" error causes searching for specific text to fail

I am listing files in a given folder (log files) and allow downloading the file as well as searching for a given text in all files. Search is not working.
I tried to download the files and noticed for today's files I get "Error : The process cannot access the file 'Z:\abcd.log' because it is being used by another process.". I contacted developer who generates this log file and was told that the log file for today is kept open until midnight (he is not doing open/write/close).
In my download-file code, I was using:
fStream = new FileStream(filepath, FileMode.Open, FileAccess.Read, FileShare.Read);
I changed it to:
fStream = new FileStream(filepath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
This fixed the download issue; but search still does not work and I need some help.
I am using ReadAllLines that by my understanding (I could be wrong) opens and closes the file.
public string SearchFiles(string SearchStr, string FolderName, string DaysPrior)
{
string JSONresult = string.Empty;
var dir = Server.UrlDecode(FolderName);
System.IO.DirectoryInfo di = new System.IO.DirectoryInfo(dir);
int iDaysPrior = 0;
if (!string.IsNullOrEmpty(DaysPrior))
iDaysPrior = int.Parse(DaysPrior);
var fileList = di.GetFiles().Where(x => x.LastWriteTime.Date >= DateTime.Today.AddDays(0 - iDaysPrior)).OrderByDescending(f => f.LastWriteTime);
foreach (System.IO.FileInfo fi in fileList)
{
foreach (var line in File.ReadAllLines(fi.FullName))
{
// Using custom extension method
if (line.Contains(SearchStr, StringComparison.CurrentCultureIgnoreCase))
{
// Do something
}
}
}
return JSONresult;
}
Solution See comments for the credit, solution provided by psubsee2003.
The issue is that ReadAllLines, at the end, uses StreamReader in FileAccess.Read mode. It cannot share a file with another application with Read/Write access to the file. The solution, according to the post (and it worked for me) was to write your own ReadAllLines. Following code is from the post, in case the post itelf disappears one day.
public string[] WriteSafeReadAllLines(String path)
{
using (var csv = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (var sr = new StreamReader(csv))
{
List<string> file = new List<string>();
while (!sr.EndOfStream)
{
file.Add(sr.ReadLine());
}
return file.ToArray();
}
}

C# Transform Data in ETL Process

I am learning the ETL process in C# and have already extracted and read the sample CSV data but am unsure at what to do to transform it properly.
I have been using this website as a reference at how to transform data, but I am unsure on how to apply it to my sample data (below).
name,gender,age,numKids,hasPet,petType
Carl,M,43,2,true,gecko
Jake,M,22,1,true,snake
Cindy,F,53,3,false,null
Matt,M,23,0,true,dog
Ally,F,28,1,false,null
Megan,F,42,2,false,null
Carly,F,34,4,true,cat
Neal,M,27,2,false,null
Tina,F,21,2,true,pig
Paul,M,1,3,true,chicken
Below is how I extracted the data from the CSV file using CSVHelper
using (FileStream fs = File.Open(#"C:\Users\Grant\Documents\SampleData4.csv", FileMode.Open, FileAccess.Read))
using (StreamReader sr = new StreamReader(fs))
{
CsvConfiguration csvConfig = new CsvConfiguration()
{BufferSize = bufferSize, AllowComments = true};
using (var csv = new CsvReader(sr, csvConfig))
{
while (csv.Read())
{
var name = csv.GetField<string>(0);
var gender = csv.GetField<string>(1);
var age = csv.GetField<int>(2);
var numKids = csv.GetField<int>(3);
var hasPet = csv.GetField<bool>(4);
var petType = csv.GetField<string>(5);
}
}
}
If you need me to provide additional details, just ask below.

Although a little late, I still would like to add an answer:
To create you own ETL process and Data Flow with C#, I would recommend you the nuget package ETLBox (https://etlbox.net). It will enable you to write a ETL data flow, where the CSV reader implementation is already wrapped in a CSVSource object. E.g., you would have to do the following to load data from a CSV into a database:
Defina a CSV source
CSVSource sourceOrderData = new CSVSource("demodata.csv");
Optionally define a row transformation:
RowTransformation<string[], Order> rowTrans = new RowTransformation<string[], Order>(
row => new Order(row)
);
Define the destination
DBDestination<Order> dest = new DBDestination<Order>("dbo.OrderTable");
Link your ETL data pipeline together
sourceOrderData.LinkTo(rowTrans);
rowTrans.LinkTo(dest);
Finally start the dataflow (async) and wait for all data to be loaded.
source.Execute();
dest.Wait();

How to decompress .zip files in c# without extracting to new location

How can I decompress (.zip) files without extracting to a new location in the .net framework? Specifically, I'm trying to read a filename.csv.zip into a DataTable.
I'm aware of .extractToDirectory (which is within ZipArchive) but I just want to extract it into an object in c# and I would like to not create a new file.
Hoping to be able to do this w/o third party libraries, but I'll take what I can get.

May be some bugs because I never tested this, but here you go:
List<byte[]> urmom = new List<byte[]>();
using (ZipArchive archive = ZipFile.OpenRead(zipPath))
foreach (ZipArchiveEntry entry in archive.Entries)
using (StreamReader r = new StreamReader(entry.Open()))
urmom.Add(r.ReadToEnd(entry));
Basically you use the ZipArchive's openread class to iterate through each entry. At this point, you can use the streamreader to read each entry. From there you can create a file from the stream and even read the filename if you want to. My code doesn't do this, a bit of laziness on my part.

Keep in mind that a compressed stream might contain multiple files. To resolve this is required to iterate through all entries of zip file in order to retrieve them and treat separately.
The sample bellow converts a sequence of bytes in a list of string where each one is the context of the files included in zipped folder:
public static IEnumerable<string> DecompressToEntriesTextContext(byte[] input)
{
var zipEntriesContext = new List<string>();
using (var compressedStream = new MemoryStream(input))
using (var zip = new ZipArchive(compressedStream, ZipArchiveMode.Read))
{
foreach(var entry in zip.Entries)
{
using (var entryStream = entry.Open())
using (var memoryEntryStream = new MemoryStream())
using (var reader = new StreamReader(memoryEntryStream))
{
entryStream.CopyTo(memoryEntryStream);
memoryEntryStream.Position = 0;
zipEntriesContext.Add(reader.ReadToEnd());
}
}
}
return zipEntriesContext;
}

Isolated Storage adding caracters at the end of a Stream

I'm having problems converting long into string.
What I'm doing is trying to save a DateTime.Now.Ticks property in isolatedStorage, then retrieve it afterwords. This is what I did to save it:
IsolatedStorageFile appStorage = IsolatedStorageFile.GetUserStoreForApplication();
using (var file = appStorage.CreateFile("appState"))
{
using (var sw = new StreamWriter(file))
{
sw.Write(DateTime.Now.Ticks);
}
}
When I retrieve the file, I do it like this:
if (appStorage.FileExists("appState"))
{
using (var file = appStorage.OpenFile("appState", FileMode.Open))
{
using (StreamReader sr = new StreamReader(file))
{
string s = sr.ReadToEnd();
}
}
appStorage.DeleteFile("appState");
}
Until here I have no problem, but when I try to convert the string I retrieved, the compiler throws a FormatExeption. This are the two ways I tried to do it with:
long time = long.Parse(s);
long time = (long)Convert.ToDouble(s);
So is there any other ways to so this?
EDIT:
The problem is not in the conversion but rather in the StreamWriter adding extra characters.

I suspect you are seeing some other data at the end. Something else may have written other data to the stream.
I think you should use StreamWriter.WriteLine() instead of StreamWriter.Write() to write the data and then call StreamReader.ReadLine() instead of StreamReader.ReadToEnd() to read it back in.

How to append data to a binary file?

I have a binary file to which I want to append a chunk of data at the end of the file, how can I achieve this using C# and .net? Also is there anything to consider when writing to the end of a binary file? Thanks a lot for your help.

private static void AppendData(string filename, int intData, string stringData, byte[] lotsOfData)
{
using (var fileStream = new FileStream(filename, FileMode.Append, FileAccess.Write, FileShare.None))
using (var bw = new BinaryWriter(fileStream))
{
bw.Write(intData);
bw.Write(stringData);
bw.Write(lotsOfData);
}
}

You should be able to do this via the Stream:
using (FileStream data = new FileStream(path, FileMode.Append))
{
data.Write(...);
}
As for considerations - the main one would be: does the underlying data format support append? Many don't, unless it is your own raw data, or text etc. A well-formed xml document doesn't support append (without considering the final end-element), for example. Nor will something like a Word document. Some do, however. So; is your data OK with this...

Using StreamWriter and referencing DotNetPerls, make sure to add the True boolean to the StreamWriter constructor, if otherwise left blank, it'll overwrite as usual:
using System.IO;
class Program
{
static void Main()
{
// 1: Write single line to new file
using (StreamWriter writer = new StreamWriter("C:\\log.txt", true))
{
writer.WriteLine("Important data line 1");
}
// 2: Append line to the file
using (StreamWriter writer = new StreamWriter("C:\\log.txt", true))
{
writer.WriteLine("Line 2");
}
}
}
Output
(File "log.txt" contains these lines.)
Important data line 1
Line 2
This is the solution that I was actually looking for when I got here from Google, although it wasn't a binary file though, hope it helps someone else.