We Keep Coding

c# read and delete file used for messaging

I am writing a payments system which communicates with a very old POS interface through files.
The POS system tries to open/create a file (fixed to a single name). If it opens the file it then writes the payment request and closes the file, if it fails to open the file the POS system will wait and try again (sorry don't know how long between attempts, but very short).
I have to monitor the directory and when I see a new file arrive, then I have to try to open the file and block any access to the POS system from writing any new entries. While I have the file open I read through the records and send them to the payments system, once I receive confirmation I have to close the file and delete it.
My concern/problem is - how do I block the POS system from writing anything between the close and delete statements?
I can set the FileShare to Delete, which allows my program to delete the file before I close it, but this means other processes could also delete the file regardless of whether or not I processed the records successfully.
Option 1
void Main()
{
string filename = #"C:\Temp\CHARGES.TXT";
using (var fs = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.None))
{
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
}
File.Delete(filename);
}
Option 2
void Main()
{
string filename = #"C:\Temp\CHARGES.TXT";
using (var fs = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Delete))
{
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
File.Delete(filename);
}
}
As this is a payments system any loss of records is not acceptable. Also as I have no control over the POS system I cannot change the way it works.
Thank you in advance

Thank you to #Lasse and #Sam for your helpful comments. I have made the following changes to the code:
void Main()
{
string filename = #"C:\Temp\CHARGES.TXT";
int start = 0;
int end = 0;
using (var fs = new FileStream(filename, FileMode.Open, FileAccess.ReadWrite, FileShare.Read))
{
// get a big enough buffer to hold the whole file and read it in
long fileSize = new FileInfo(filename).Length;
var buffer = new byte[fileSize];
var bytesRead = fs.Read(buffer, 0, buffer.Length);
try
{
// convert each line to ASCII to process through to API
while (end < bytesRead)
{
while (end < bytesRead && buffer[end] != 0x0A && buffer[end] != 0x0D)
{
end++;
}
var text = ASCIIEncoding.ASCII.GetString(buffer, start, end - start);
// Send to API
Console.WriteLine("Start [{0}] End [{1}] Text [{2}]", start, end, text);
if (end < bytesRead)
{
end += 2;
start = end;
}
if (end > bytesRead / 2)
{
throw new NotImplementedException();
}
}
}
catch (Exception e)
{
Console.WriteLine("just to prove the partial");
}
finally
{
// convert any remaining records back to bytes and write back to file
fs.SetLength(0);
fs.Write(buffer, end, bytesRead - end);
}
}
}
The process will now go through the file and write back any failed/unprocessed messages to the file at the end - this covers off my locking concerns as well as covering the individual record failures.
Thank you agin

Fixed read-only folders

I am trying to copy or move folders and files in my C# application, but the folders are Read-Only, and they cannot be disabled, since when I try, it's enabled again. I tried many solutions, but not worked... Yes, I am administrator with all rights. I tried disable Read-only in script too, but don't work.
const int CopyBufferSize = 64 * 1024;
public void CopyFile(string source, string destination)
{
//File.Copy(source, destination);
//Stopwatch swTotal = Stopwatch.StartNew();
using (var outputFile = File.Create(destination))
{
using (var inputFile = File.OpenRead(source))
{
// we need two buffers so we can ping-pong
var buffer1 = new byte[CopyBufferSize];
var buffer2 = new byte[CopyBufferSize];
var inputBuffer = buffer1;
int bytesRead;
IAsyncResult writeResult = null;
while ((bytesRead = inputFile.Read(inputBuffer, 0, CopyBufferSize)) != 0)
{
// Wait for pending write
if (writeResult != null)
{
writeResult.AsyncWaitHandle.WaitOne();
outputFile.EndWrite(writeResult);
writeResult = null;
}
// Assign the output buffer
var outputBuffer = inputBuffer;
// and swap input buffers
inputBuffer = (inputBuffer == buffer1) ? buffer2 : buffer1;
// begin asynchronous write
writeResult = outputFile.BeginWrite(outputBuffer, 0, bytesRead, null, null);
}
if (writeResult != null)
{
writeResult.AsyncWaitHandle.WaitOne();
outputFile.EndWrite(writeResult);
}
}
}
//swTotal.Stop();
//Console.WriteLine("Total time: {0:N4} seconds.", swTotal.Elapsed.TotalSeconds);
}
I tried with
File.Copy(source, destination);
too.
Thank you.

more details..
var fileName = "sourceFile.txt";
var source = Path.Combine(Environment.CurrentDirectory, fileName);
var destination = Path.Combine(destinationFolder, fileName);
File.Copy(source, destination);
OR
File.Copy(#"someDirectory\someFile.txt", #"otherDirectory\someFile.txt");

Extract zip entries to another Zip file

Can anyone tell me why the following code doesnt work? I am using the SharpZipLib API for the Zip streams, latest version DL'ed today from their site. Im attempting to use this logic to merge the contents of one zip file into another, without having to perform IO on the disk, as the intended zip files may contain reserved file names for windows. I have tried this with multiple different source and destination zip files (those that contain reserved names and those that dont). The code does not throw any exception, and if you inspect the buffer prior to each write operation, you can see that it contains real data, yet after the entire operation finishes the size of the target zip file has not changed, and you can explore it to confirm that no new files (the ones the code is supposed to add) have actually been added to the destination file. :(
public static void CopyToZip(string inArchive, string outArchive)
{
ZipOutputStream outStream = null;
ZipInputStream inStream = null;
try
{
outStream = new ZipOutputStream(File.OpenWrite(outArchive));
outStream.IsStreamOwner = false;
inStream = new ZipInputStream(File.OpenRead(inArchive));
ZipEntry currentEntry = inStream.GetNextEntry();
while (currentEntry != null)
{
byte[] buffer = new byte[1024];
ZipEntry newEntry = new ZipEntry(currentEntry.Name);
newEntry.Size = currentEntry.Size;
newEntry.DateTime = currentEntry.DateTime;
outStream.PutNextEntry(newEntry);
int size = 0;
while ((size = inStream.Read(buffer, 0, buffer.Length)) > 0)
{
outStream.Write(buffer, 0, size);
}
outStream.CloseEntry();
currentEntry = inStream.GetNextEntry();
}
outStream.IsStreamOwner = true;
}
catch (Exception e)
{
throw e;
}
finally
{
try { outStream.Close(); }
catch (Exception ignore) { }
try { inStream.Close(); }
catch (Exception ignore) { }
}
}

I ended up doing this using a different API. DotNet zip from http://dotnetzip.codeplex.com/. Here is the implementation:
public static void CopyToZip(string inArchive, string outArchive, string tempPath)
{
ZipFile inZip = null;
ZipFile outZip = null;
try
{
inZip = new ZipFile(inArchive);
outZip = new ZipFile(outArchive);
List<string> tempNames = new List<string>();
List<string> originalNames = new List<string>();
int I = 0;
foreach (ZipEntry entry in inZip)
{
if (!entry.IsDirectory)
{
string tempName = Path.Combine(tempPath, "tmp.tmp");
string oldName = entry.FileName;
byte[] buffer = new byte[4026];
Stream inStream = null;
FileStream stream = null;
try
{
inStream = entry.OpenReader();
stream = new FileStream(tempName, FileMode.Create, FileAccess.ReadWrite);
int size = 0;
while ((size = inStream.Read(buffer, 0, buffer.Length)) > 0)
{
stream.Write(buffer, 0, size);
}
inStream.Close();
stream.Flush();
stream.Close();
inStream = new FileStream(tempName, FileMode.Open, FileAccess.Read);
outZip.AddEntry(oldName, inStream);
outZip.Save();
}
catch (Exception exe)
{
throw exe;
}
finally
{
try { inStream.Close(); }
catch (Exception ignore) { }
try { stream.Close(); }
catch (Exception ignore) { }
}
}
}
}
catch (Exception e)
{
throw e;
}
}

One issue that I see is that you are opening the output zip file using File.OpenWrite(), which will replace the existing output file rather than merging new entries into it.
There is an example on the SharpDevelop Wiki that gives an example of updating a zip file using memory streams. It can be found at http://wiki.sharpdevelop.net/SharpZipLib_Updating.ashx#Updating_a_zip_file_in_memory_1

The following is some simpler code that will read from the input zip and write to the output zip, which potentially already exists. It does not require writing temporary data to the filesystem.
public static void CopyToZip(string inArchive, string outArchive)
{
using (inZip = new ZipFile(inArchive),
outZip = new ZipFile(outArchive))
{
Func<String,Func<String,Stream>> getInStreamReturner = (name) => {
return new Func<String,Stream>(a){ return inZip[a].OpenReader(); };
};
foreach (ZipEntry entry in inZip)
{
if (!entry.IsDirectory)
{
string zipEntryName = entry.FileName;
outZip.AddEntry(zipEntryName,
getInStreamReturner(zipEntryName),
(name, stream) => stream.Close() );
}
}
outZip.Save();
}
}
Notes:
This approach uses the ZipFile.AddEntry overload that accepts two delegates: an opener and a closer. These functions get called at the time of ZipFile.Save. The former delegate needs to open and return the stream that contains the data to be zipped. The latter delegate needs to just close the stream.
It is necessary to define the getInStreamReturner Func , in order to open the right stream at the time of ZipFile.Save. Bear in mind that the zipEntryName changes value each time through the loop. Also ZipEntry.OpenReader() opens a stream on the actual zip data, which reads-and-decompresses as it goes. You can have only one of those open, at any one time, per ZipFile. getInStreamReturner creates a new function each time through the loop, thereby creating a closure to retain the value of the zipEntryName for reference at the time of ZipFile.Save.
This approach will fail if there are name clashes between the inArchive and outArchive. To avoid that you'd need to check for that and somehow avoid it. Either contrive a new, unique name, or skip adding entries with duplicate names into the outarchive.
I haven't tested this.
While this approach does not write to the filesystem, it does decompress and recompress file data. There is an open request to provide a feature to DotNetZip to migrate entries without that decompress/recompress jump. I haven't implemented that yet.

using WebClient to read and save large files in WP7

I am developing a music player in wp7 using silverlight.My end users don't have good internet connection when they are roaming. I want to give them an option to download music in their phone.
For that i have written a download manager which they can use to download music when they have wifi connection.
I ussue i am facing is that,My music is stored on my server and i am downloading the music on the phone using WebCLient class.
if size of file goes beyond 68 MB , my application gets OutOfMemory exception.
here is the code:
public void DownloadKirtan(KirtanViewModel kirtanVm,bool QueueItem=true)
{
{
if (kirtanVm.LocationPath != null)
{
WebClient webClient = new WebClient();
//webClient.AllowWriteStreamBuffering = false;
// webClient.AllowReadStreamBuffering = false;
if (QueueItem == false)
{
//App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload[kirtanVm] = webClient;
// App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.Add(kirtanVm//
webClient = App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload[kirtanVm];
kirtanVm.IsDownloadedForOfflineViewing = "Started";
webClient.OpenReadCompleted += new OpenReadCompletedEventHandler(webClient_OpenReadCompleted);
webClient.OpenReadAsync(kirtanVm.LocationPath, kirtanVm);
}
else if (!App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.ContainsKey(kirtanVm))
{
App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.Add(kirtanVm, webClient);
kirtanVm.IsDownloadedForOfflineViewing = "Started";
webClient.OpenReadCompleted += new OpenReadCompletedEventHandler(webClient_OpenReadCompleted);
webClient.OpenReadAsync(kirtanVm.LocationPath, kirtanVm);
}
// webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(webClient_DownloadProgressChanged);
}
}
}
void webClient_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
KirtanViewModel kirtanVm = e.UserState as KirtanViewModel;
try
{
if (e.Cancelled == false)
{
if (e.Result != null)
{
((WebClient)sender).OpenReadCompleted -= webClient_OpenReadCompleted;
#region Isolated Storage Copy Code
IsolatedStorageFile isolatedStorageFile = IsolatedStorageFile.GetUserStoreForApplication();
bool checkQuotaIncrease = IncreaseIsolatedStorageSpace(e.Result.Length);
if (checkQuotaIncrease)
{
string VideoFile = "";
VideoFile = GetUrlOfOfflineContent(kirtanVm);
using (IsolatedStorageFileStream isolatedStorageFileStream = new IsolatedStorageFileStream(VideoFile, FileMode.Create, isolatedStorageFile))
{
long VideoFileLength = (long)e.Result.Length;
byte[] byteImage = new byte[VideoFileLength];
e.Result.Read(byteImage, 0, byteImage.Length);
isolatedStorageFileStream.Write(byteImage, 0, byteImage.Length);
kirtanVm.IsDownloadedForOfflineViewing = "True";
kirtanVm.DownloadSize = "Size=" + (VideoFileLength / 1000000).ToString() + " MB";
kirtanVm.DownloadProgress = "100%";
Settings.OfflineKirtanContents.Value.Add(kirtanVm);
AddRemoveKirtanInOfflineModels(kirtanVm, true);
}
#endregion
}
else
{
kirtanVm.IsDownloadedForOfflineViewing = "False";
App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.Remove(kirtanVm);
MessageBox.Show("There is not enough space in your phone to store this media. You need " + e.Result.Length + " bytes of storage.Please free some offline contents you have downloaded by going to Offline content managemnt screen in Settings Section and try again or increase the storage on your phone.");
}
// mediaFile.SetSource(isolatedStorageFileStream);
// mediaFile.Play();
// progressMedia.Visibility = Visibility.Collapsed;
}
}
}
catch (Exception ex)
{
kirtanVm.IsDownloadedForOfflineViewing = "Failed";
//App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.Remove(kirtanVm);
MessageBox.Show(ex.ToString());
}
}
The problem i am having is When i get e.result which is a stream object. To write to isolatedStoreFileStream, i have to read it again to byte array and then save it to isolatedstoragefile. This is inefficient, it consumes double memory. Memory of 60 MB (for 60MB file) by WebClient e.result stream object and then 60 MB to convert to array. So memory consumtion is 128 MB . Is there is better way to download big files and store it to IsolatedStorage>
UPDATE : I am now using following code using the chunck size , instead of reading all in memory but i still get out of memory error on large files 100 MB
void webClient_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
KirtanViewModel kirtanVm = e.UserState as KirtanViewModel;
try
{
if (e.Cancelled == false)
{
if (e.Result != null)
{
((WebClient)sender).OpenReadCompleted -= webClient_OpenReadCompleted;
#region Isolated Storage Copy Code
bool checkQuotaIncrease = IncreaseIsolatedStorageSpace(e.Result.Length);
if (checkQuotaIncrease)
{
string VideoFile = "";
VideoFile = GetUrlOfOfflineContent(kirtanVm);
ThreadPool.QueueUserWorkItem( k =>{
using (IsolatedStorageFile isolatedStorageFile = IsolatedStorageFile.GetUserStoreForApplication())
{
using (IsolatedStorageFileStream isolatedStorageFileStream = new IsolatedStorageFileStream(VideoFile, FileMode.Create, isolatedStorageFile))
{
long VideoFileLength = (long)e.Result.Length;
using (BinaryWriter writer = new BinaryWriter(isolatedStorageFileStream))
{
Stream resourceStream = e.Result;//streamResourceInfo.Stream;
long length = resourceStream.Length;
byte[] buffer = new byte[32];
int readCount = 0;
using (BinaryReader reader = new BinaryReader(resourceStream))
{ // read file in chunks in order to reduce memory consumption and increase performance
while (readCount < length)
{
int actual = reader.Read(buffer, 0, buffer.Length);
readCount += actual;
writer.Write(buffer, 0, actual);
}
}
}
kirtanVm.IsDownloadedForOfflineViewing = "True";
kirtanVm.DownloadSize = "Size=" + (VideoFileLength / 1000000).ToString() + " MB";
kirtanVm.DownloadProgress = "100%";
Settings.OfflineKirtanContents.Value.Add(kirtanVm);
AddRemoveKirtanInOfflineModels(kirtanVm, true);
}
}
});
//byte[] byteImage = new byte[VideoFileLength];
//e.Result.Read(byteImage, 0, byteImage.Length);
// e.re
//ThreadPool.QueueUserWorkItem( k =>{
// isolatedStorageFileStream.Write(byteImage, 0, byteImage.Length);
//isolatedStorageFileStream.Close();
//Application.Current.RootVisual.Dispatcher.BeginInvoke( ()=>
// {
// kirtanVm.IsDownloadedForOfflineViewing = "True";
// kirtanVm.DownloadSize = "Size=" + (VideoFileLength / 1000000).ToString() + " MB";
// kirtanVm.DownloadProgress = "100%";
// Settings.OfflineKirtanContents.Value.Add(kirtanVm);
// AddRemoveKirtanInOfflineModels(kirtanVm, true);
// });
//});
//StreamWriter writer=new StreamWriter(isolatedStorageFileStream);
// writer.Write(e.Result);
// writer.Close();
// isolatedStorageFileStream.Write(
// e.Result.Write(
// isolatedStorageFileStream.Write(byteImage, 0, byteImage.Length);
// isolatedStorageFileStream.Close();
}
#endregion
else
{
kirtanVm.IsDownloadedForOfflineViewing = "False";
App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.Remove(kirtanVm);
MessageBox.Show("There is not enough space in your phone to store this media. You need " + e.Result.Length + " bytes of storage.Please free some offline contents you have downloaded by going to Offline content managemnt screen in Settings Section and try again or increase the storage on your phone.");
}
}
}
else
{
lock (App.ViewModel.LockForCancelForCurrentOfflineDownload)
{
if (App.ViewModel.cancelInitiatedByUserForCurrentOfflineDownload)
{
kirtanVm.IsDownloadedForOfflineViewing = "False";
// kirtanVm.IsOfflineDownloadCancelled = false;
App.ViewModel.cancelInitiatedByUserForCurrentOfflineDownload = false;
}
else
{
if (kirtanVm.IsDownloadedForOfflineViewing == "Started")// don't queue things again
{
kirtanVm.IsDownloadedForOfflineViewing = "Failed";
// bool ItemExist = App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.Any(k => k.Key.Kirtan.KirtanId == kirtanVm.Kirtan.KirtanId);
DownloadKirtan(kirtanVm, false);
}
}
}
}
}
catch (Exception ex)
{
kirtanVm.IsDownloadedForOfflineViewing = "Failed";
//App.ViewModel.ActiveInstancesOfWebClientForKirtanDownload.Remove(kirtanVm);
MessageBox.Show(ex.ToString());
}
}
Thanks
Verinder

Finally i found the solution
turn on webClient.AllowReadStreamBuffering = false; in the above code which i shared and then in
void webClient_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e) {} method.. Remove all refrences to e.Result.Length. When using no buffering solution you cannot know the stream size in advance
long length = resourceStream.Length;
byte[] buffer = new byte[1024];
int readCount = 0;
using (BinaryReader reader = new BinaryReader(resourceStream)) { // read file in chunks in order to reduce memory consumption and increase performance
while (true)
{
int actual = reader.Read(buffer, 0, 1024);
if(actual==0)
{
break;
}
else
{
readCount += actual;
writer.Write(buffer, 0, actual);
}
}
}
Keep on reading the stream with 1024 bytes at a time unless you are done reading. This way WebClient will not buffer whole file in memory and max mem usage on your phone will be 1KB at a time. You can download Gigs of data like this. INcrease the buffer size to 1MB or 3 MB , if you want to read faster.
Make sure you run this code in ThreadPool instead of user thread

You need to write it one chunk at a time by reading into a 4KB (or so) array and writing that to isolated storage until you run out of data.

how to split a wmv file

im doing a application in which i split a wmv file and transfer it to otherlocation(in 'x' kbs) .after the transfer gets completed the file doesnt play,it gives a message as the format is not supported.is there anyother way to do it.
sory i will explain what im doing now
i wrote an remote application,i want to transfer a .wmv file from one machine to other,i want to split the .wmv and send it to the remote machine and use it there.if i try to send the complete file means it will take lot of memory that seems very bad.so i want to split it and send it.but the file doesnt gets played it raises an exception the format is not supported.
the following is the code im doing i just done it in the local machine itself(not remoting):
try
{
FileStream fswrite = new FileStream("D:\\Movie.wmv", FileMode.Create);
int pointer = 1;
int bufferlength = 12488;
int RemainingLen = 0;
int AppLen = 0;
FileStream fst = new FileStream("E:\\Movie.wmv", FileMode.Open);
int TotalLen = (int)fst.Length;
fst.Close();
while (pointer != 0)
{
byte[] svid = new byte[bufferlength];
using (FileStream fst1 = new FileStream("E:\\Movie.wmv", FileMode.Open))
{
pointer = fst1.Read(svid, AppLen, bufferlength);
fst1.Close();
}
fswrite.Write(svid, 0, pointer);
AppLen += bufferlength;
RemainingLen = TotalLen-AppLen;
if(RemainingLen < bufferlength)
{
byte[] svid1 = new byte[RemainingLen];
using (FileStream fst2 = new FileStream("E:\\Movie.wmv", FileMode.Open))
{
pointer = fst2.Read(svid1, 0, RemainingLen);
fst2.Close();
}
fswrite.Write(svid, 0, pointer);
break;
}
}
fswrite.Close();
}
catch(Exception ex)
{
MessageBox.Show(ex.Message);
}

You'll probably find Good way to send a large file over a network in C# helpful.

Im going to make the assumtion your spliting the file when your sending it, and not trying to have the wmv in 3 different files on the remote machine.
When your sending the file what you basicly do is this:
Local machine
1) Read 16k bytes ( Or whatever number you prefere )
2) Send those 16k bytes over the network
3) Repeat above steps untill done
Remote machine
1) Listen for a connection
2) Get 16k bytes
3) Write 16k bytes
4) Repeat untill done.
This method will work, but your kind of inventing the wheel again, i would recommend using either something as simple as File.Copy ( Works fine over the network ) or if that does not meet your needs perhaps using a FTP client / server solution ( Plenty of C# examples on the net that can be hosted inside your application ).

i tried this
private void Splitinthread()
{
int bufferlength = 2048;
int pointer = 1;
int offset = 0;
int length = 0;
byte[] buff = new byte[2048];
FileStream fstwrite = new FileStream("D:\\TEST.wmv", FileMode.Create);
FileStream fst2 = new FileStream("E:\\karthi.wmv", FileMode.Open);
int Tot_Len = (int)fst2.Length;
int Remain_Buff = 0;
//Stream fst = File.OpenRead("E:\\karth.wmv");
while (pointer != 0)
{
try
{
fst2.Read(buff, 0, bufferlength);
fstwrite.Write(buff, 0, bufferlength);
offset += bufferlength;
Remain_Buff = Tot_Len - offset;
Fileprogress.Value = CalculateProgress(offset, Tot_Len);
if (Remain_Buff < bufferlength)
{
byte[] buff1 = new byte[Remain_Buff];
pointer = fst2.Read(buff1, 0, Remain_Buff);
fstwrite.Write(buff1, 0, pointer);
Fileprogress.Value = CalculateProgress(offset, Tot_Len);
fstwrite.Close();
fst2.Close();
break;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
MessageBox.Show("Completed");
}