Is there any way to write to files faster? (windows) - c#

I make an application that must be able to process and store a lot of measurements.
There can be up to 129600 measurements and every measurement can have 1499 values.
All data is stored in an array, each column is a measurement.
After 10 measurements I save the data from the array to a file. Because if the application crashes, then i haven't lost all data.
In the beginning of the measurements, saving to the file only takes a few millieseconds. But when i am at measurement 200, it already takes 13 seconds. The time to save to the file keeps increasing. This makes sense, of course, because the array is getting bigger.
I can choose to only save every 50 or 100 measurements. But saving to the file will still take a long time.
Below you can see what the data in the file looks like:
Frequency(Hz);S21(dB)_1;S21(dB)_2
10000000;-42.87726786;-35.66746585
79950000;-60.5887682;-63.55421833
149900000;-84.56555597;-74.36793049
219850000;-76.71335093;-80.68063652
289800000;-68.96360042;-68.41861962
359750000;-71.24272337;-74.90287556
429700000;-78.65528362;-75.50614099
...
...
13860100000;-85.80839142;-84.03051407
13930050000;-79.31238317;-82.87468675
14000000000;-88.9003575;-75.48071161
Is there a possibility to work in a different way, In such way that it takes less time to save?
The code below converts the array to a large string.
measurementArrayToSave is a 2D array where each measurement data is added.
private string StringToSave() // Create a large string from the array.
{
// Create a file to write to.
string writeToFile = "";
writeToFile = writeToFile + "Frequency(Hz)"; // Title is depending on the number of measurements.
for (int titles = 0; titles < (numberOfMeasurements); titles++)
{
writeToFile = writeToFile + ";S21(dB)_" + (titles + 1).ToString() + " ";
}
writeToFile = writeToFile + ";Average" + "\n";
for (int i = 0; i < (Points + 1); i++) // write array with measurements to string
{
int column = 0;
if (i == Points)
{
writeToFile = writeToFile + "Average"; // Set name "Average" on the last line
column = 1;
}
for (; column < (numberOfMeasurements + 2); column++)
{
if (column == 0)
{
writeToFile = writeToFile + measurementArrayToSave[column, i].ToString().Replace(",", ".");
}
else
{
writeToFile = writeToFile + ";" + measurementArrayToSave[column, i].ToString().Replace(",", ".");
}
}
writeToFile = writeToFile + "\n";
}
return writeToFile;
}
The code below save the string to a file.
private void SaveMeasurementStirrerAutomatic() // automatic save without dialogBox
{
// Full name will depend on date and time.
string path = #"..\\measurementFiles\\" + textBoxName.Text +"_"+ thisDay.ToString("yyyy_MM_dd_HH_mm_ss") + ".txt";
if (File.Exists(path))
{
File.Delete(path);
}
string writeToFile = StringToSave(); // call StringToSave()
File.WriteAllText(path, writeToFile); // write string to file.
labelStatus.Invoke((MethodInvoker)delegate { labelStatus.Text = "Data is saved!"; });
}

Take a look at File.AppendText method:
// This text is always added, making the file longer over time
// if it is not deleted.
using (StreamWriter sw = File.AppendText(path))
{
sw.WriteLine("This");
sw.WriteLine("is Extra");
sw.WriteLine("Text");
}

#techgirl2000 you are probably using the entire history on every single "Save" operation.
You better use that in-memory array as a temporary buffer. i.e. the data in-memory should be "appended" to the file, AND cleared once the save is successful.
private void SaveMeasurementStirrerAutomatic() // automatic save without dialogBox
{
// Full name will depend on date and time.
string path = #"..\\measurementFiles\\" + textBoxName.Text +"_"+ thisDay.ToString("yyyy_MM_dd_HH_mm_ss") + ".txt";
// Create the file, only if needed
if (!File.Exists(path))
{
string headers = HeadersToSave(); // Get the first line (Headers)
File.WriteAllText(path, headers); // write string to file.
}
// append new lines
string writeToFile = StringToSave(); // call StringToSave()
using (StreamWriter sw = File.AppendText(path))
{
sw.Write(writeToFile); // not sure about this
}
//TODO: clear the array in memory
labelStatus.Invoke((MethodInvoker)delegate { labelStatus.Text = "Data is saved!"; });
}
Now you need to split the StringToSave() into another function with only the header line
private string HeadersToSave() // Create a large string from the array.
{
// Create a file to write to.
string writeToFile = "Frequency(Hz)"; // Title is depending on the number of measurements.
for (int titles = 0; titles < (numberOfMeasurements); titles++)
{
writeToFile = writeToFile + ";S21(dB)_" + (titles + 1).ToString() + " ";
}
writeToFile = writeToFile + ";Average" + "\n";
return writeToFile;
}
// ...
If you can't clear the data, then use a marker on every row to confirm if that specific line was appended.

Related

C# ASP.Net: Read & Rewrite File based on session variables

I am working on a website ATM school project, and I need to be able to rewrite the account balance in the original .txt file based on session transactions. The .txt file contains 3 lines, each with (username),(password),(balance). The issue I'm having is that the program cannot write to file while it is still reading it. The loop works fine if I write to a different file, but I have to edit the original (so the updated balance is retained the next time the program is run). Below is the code from the Logout page load event.
//Stream variable
StreamReader readFile;
StreamWriter writeFile;
//Counter variable
int index = 0;
//Open file
readFile = File.OpenText(#"C:\C#\Project4_TSullivan\loginFile.txt");
//Array rows
const int ROWS = 3;
while (index < ROWS && !readFile.EndOfStream)
{
string str = readFile.ReadLine();
string[] tokens = str.Split(',');
//Check if username matches session username
if (tokens[0] == Convert.ToString(Session["sessionUserName"]))
{
//Update balance
tokens[2] = Convert.ToString(Session["sessionBalance"]);
}
if (index == 0)
{
writeFile = File.CreateText(#"C:\C#\Project4_TSullivan\loginFile.txt");
writeFile.WriteLine(tokens[0] + "," + tokens[1] + "," + tokens[2]);
writeFile.Close();
}
else
{
writeFile = File.AppendText(#"C:\C#\Project4_TSullivan\loginFile.txt");
writeFile.WriteLine(tokens[0] + "," + tokens[1] + "," + tokens[2]);
writeFile.Close();
}
index++;
}
//Close file
readFile.Close();
Please let me know if any additional info would be helpful. I may be going about this in entirely the wrong way, and any advice would be much appreciated. Thanks!
Edited
Here is the solution in case the mods want to save it. Using File.ReadAllLines I was able to store the contents of the file in an array. Then I created 3 more arrays (1 for each line in the file) to tokenize each line.
string path = #"C:\C#\Project4_TSullivan\loginFile.txt";
string[] readFile = File.ReadAllLines(path);
string[] token0 = readFile[0].Split(',');
string[] token1 = readFile[1].Split(',');
string[] token2 = readFile[2].Split(',');
if (token0[0] == Convert.ToString(Session["sessionUsername"]))
{
token0[2] = Convert.ToString(Session["sessionBalance"]);
}
else if (token1[0] == Convert.ToString(Session["sessionUsername"]))
{
token1[2] = Convert.ToString(Session["sessionBalance"]);
}
else
{
token2[2] = Convert.ToString(Session["sessionBalance"]);
}
using (StreamWriter writeFile = File.CreateText(path))
{
writeFile.WriteLine(token0[0] + "," + token0[1] + "," + token0[2]);
writeFile.WriteLine(token1[0] + "," + token1[1] + "," + token1[2]);
writeFile.WriteLine(token2[0] + "," + token2[1] + "," + token2[2]);
}

How can I write data to text file one once is the same data ? And how to read back and assign?

if (GUILayout.Button("Copy settings"))
{
var selection = Selection.gameObjects.ToList();
for (int i = selection.Count - 1; i >= 0; --i)
{
var selected = selection[i];
WriteDataToFile("Transform " + i.ToString() + " Name ==> " + selected.name);
WriteDataToFile("************************" +
"********************");
if (selected.transform.parent != null)
WriteDataToFile(selected.transform.parent.ToString());
WriteDataToFile("local position " + selected.transform.localPosition.ToString());
WriteDataToFile("local rotation " + selected.transform.localRotation.ToString());
WriteDataToFile("local scale " + selected.transform.localScale.ToString());
WriteDataToFile("************************" +
"********************");
WriteDataToFile(" ");
}
}
And the WriteDataToFile:
private void WriteDataToFile(string line)
{
string path = "Assets/Resources/test.txt";
StreamWriter writer = new StreamWriter(path, true);
writer.WriteLine(line);
writer.Close();
}
First I want to check that if I click more then once on the button and it's the same data: name position rotation scale don't write again.
Second how can I read back the lines of the data and assign them into a object ? Also the name. So a new object will be created in the same name parent if the original was parent position rotation and scale.
This is how I'm reading now:
void ReadString()
{
string path = "Assets/Resources/test.txt";
StreamReader reader = new StreamReader(path);
Debug.Log(reader.ReadToEnd());
reader.Close();
}
To add new data to the file that doesn't already exist:
private void WriteDataToFile(string line)
{
string path = "Assets/Resources/test.txt";
string[] text = new string[];
if(File.Exists(path))
{
text = File.ReadAllLines(path);
if(!text.Contains(line))
File.AppendAllText(path, Line);
}
}
If you are not limited to using a text file for storing and retrieving data then I recommend finding a way to write all this data:
WriteDataToFile("Transform " + i.ToString() + " Name ==> " + selected.name);
WriteDataToFile("************************" +
"********************");
if (selected.transform.parent != null)
WriteDataToFile(selected.transform.parent.ToString());
WriteDataToFile("local position " + selected.transform.localPosition.ToString());
WriteDataToFile("local rotation " + selected.transform.localRotation.ToString());
WriteDataToFile("local scale " + selected.transform.localScale.ToString());
WriteDataToFile("************************" +
"********************");
WriteDataToFile(" ");
in less writes because opening and closing the file could be expensive. Maybe something like this:
var selected = selection[i].transform;
string toWrite = $"{parent}:{localPosition}:{localRotation}:{localScale}";
WriteDataToFile(toWrite);
This would mean retrieval would be simply - (not sure the type)
private gameObject GetObjectFromFile(Path, Id)
{
string[] text = new string[];
if(File.Exists(path))
{
text = File.ReadAllLines(path);
foreach(string s in text)
{
if(s.Split(':')[0] == Id.ToString())
{
text = s.Split(':');
break;
}
}
var Id = Convert.ToInt32(text[0]);
var localPosition = Convert.ToInt32(text[1]);
var localRotation = Convert.ToInt32(text[2]);
var localScale = Convert.ToInt32(text[3]);
return new gameObject(Id, localPosition, localRotation, localScale);
}

C# - How do I create a txt file in memory and then open it programatically?

I'm having a problem creating a txt file on C#. I am trying to create this file in memory (I don't want to create it in a physical path) and then open this file programmatically with the defaul app. The PC must detect the file extension (in this case .txt) and choose the right program to display the file (in this case maybe Notepad, Word, Wordpad...).
I got this now:
using (var writer = new StreamWriter("file.txt"))
{
writer.WriteLine(
grr[0].Keys.ToArray()[0] + "," + grr[0].Keys.ToArray()[1] + "," +
grr[0].Keys.ToArray()[2] + "," + grr[0].Keys.ToArray()[3]);
for (int r = 0; r < row - 1; r++)
{
writer.WriteLine(
grr[r].Values.ToArray()[0] + "," + grr[r].Values.ToArray()[1] + "," +
grr[r].Values.ToArray()[2] + "," + grr[r].Values.ToArray()[3]);
}
}
But I don't know how to open this file.
You want a file system in memory that contains filename and data. So use something like this:
public class MyFolder
{
string folderName { get; set;}
List<MyFolder> childFolders { get; set; }
Dictionary<string, List<byte>> files { get; set; }
}
Ok, the only solution I see is quite a hack:
Write the data on a file;
Open the file with the default app using Process.Start;
Delete the file with File.Delete.
Code:
// test data that I'll write on the file
var text = Enumerable.Range(0, 1000000).Select(x => x.ToString()).ToArray();
// choose the Desktop folder to verify that
// the file is deleted at the end of the method
var tempDir = Environment.GetFolderPath(Environment.SpecialFolder.Desktop);
// choose a random file name that will be unique 99.999999999% of the times
var filePath = Path.Combine(tempDir, Guid.NewGuid().ToString() + ".txt");
// write the file. Here I use WriteAllLines, you can use StreamWriter
File.WriteAllLines(filePath, text);
// start the default app for this path
Process.Start(filePath);
// wait to let the default app to open the file;
// otherwise the app will crash, not finding the file anymore
// just in the middle of the read
// (I put 2 sec, but the time must be verified
// depending on your system and the file size)
Thread.Sleep(2000);
// this will totally delete the file
File.Delete(filePath);
If you have Notepad as default app for txt files, that's what you'll see: Notepad opens up with your data, but the file doesn't exist anymore. That's quite what you wanted, isn't it? You won't find the file in the Recycle Bin neither, so you won't have disk space leaks.
The only defect of this trick is: if you click "Save" on your app, it won't ask you the path where you want to save the file. Instead, it will simply re-create the file as it was before deletion, and will save the data directly. That's because it opened a physical file, it didn't created a new one, so it remembers the filePath and will use it to save.
If you don't find more correct/professional solutions, this one can do its job.
ASIDE:
I'd suggest to you a little refactoring.
First step, avoid repetitions:
using (var writer = new StreamWriter("file.txt"))
{
var array = grr[0].Keys.ToArray();
writer.WriteLine(array[0] + "," + array[1] + "," + array[2] + "," + array[3]);
for (int r = 0; r < row - 1; r++)
{
var grrr = grr[r].Values.ToArray();
writer.WriteLine(grrr[0] + "," + grrr[1] + "," + grrr[2] + "," + grrr[3]);
}
}
Second step, use more advanced built-in functions:
using (var writer = new StreamWriter("file.txt"))
{
writer.WriteLine(string.Join(",", grr[0].Keys.ToArray()));
for (int r = 0; r < row - 1; r++)
{
writer.WriteLine(string.Join(",", grr[r].Values.ToArray()));
}
}

Changing text in a .txt file using c#

Basically im trying to save a new password and avatar for my twitter type website.
Any help would be appreciated
My coding is:
string newPasswordString = Server.MapPath("~") + "/App_Data/tuitterUsers.txt";
string[] newPasswordArray = File.ReadAllLines(newPasswordString);
string newString = Server.MapPath("~") + "/App_Data/tuitterUsers.txt";
newString = File.ReadAllText(newString);
string[] newArray = newString.Split(' ');
for (int i = 0; i < 3; i++)
{
for (int j = 0; j < 3; i++)
{
newArray[1] = newPasswordTextBox.Text;
newArray[2] = avatarDropDownList.SelectedValue;
newPasswordArray.Replace(" " + Session["Username"].ToString() + " " + Session["UserPassword"].ToString() + " " + Session["UserAvatarID"].ToString() + " ", " " + Session["Username"].ToString() + " " + newPasswordArray[1] + " " + newPasswordArray[2]);
}
}
string newPasswordString = string.Join(Environment.NewLine, newPasswordArray);
File.WriteAllText(Server.MapPath("~") + "/App_Data/tuitterUsers.txt", newPasswordString);
If I understand your problem correctly you need to move the
File.WriteAllText(Server.MapPath("~") + "/App_Data/tuitterUsers.txt", newPasswordArray);
outside the loop, otherwise you rewrite the file at each loop, but this is not enough, you need also to rebuild the Whole text file
string fileToWrite = string.Join(Environment.NewLine, newPasswordArray);
File.WriteAllText(Server.MapPath("~") + "/App_Data/tuitterUsers.txt", fileToWrite);
EDIT: After the code update and the comment below
The looping is totally wrong as well the rebuilding of the array
string userDataFile = Server.MapPath("~") + "/App_Data/tuitterUsers.txt";
string[] userDataArray = File.ReadAllLines(userDataFile);
for(int x = 0; x < userDataArray.Length; x++)
{
string[] info = userData[x].Split(' ');
if(Session["Username"].ToString() == info[0])
{
userData[x] = string.Join(" ", Session["UserName"].ToString(),
newPasswordTextBox.Text,
avatarDropDownList.SelectedValue.ToString());
break;
}
}
string fileToWrite = string.Join(Environment.NewLine, userDataArray);
File.WriteAllText(Server.MapPath("~") + "/App_Data/tuitterUsers.txt", fileToWrite);
Keep in mind that this works for a limited number of users.
If you are lucky and you site becomes the new Twitter, you cannot think to use a solution where you read in memory the names of all your users.
Firstly, what you're doing is A Bad Idea™. Given that a web server can have multiple threads in operation, you can't be certain that two threads aren't going to be writing different data at the same time. The more users you have the larger your user file will be, which means it takes longer to read and write the data, which makes it more likely that two threads will come into conflict.
This is why we use databases for things like this. Instead of operating on the whole file every time you want to read or write, you operate on a single record. There are plenty of other reasons to do it to.
That said, if you insist on using a text file...
If you treat each line in the file as a record - a single user's details in this case - then it makes sense to build a class to handle the content of those records, and make that class able to read and write the line format.
Something like this:
class UserRecord
{
public string Name;
public string Password;
public string Avatar;
public UserRecord(string name, string password, string avatar)
{
Name = name;
Password = password;
Avatar = avatar;
}
// static factory method
public static UserRecord Parse(string source)
{
if (string.IsNullOrEmpty(source))
return null;
string[] parts = source.Split(',');
if (parts.Length < 3)
return null;
return new UserRecord(parts[0], parts[1], parts[2]);
}
// convert to string
public string ToString()
{
return (new string[] { Name, Password, Avatar }).Join(",");
}
}
Adjust the Parse method to handle whatever format you're using for the data in the line, and change the ToString method to produce that format.
Once you have that working, use it to parse the contents of your file like this:
// Somewhere to put the data - a Dictionary is my first choice here
Dictionary<string, UserRecord> users = new Dictionary<string, UserRecord>();
// Don't forget to use 'using' where appropriate
using (TextReader userfile = File.OpenText(userDataFile))
{
string srcline;
while ((srcline = userfile.ReadLine()) != null)
{
UserRecord user = UserRecord.Parse(line);
if (user != null)
users[user.Name] = user;
}
}
Then you can access the user's data by username, manipulate it as required, and save it back out whenever you like.
Writing the data back out from a Dictionary of users is as simple as:
StringBuilder sb = new StringBuilder;
foreach (UserRecord user in users.Values)
{
sb.AppendFormat("{0}\n", user);
}
File.WriteAllText(userDataFile, sb.ToString());
Meanwhile, you have a users collection that you can save for future checks and manipulations.
I still think you should use a database though. They're not hard to learn and they are far better for this sort of thing.

c# service renaming files!

I have a windows service , that takes files with metadata(FIDEF) and corresponding video file and , translates the XML(FIDEF) using XSLT .
I get the file directory listing for FIDEF's and if a video file of the same name exists it translates it. That works ok , but it is on a timer to search every minute. I am trying to handle situations where the same file name enters the input directory but is already in the output directory. I just have it changing the output name to (copy) thus if another file enters i should get (copy)(copy).mov but the service won't start with filenames of the same directory already in the output , it works once and then does not seem to pick up any new files.
Any Help would be great as I have tried a few things with no good results. I believe its the renaming methods, but I've put most of the code up in case its a clean up issue or something else.
(forgive some of the names just trying different things).
private void getFileList()
{
//Get FILE LIST FROM Directory
try
{
// Process Each String/File In Directory
string result;
//string filename;
filepaths = null;
filepaths = Directory.GetFiles(path, Filetype);
foreach (string s in filepaths)
{
for (int i = 0; i < filepaths.Length; i++)
{
//Result Returns Video Name
result = Path.GetFileNameWithoutExtension(filepaths[i]);
FileInfo f = new FileInfo(filepaths[i]);
PreformTranslation(f, outputPath + result , result);
}
}
}
catch (Exception e)
{
EventLog.WriteEntry("Error " + e);
}
}
private void MoveVideoFiles(String Input, String Output)
{
File.Move(Input, Output);
}
private string GetUniqueName(string name)
{
//Original Filename
String ValidName = name;
//remove FIDEF from filename
String Justname1 = Path.GetFileNameWithoutExtension(name);
//get .mov extension
String Extension2 = Path.GetExtension(Justname1);
//get filename with NO extensions
String Justname = Path.GetFileNameWithoutExtension(Justname1);
//get .Fidef
String Extension = Path.GetExtension(name);
int cnt = 0;
//string[] FileName = Justname.Split('(');
//string Name = FileName[0];
while (File.Exists(ValidName)==true)
{
ValidName = outputPath + Justname + "(Copy)" + Extension2 + Extension;
cnt++;
}
return ValidName;
}
private string getMovFile(string name)
{
String ValidName = name;
String Ext = Path.GetExtension(name);
String JustName = Path.GetFileNameWithoutExtension(name);
while(File.Exists(ValidName))
{
ValidName = outputPath + JustName + "(Copy)" + Ext;
}
return ValidName;
}
//Preforms the translation requires XSL & FIDEF name.
private void PreformTranslation(FileInfo FileName, String OutputFileName , String result)
{
string FidefName = OutputFileName + ".FIDEF";
String CopyName;
String copyVidName = outputPath + result;
XslCompiledTransform myXslTransform;
myXslTransform = new XslCompiledTransform();
try
{
myXslTransform.Load(XSLname);
}
catch
{
EventLog.WriteEntry("Error in loading XSL");
}
try
{ //only process FIDEF's with corresponding Video file
if (AllFidef == "no")
{
//Check if video exists if yes,
if (File.Exists(path + result))
{
//Check for FIDEF File Already Existing in the Output Directory.
if (File.Exists(FidefName))
{
//Get unique name
CopyName = GetUniqueName(FidefName);
copyVidName= getMovFile(copyVidName);
//Translate and create new FIDEF.
//double checking the file is here
if (File.Exists(outputPath + result))
{
myXslTransform.Transform(FileName.ToString(), CopyName);
File.Delete(FileName.ToString());
MoveVideoFiles(path + result, copyVidName);
}
////Move Video file with Corresponding Name.
}
else
{ //If no duplicate file exsists in Directory just move.
myXslTransform.Transform(FileName.ToString(), OutputFileName + ".FIDEF");
MoveVideoFiles(path + result, outputPath + result);
}
}
}
else
{
//Must have FIDEF extension
//Processes All FIDEFS and moves any video files if found.
myXslTransform.Transform(FileName.ToString(), OutputFileName + ".FIDEF");
if (File.Exists(path + result))
{
MoveVideoFiles(path + result, outputPath + result);
}
}
}
catch (Exception e)
{
EventLog.WriteEntry("Error Transforming " + "FILENAME = " + FileName.ToString()
+ " OUTPUT_FILENAME = " + OutputFileName + "\r\n" +"\r\n"+ e);
}
}
There is a lot wrong with your code. getFileList has the unneeded inner for loop for starters. Get rid of it. Your foreach loop has s, which can replace filepaths[i] from your for loop. Also, don't do outputPath + result to make file paths. Use Path.Combine(outputPath, result) instead, since Path.Combine handles directory characters for you. Also, you need to come up with a better name for getFileList, since that is not what the method does at all. Do not make your method names liars.
I would simply get rid of MoveVideoFiles. The compiler just might too.
GetUniqueName only works if your file name is of the form name.mov.fidef, which I'm assuming it is. You really need better variable names though, otherwise it will be a maintenance nightware later on. I would get rid of the == true in the while loop condition, but that is optional. The assignment inside the while is why your files get overwritten. You always generate the same name (something(Copy).mov.fidef), and as far as I can see, if the file exists, I think you blow the stack looping forever. You need to fix that loop to generate a new name (and don't forget Path.Combine). Maybe something like this (note this is untested):
int copyCount = 0;
while (File.Exists(ValidName))
{
const string CopyName = "(Copy)";
string copyString = copyCount == 0 ? CopyName : (CopyName + "(" + copyCount + ")");
string tempName = Justname + copyString + Extension2 + Extension;
ValidName = Path.Combine(outputPath, tempName);
copyCount++;
}
This generates something(Copy).mov.fidef for the first copy, something(Copy)(2).mov.fidef for the second, and so on. Maybe not what you want, but you can make adjustments.
At this point you have a lot to do. getMovFile looks as though it could use work in the same manner as GetUniqueName. You'll figure it out. Good luck.

Categories