I have an MVC web app where users upload a text file and I parse it out.
The requirement just changed and they will be uploading multiple files of the same kind now. I parse a single file by sending a file-path to the method below, ReadParts which opens a stream and calls the method parseReplicateBlock to retrieve desired fields. For multiple files I could read all the files into one big stream but I am afraid it could exceed the buffer limit etc.
So I am thinking to parse file by file and populate results into an object. My requirement then, is to sort the records based on a date field.
I just need some help in how to write this method ReadLogFile in a better way, espceially for sorting based on initialtionDate and initiationTime. I want to find the minimum record based on initiationDate and Time and then do some other logic.
The problem is if I sort the list member within the object, I would loose positiong of the other records.
You appear to be storing each field of the record in a separate collection within LogFile. This seems a very strange way to store your data.
If you sort one of these collections, then of course it will bear no relationship to the other fields any longer since they are unrelated. There are huge areas for bugs too if you are relying on all the collections tallying up (eg if a field is missing from one of the parsed records)
Instead you should be have a class that represents a SINGLE record, and then Logfile has a SINGLE collection of these records. eg:
public class ReplicateBlock
{
public string ReplicateId { get; set; }
public string AssayNumber { get; set; }
public DateTime InitiationDate { get; set; }
//etc
}
public class LogFile
{
public List<ReplicateBlock> ReplicateBlocks = new List<ReplicateBlock>();
}
I have to say that your code is very difficult to follow. The fact that all your functions are static makes me think that you're not particularly familiar with object oriented programming. I would suggest getting a good book on the subject.
Related
I have created a class to store data from API calls I am making. It returns JSON with some meta information, and then an array of data depending on the call being made. The meta information will always have the same fields, so I have created a "Root" class for this, but the data will be different depending on the call being made, so I have created different classes for each type of data, e.g. user data, company data, etc. As shown below, I currently have the "data" property set to a list of objects, but I am trying to figure out the best way to incorporate the different types of data that can be returned, since it will vary based on the call being made.
Right now I have the data saved as a list of objects, but I would like this to change depending on what data I am receiving. Like, if I am retrieving users, I would like for it to be a list of users.
What is the ideal way to accommodate for this? The only way I can think to do it now is to create a different "Root" class for every type of data I am expecting to receive, but that doesn't feel like it should be the most concise way to do it. I was looking into making this a factory design pattern but I wasn't sure that it fit this scenario.
Just use a generic base class:
public abstract class ApiCallResult<T>
{
// With your properties
// public int Limit { get; set; }
// [...]
//
public IEnumerable<T> Data { get; set; }
}
Then define a result per api call.
public class UserApiCallResult : ApiCallResult<User>
{
}
Created a small working example here:
dotnet fiddle
I'm implementing an import functionality, wherein I need to read the data from the user uploaded CSV or Excel file and run a few validations and sanitize the data before writing the data to DB.
I'm able to get the data from the file to a list of objects, with the following structure:
public class Order
{
public string Sku { get; set; }
public decimal Cost { get; set; }
public DateTime OrderFulfillmentStartDate { get; set; }
public DateTime OrderFulfillmentEndDate { get; set; }
public string ValidationErrors{ get; set; }
}
Following are the validations that need to happen within the objects in the list, a few examples below:
No two orders with the same SKU and OrderFulfillmentStartDate, OrderFulfillmentEndDate are allowed.
No two orders with the same SKU and overlapping OrderFulfillmentStartDate, OrderFulfillmentEndDate are allowed.
etc.
The way I implemented it:
During the first encounter of a distinct record (passing all the validations AND "ValidationErrors" == string.empty), I'm adding the record to a temp list.
During the next iteration, I'm validating the currently processing record with the record present in the temp list, if the validation fails, I'm populating the "ValidationErrors" field and adding the temp list.
E.g:
Now coming to the crux of the issue:
The size of the data can be around One Million rows.
When I implemented the validations sequentially using foreach loop, the validation process is taking more than 8 hours.
Having said that, I believe doing the validation in parallel would cut down the time needed drastically.
I did try implementing the logic using Parallel.ForEach and Partitioner concept. The processing did quicken up, but I'm not sure, how I can keep a temp list that can be used/updated by multiple threads in the ForEach loop, during the validation.
Is there is a better or quicker approach to achieve what I'm trying to do here? Please do let me know about it.
Thank You!
I need a bit of advice/opinions on storing data.
Im looking at essentially storing user input data and then reference it on a frequent basis. I would like to avoid data base storage as this is all going to be referenced locally. Ive seen that you can do XML serialised objects etc and wanted to know what you think about data storage.
If i give you an idea of what im looking at doing in a step by step it may give a better idea of what im trying to achieve:
User loads the program
Program reads the storage file
Different controls such as GridView and text boxes displays specific parts of the document such as Names, Dates etc.
User can add to the file by entering data into different fields across the whole sheet.
User is free to save the file when required, can create new files when required and can open sheets at any point
All information needs to scale. i.e. user can add up to 50 dates, 30 names etc and each of these cannot interfere with another data field. All of this data for each specific type will be able to be referenced by the ui controls
Its a bit of a big task and i was looking at doing it all using XML. Im not sure if this is the best way to do it and wanted your opinions. Mainly due to the way the data can dynamically adjust all the time through user input.
Any input on this would be appreciated. I can provide images of what im trying to achieve if necessary.
I personally use Newtonsoft JSON for saving my user data. You can build all your classes as normal and then fairly easily integrate the JSON serialiser/deserialiser without too much hassle. There will be some exceptions that may require a custom serialiser, but it's not too difficult to make those, especially if you only need to interpret one small part and then can just pass the rest of the file off to the default serialiser.
This is a very quick example how you could store some key-value preferences for a number of users in a file.
using Newtonsoft.Json;
[JsonObject]
public class UserData {
[JsonProperty] //Add a JSON property "Username" which is a string
public string Username { get; set; }
//IEnumerable types are converted to/from arrays automatically by Newtonsoft
[JsonProperty("Options")] //Set the name in JSON file to "Options"
public Dictionary<string, string> Preferences { get; set; }
[JsonIgnore] //Excludes this property from the JSON output
public bool SaveRequired { get; set; } //Set true when a change is made, set false when saved
}
There are very similar libraries like this that do the exact same thing for XML, but I've not had much luck figuring them out and I'm usually on a very tight time scale when I just need something that I know works.
If you can find a decent XML library and understand how to use it correctly, I'd recommend XML over JSON due to its strictly structured nature, and you can include schema versions and aid with integration into other systems by providing a well written schema.
I have an issue with the project I'm working on. I'm using Entity Framework. Some quick background on the db model:
public class AssetType{
public ICollection<Field> Fields { get; set; }
}
public class Field{
public int Id {get;set;
public string Name {get;set;
}
Now I'm creating a view that would create a new Asset Type. As part of this process the user must also create all of the fields they want for that type. The issue is that I'm not sure how to represent the list of "Fields" on the page. The idea is that the user can add a new field, or remove one at any time with jQuery.
I can't figure how the data could be posted back to the server as part of the form. I thought about constructing the list in JSON form, but this seemed a bit messy. Has anyone got any better ideas?
You're going to have problems with this. The object parser does not handle complex objects very well. Collections usually need to be primitive types, or collections of primitive types themselves.
There are ways to do it, but if this is a requirement for you, I would look at storing your data in a JSON string variable, and parsing it where/ when needed.
I'm going to do a project task as the last part of a programming course in C#, and I have choosen to do a media library for storing information of DVD movie collections. The thing I need some advice is about how to save this information since the task should be solved by using simple text files and not databases which had been better.
So, which is the best way to save this information to be able to add, edit and allow search for titles and actors and also sort after genre to mention some of the idéas I have for this program.
Should I just save the information by just add the title, year, playtime, director, actor1, actor2, actor3, genre, grade, comment for each movie lika a row at the end of the file? Can I search in the file or should I, in some way, first read it all into an array, and then do the serach and perhaps edit and save the complete file again?
Or is it better to use XML like this:
<movie_collections>
<movie>
<title=Tron Legacy"></title>
<year=2010></year>
<playtime=120></playtime>
etc.
</movie>
</movie_collections>
If I use XML, can I search for a title and just load or edit that part? Are there better alternatives than these?
You may store the data in XML file. An XML file can store data similar to a database table. You can add multiple records, hierarchical data etc... You may easily query the data using LINQ to XML.
If you dont want to use LINQ to XML, You can use so typical XMLDocument to handle the XML data.
Maybe you're approaching this at too low a level. The file is realistically just for persistance rather than implement your own DB as a file.
Just implement the object that you're after and serialize it out imo.
public class MovieCollections
{
private ICollection<Movie> Movies {get; set;}
// etc...
}
public class Movie
{
public string Title {get; private set;}
public int PlayTime {get; private set;}
public DateTime ReleaseDate {get; private set;}
}
Serialize instance of MovieCollections.
Keep MovieCollections perhaps as a Singleton instance since you only want one collection.
By the way this seems to be very general question & a quite common homework question too!!
I think you should google for this, you will get better projects ideas.
Something similar on this lines :
1.Simple Movie Database in C# using Microsoft Access
2. Create a Movie Database Application
3. imdbapi
& finally
4. SO's similar post
And as far as comparison between a database & XML is concern, I did recommend you a database because several pros over XML as far such type of project is considered.
As #M_Affifi suggests, think of all your interaction with your data (e.g. sort, add, find, delete, update, etc) through your objects which reside in memory (e.g. Instance of MovieCollections class and instances of Movie class), the XML file is only used to store the state of your objects, so you have that data available next time you run your application.
Once you're done manipulating your data just serialize it to XML. Serialization is just the process to convert your objects into a format that you can store or transfer, in this case will be conversion to XML. Take a look at this reference, Your serialization needs are very basic, so you'll need just few lines of code.
System.Xml.Serialization.XmlSerializer x = new System.Xml.Serialization.XmlSerializer(YourObject.GetType());
x.Serialize(Console.Out, YourObject)
Finally, I would suggest thinking of your application logic. When program starts you may want to check if your Movies.xml exists, if so, then 'Deserialize' it, which means loading it from XML to your object collection. If XML file doesn't exist then just start manipulating your objects and when you're done give the option to Save(Serialize). I can help with the serialization/deserialization process if needed.
Hope this is helpful
;