The program below is a socket program that receives data at a pretty fast pace. It runs fine with the CodeFirst section disabled. If I enable it, the program starts out fine but then degrades slowly to the point where nothing appears to be written to the SQL EXPRESS 2012 db. I check this by running the SQL statement below which just selects the last five rows n SQL Management Studio 2012.
Is there something that I am doing wrong?
select * from [MarketDataEntities]
where MarketDataEntities.MarketDataEntityID not in (
select top (
(select count(*) from [MarketDataEntities]) - 5
) MarketDataEntities.MarketDataEntityID
from [MarketDataEntities]
)
using (var dbTDC = new TickDataTestContext())
{
var tde = new SymbolTickDataEntity { Symbol = symbol };
if (!dbTDC.SymbolTickDataDbSet.Any(a => a.Symbol.Equals(symbol)))
{
dbTDC.SymbolTickDataDbSet.Add(tde);
dbTDC.SaveChanges();
}
var mdde = new MarketDataDepthEntity();
dbTDC.MarketDataDepthDbSet.Add(mdde);
dbTDC.SaveChanges();
while (true)
{
// Wait for next request from client
int len = socket.Receive(zmq_buffer);
if (len < 1)
{
Console.WriteLine("Len < 1");
continue;
}
//Console.WriteLine("Got quote");
count++;
// Copy the bytes
byte[] bytes = new byte[len];
Buffer.BlockCopy(zmq_buffer, 0, bytes, 0, len);
MarketDataDepth mdd = MarketDataDepth.CreateBuilder().MergeFrom(bytes).Build();
PrintMarketDataDepth(mdd);
#if false
for (int i = 0; i < mdd.MdCount; i++)
{
MarketData md = mdd.MdList[i];
string timestamp = md.Time;
int index = timestamp.IndexOf(",");
if(-1 != index)
timestamp = timestamp.Remove(index);
DateTime parseResult;
if (false == System.DateTime.TryParseExact(timestamp, format, provider, DateTimeStyles.None, out parseResult))
continue;
var mde = new MarketDataEntity
{
NMDDTag = (long)mdd.NMDDTag,
QType = (0 == md.QuoteType ? QuoteType.Bid : QuoteType.Ask),
QPrice = md.Price,
QSize = md.Size,
QSource = md.Source,
QLiquidityTag = md.ID,
QSilo = md.Silo,
QTimeStamp = parseResult
};
dbTDC.MarketDataDbSet.Add(mde);
mdde.Depth.Add(mde);
}
if (0 == count % 500)
dbTDC.SaveChanges();
#endif
}
}
}
catch (DbEntityValidationException e)
{
foreach (var eve in e.EntityValidationErrors)
{
Console.WriteLine("Entity of type \"{0}\" in state \"{1}\" has the following validation errors:",
eve.Entry.Entity.GetType().Name, eve.Entry.State);
foreach (var ve in eve.ValidationErrors)
{
Console.WriteLine("- Property: \"{0}\", Error: \"{1}\"",
ve.PropertyName, ve.ErrorMessage);
}
}
throw;
}
}
}
public enum QuoteType { Bid = 0, Ask }
public class MarketDataEntity
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int MarketDataEntityID { get; set; }
public long NMDDTag { get; set; }
public QuoteType QType { get; set; }
public double QPrice { get; set; }
public double QSize { get; set; }
public string QSource { get; set; }
public string QLiquidityTag { get; set; }
public string QSilo { get; set;}
[Column("timestamp", TypeName = "datetime2")]
public DateTime QTimeStamp { get; set; }
}
public class MarketDataDepthEntity
{
public int MarketDataDepthEntityID { get; set; }
public virtual IList<MarketDataEntity> Depth { get; set; }
[Column("timestamp", TypeName = "datetime2")]
public DateTime TimeStamp { get; set; }
public MarketDataDepthEntity() { Depth = new List<MarketDataEntity>(); }
}
public class SymbolTickDataEntity
{
public int SymbolTickDataEntityID { get; set; }
[Key]
[Required]
public string Symbol { get; set; }
public virtual IList<MarketDataDepthEntity> Mdds { get; set; }
public SymbolTickDataEntity() { Mdds = new List<MarketDataDepthEntity>(); }
}
public class TickDataTestContext : DbContext
{
public DbSet<MarketDataEntity> MarketDataDbSet { get; set; }
public DbSet<MarketDataDepthEntity> MarketDataDepthDbSet { get; set; }
public DbSet<SymbolTickDataEntity> SymbolTickDataDbSet { get; set; }
}
From your code looks like you are keeping around the TickDataTestContext for the lifetime of your application. So as you keep adding data the local cache keeps increasing increasing memory usage hence performance degradation.
You should rewrite the code to create a new instance of TickDataTestContext for each request that needs to be saved then do the work, save changes and dispose the context.
Related
I'm using LINQtoCSV within a program that allows the user to import an order from a CSV file. I have all the code working however, if the CSV file doesn't have the exact column headers then it doesn't work.
Below is my class that LINQtoCSV reads into -
public class orderProduct
{
public orderProduct() { }
public string product { get; set; }
public string price { get; set; }
public string orderQty { get; set; }
public string value { get; set; }
public string calculateValue()
{
return (Convert.ToDouble(price) * Convert.ToDouble(orderQty)).ToString();
}
}
If the CSV file doesn't have the exact headers it won't work. The data I actually only need is the first 4 strings.
Below is my function that actually reads in the data.
private void csvParse()
{
// order.Clear();
string fileName = txt_filePath.Text.ToString().Trim();
try
{
CsvContext cc = new CsvContext();
CsvFileDescription inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true
};
IEnumerable<orderProduct> fromCSV = cc.Read<orderProduct>(fileName, inputFileDescription);
foreach (var d in fromCSV)
{
MessageBox.Show($#"Product:{d.product},Quantity:""{d.orderQty}"",Price:""{d.price}""");
orderReturn.Add(d);
}
this.DialogResult = DialogResult.Yes;
this.Close();
}
catch (Exception ex)
{
if (ex.ToString().Contains("being used by another process"))
{
MessageBox.Show("Error: Please close the file in Excel and try again");
}
else
{
MessageBox.Show(ex.ToString());
}
}
}
I want the user to be able to just pass in a file and then select the relevant columns which relate to the corresponding values and then read in the data ignoring any columns that haven't been selected.
Hope this all makes sense, is something like this possible within LINQtoCSV
You have to add IgnoreUnknownColumns = true to your CsvFileDescription
CSV:
product,price,someColumn,orderQty,value,otherColumn
my product,$123,xx,2,$246,aa
my other product,$10,yy,3,$30,bb
Working code (I modified your code a little bit, to run it in a console)
using System;
using System.Collections.Generic;
using LINQtoCSV;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
csvParse();
Console.ReadLine();
}
private static void csvParse()
{
string fileName = "../../../test.csv"; // provide a valid path to the file
CsvContext cc = new CsvContext();
CsvFileDescription inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true,
IgnoreUnknownColumns = true // add this line
};
IEnumerable<orderProduct> fromCSV = cc.Read<orderProduct>(fileName, inputFileDescription);
foreach (var d in fromCSV)
{
Console.WriteLine($#"Product:{d.product},Quantity:""{d.orderQty}"",Price:""{d.price}""");
}
}
}
public class orderProduct
{
public orderProduct() { }
public string product { get; set; }
public string price { get; set; }
public string orderQty { get; set; }
public string value { get; set; }
public string calculateValue()
{
return (Convert.ToDouble(price) * Convert.ToDouble(orderQty)).ToString();
}
}
}
Output:
Product:my product,Quantity:"2",Price:"$123"
Product:my other product,Quantity:"3",Price:"$10"
If your properties have different names than CSV columns, you should use CsvColumn attribute:
public class OrderProduct
{
[CsvColumn(Name = "product")]
public string Product { get; set; }
[CsvColumn(Name = "price")]
public string Price { get; set; }
[CsvColumn(Name = "orderQty")]
public string OrderQuantity { get; set; }
public string Value { get; set; }
public string calculateValue()
{
return (Convert.ToDouble(Price) * Convert.ToDouble(OrderQuantity)).ToString();
}
}
Or if you prefer mapping columns by their indices:
public class OrderProduct
{
[CsvColumn(FieldIndex = 0)]
public string Product { get; set; }
[CsvColumn(FieldIndex = 1)]
public string Price { get; set; }
[CsvColumn(FieldIndex = 2)]
public string OrderQuantity { get; set; }
public string Value { get; set; }
public string calculateValue()
{
return (Convert.ToDouble(Price) * Convert.ToDouble(OrderQuantity)).ToString();
}
}
If you have to specify the columns on the fly, the only way seems to be to read raw data and process it yourself (the solution is based on this article):
internal class DataRow : List<DataRowItem>, IDataRow
{
}
...
int productColumnIndex = 0; // your users will provide it
var fromCSV = cc.Read<DataRow>(fileName);
foreach (var row in fromCSV)
{
var orderProduct = new OrderProduct
{
Product = row[productColumnIndex].Value,
};
Console.WriteLine(orderProduct.Product);
}
I've sourced my data for a treeview using the Proces datastructure below. expanded simply indicates whether or not the tree item has been expanded to show all of its children. This is my attempt to iterate through a map all showing items into a DataTable.
public class Proces
{
public string PN { get; set; }
public string Description { get; set; }
public int Qty { get; set; }
public string PartType { get; set; }
public decimal PricePer { get; set; }
public string Mfr { get; set; }
public int Stock { get; set; }
public int OnOrder { get; set; }
public string parent { get; set; }
public bool expanded { get; set; } = false;
public List<Proces> subProcesses { get; set; }
}
I'm trying to map this out into a DataTable, but i keep getting a stack overflow.
void generateShownTree(List<Proces> proccess)
{
foreach (Proces proc in processes)
{
DataRow drNew = export.NewRow();
drNew["Parent"] = proc.parent;
drNew["PN"] = proc.PN;
drNew["Description"] = proc.Description;
drNew["Qty"] = proc.Qty;
drNew["PartType"] = proc.PartType;
drNew["PricePer"] = proc.PricePer;
drNew["Mfr"] = proc.Mfr;
drNew["Stock"] = proc.Stock;
drNew["OnOrder"] = proc.OnOrder;
export.Rows.Add(drNew);
if (proc.expanded == true)
{
foreach (Proces subProc in proc.subProcesses)
{
subProc.parent = proc.PN;
drNew = export.NewRow();
drNew["Parent"] = subProc.parent;
drNew["PN"] = subProc.PN;
drNew["Description"] = subProc.Description;
drNew["Qty"] = subProc.Qty;
drNew["PartType"] = subProc.PartType;
drNew["PricePer"] = subProc.PricePer;
drNew["Mfr"] = subProc.Mfr;
drNew["Stock"] = subProc.Stock;
drNew["OnOrder"] = subProc.OnOrder;
export.Rows.Add(drNew);
generateShownTree(proc.subProcesses);
}
}
}
}
I assume you don't want to iterate the list of subprocesses as well as invoke the generateShownTree method recursively. I also changed the name of the argument passed to generateShownTree to match the object being iterated.
static void generateShownTree(List<Proces> processes)
{
foreach (Proces proc in processes)
{
DataRow drNew = export.NewRow();
drNew["Parent"] = proc.parent;
drNew["PN"] = proc.PN;
drNew["Description"] = proc.Description;
drNew["Qty"] = proc.Qty;
drNew["PartType"] = proc.PartType;
drNew["PricePer"] = proc.PricePer;
drNew["Mfr"] = proc.Mfr;
drNew["Stock"] = proc.Stock;
drNew["OnOrder"] = proc.OnOrder;
export.Rows.Add(drNew);
if (proc.expanded)
{
generateShownTree(proc.subProcesses);
}
}
}
I have a list defined as below in each of 11 different classes (which handle web-services)
private List<edbService> genEdbService;
internal class edbService
{
public string ServiceID { get; set; }
public string ServiceName { get; set; }
public string ServiceDescr { get; set; }
public string ServiceInterval { get; set; }
public string ServiceStatus { get; set; }
public string ServiceUrl { get; set; }
public string SourceApplication { get; set; }
public string DestinationApplication { get; set; }
public string Function { get; set; }
public string Version { get; set; }
public string userid { get; set; }
public string credentials { get; set; }
public string orgid { get; set; }
public string orgunit { get; set; }
public string customerid { get; set; }
public string channel { get; set; }
public string ip { get; set; }
}
The list is populated in each class by reading the web-service configuration data from xml files in each class:
public DCSSCustomerCreate_V3_0()
{
try
{
XElement x = XElement.Load(global::EvryCardManagement.Properties.Settings.Default.DataPath + "CustomerCreate.xml");
// Get global settings
IEnumerable<XElement> services = from el in x.Descendants("Service")
select el;
if (services != null)
{
edb_service = new List<edbService>();
// edb_service= Common.populateEDBService("CustomerCreate.xml");
foreach (XElement srv in services)
{
edbService edbSrv = new edbService();
edbSrv.ServiceID = srv.Element("ServiceID").Value;
edbSrv.ServiceName = srv.Element("ServiceName").Value;
edbSrv.ServiceDescr = srv.Element("ServiceDescr").Value;
edbSrv.ServiceInterval = srv.Element("ServiceInterval").Value;
edbSrv.ServiceStatus = srv.Element("ServiceStatus").Value;
edbSrv.ServiceUrl = srv.Element("ServiceUrl").Value;
foreach (XElement ServiceHeader in srv.Elements("ServiceHeader"))
{
...
now what I want to do is have this code in one place in my Common.cs class so I tried:
public static List<edbService> populateEDBService(string xmlDataFile)
{
try
{
XElement x = XElement.Load(global::EvryCardManagement.Properties.Settings.Default.DataPath + xmlDataFile);
// Get global settings
IEnumerable<XElement> services = from el in x.Descendants("Service")
select el;
if (services != null)
{
//edb_Service = new List<edbService>();
foreach (XElement srv in services)
{
edbService edbSrv = new edbService();
edbSrv.ServiceID = srv.Element("ServiceID").Value;
edbSrv.ServiceName = srv.Element("ServiceName").Value;
edbSrv.ServiceDescr = srv.Element("ServiceDescr").Value;
edbSrv.ServiceInterval = srv.Element("ServiceInterval").Value;
edbSrv.ServiceStatus = srv.Element("ServiceStatus").Value;
edbSrv.ServiceUrl = srv.Element("ServiceUrl").Value;
foreach (XElement ServiceHeader in srv.Elements("ServiceHeader"))
{
edbSrv.SourceApplication = ServiceHeader.Element("SourceApplication").Value;
edbSrv.DestinationApplication = ServiceHeader.Element("DestinationApplication").Value;
edbSrv.Function = ServiceHeader.Element("Function").Value;
edbSrv.Version = ServiceHeader.Element("Version").Value;
foreach (XElement ClientContext in ServiceHeader.Elements("ClientContext"))
{
edbSrv.userid = ClientContext.Element("userid").Value;
edbSrv.credentials = ClientContext.Element("credentials").Value;
edbSrv.orgid = ClientContext.Element("orgid").Value;
edbSrv.orgunit = ClientContext.Element("orgunit").Value;
edbSrv.customerid = ClientContext.Element("customerid").Value;
edbSrv.channel = ClientContext.Element("channel").Value;
edbSrv.ip = ClientContext.Element("ip").Value;
}
}
// populateEDBService.Add(edbSrv);
}
}
}
catch (Exception ex)
{
/* Write to log */
Common.logBuilder("CustomerCreate : Form --> CustomerCreate <--", "Exception", Common.ActiveMQ,
ex.Message, "Exception");
/* Send email to support */
emailer.exceptionEmail(ex);
}
return;
}
Now I get a compile error on the return; saying that An object of a type convertible to 'System.Collections.Generic.List<EvryCardManagement.Common.edbService>' is required
and in the class that should call this method, I want to do something like:
edb_service = Common.populateEDBService("CustomerUpdate.xml");
but I get an error Cannot implicitly convert type 'System.Collections.Generic.List<EvryCardManagement.Common.edbService>' to 'System.Collections.Generic.List<EvryCardManagement.CustomerUpdate.edbService>'
So firstly how should I return the list from my generic method and how should I call it to return the list populated with the configuration data?
It sounds like you have your class edbService defined in two namespaces,
EvryCardManagement.Common and
EvryCardManagement.CustomerUpdate
I would suggest defining it in only EvryCardManagement.Common and have everything reference it from there.
I wrote very simple class, that perfom data access.
It checks if line with that day exist in table and update her or create a new line.
public class DataAccessClass
{
public static DayWeather GetDayWeather(DateTime date)
{
try
{
using (var db = new Context())
{
var query =
(from day in db.DayWeather
where ((DateTime)day.DateOfDay).Date == date.Date
select new DayWeather((short)day.Temperature, (ushort)day.WindSpeed, (ushort)day.Pressure, (ushort)day.Humidity, day.Cloudiness, day.TypeRecip, (DateTime)day.DateOfDay)).First();
return query;
}
}
catch (Exception exp)
{
if (!EventLog.SourceExists("DataAccessSource"))
{
EventLog.CreateEventSource("DataAccessSource", "DataAccessErrorLog");
}
EventLog.WriteEntry("DataAccessSource", exp.Message);
throw new Exception("Problem with data get.");
}
}
public static void SaveDayWeather(DayWeather day)
{
try
{
using (var db = new Context())
{
var existingDay =
(from d in db.DayWeather
where ((DateTime)day.DateOfDay).Date == day.DateOfDay.Date
select d).SingleOrDefault<DayWeather>();
if (existingDay != null)
{
existingDay.Temperature = day.Temperature;
existingDay.WindSpeed = day.WindSpeed;
existingDay.Pressure = day.Pressure;
existingDay.Humidity = day.Humidity;
existingDay.Cloudiness = day.Cloudiness;
existingDay.TypeRecip = day.TypeRecip;
db.SaveChanges();
}
else
{
DayWeather newDay = new DayWeather();
newDay.DateOfDay = day.DateOfDay;
newDay.Temperature = day.Temperature;
newDay.WindSpeed = day.WindSpeed;
newDay.Pressure = day.Pressure;
newDay.Humidity = day.Humidity;
newDay.Cloudiness = day.Cloudiness;
newDay.TypeRecip = day.TypeRecip;
db.DayWeather.Add(newDay);
db.SaveChanges();
}
}
}
It use EF for generate database. The Contex class and class for save look like this:
public class DayWeather
{
public short Temperature { get; set; }
public ushort WindSpeed { get; set; }
public ushort Pressure { get; set; }
public ushort Humidity { get; set; }
public string Cloudiness { get; set; }
public string TypeRecip { get; set; }
public DateTime DateOfDay { get; set; }
public DayWeather(short Temperature, ushort WindSpeed, ushort Pressure, ushort Humidity, string Cloudiness, string TypeRecip, DateTime Date)
{
this.Temperature = Temperature;
this.WindSpeed = WindSpeed;
this.Pressure = Pressure;
this.Humidity = Humidity;
this.Cloudiness = Cloudiness;
this.TypeRecip = TypeRecip;
this.DateOfDay = Date;
}
public DayWeather()
{
}
}
internal class Context : DbContext
{
public DbSet<DayWeather> DayWeather { get; set; }
}
I call this methods by this code:
DataAccessClass.SaveDayWeather(new DayWeather(12, 12, 12, 12, "Yes", "rain", DateTime.Now));
DayWeather day = DataAccessClass.GetDayWeather(DateTime.Now);
Console.WriteLine(day.ToString());
Console.ReadKey();
It should generate new database, but error occurs. In message it write that can`t connect to the SQL Server.
Is somebody know what is wrong?
P.S. Sorry for my bad English.
P.P.S. I added EF by NuGet.
You can manually specify the connection string as follows
using (var db = new Context("connectionString"))
The default constructor looks up a connection string in the web.config with the same name as the derived context class Context.
If it fails to find one it defaults to
Data Source=.\SQLEXPRESS;
or
Data Source=(LocalDb)\v11.0;
depending on the version of sql server you are using.
This is my code for retrieving the list of comments from couchbase. The design document name is "Task" and the view name is: "GetComments".
public List<CommentsVO> GetComments(string TaskID, int LastCommentID, int totalCommentCount)
{
int startCount = LastCommentID - 1;
int endCount = startCount - 19;
int remainingCount = totalCommentCount - endCount;
if (endCount < 0)
{
endCount = 0;// totalCommentCount - remainingCount;
}
IView<CommentsVO> results = oCouchbase.GetView<CommentsVO>("Task", "GetComments");
results.StartKey(new object[] { TaskID, startCount }).EndKey(new object[] { TaskID, endCount });
if (results != null)
{
List<CommentsVO> resultlist = new List<CommentsVO>();
foreach (CommentsVO vo in results)//Here it is not entering inside the loop... Am i missing anything in this condition
{
resultlist.Add(vo);
}
resultlist.Reverse();
return resultlist;
}
return null;
}
My CommentsVo code is:
public class CommentsVO
{
public CommentsVO()
{
CommentedOn = Convert.ToString(DateTime.Now);
IsActive = "1";
}
[JsonIgnore]
public string TaskID { get; set; }
[JsonProperty("commented_user_id")]
public string CommentedUserID { get; set; }
[JsonProperty("commented_user_name")]
public string CommentedUserName { get; set; }
[JsonProperty("comment_description")]
public string CommentDescription { get; set; }
[JsonProperty("commented_on")]
public string CommentedOn { get; set; }
[JsonProperty("is_active")]
public string IsActive { get; set; }
[JsonProperty("seq")]
public string Sequence { get; set; }
}
My couchbase view code is:
function(doc) {
for(var i in doc.comments) {
emit([doc._id,doc.comments[i].seq],doc.comments[i]);
}
}
I have tried without using startkey and endkey its iterating but when i tried using startkey and endkey it is not entering inside the loop..
Kindly help me out..
When using a composite key, you would specify an array of keys in StartKey/EndKey. In your code, you're actually overwriting the keys with your second calls to StarKey and EndKey.
So something like:
results.StartKey(new object[] { TaskId, startCount }).EndKey(new object[] { TaskId, endCount });