We Keep Coding

Mystery Memory Leak when overwriting a static DataSet

I am running a test function to identify a memory leak:
[TestMethod]
public void DatabaseTools_Other_MemoryTest()
{
for (int i = 0; i < 100; i++)
{
try
{
var r = DatabaseTools.GetDataSet(true);
r = null;
}
catch (Exception e)
{
int EndPoint = i;
}
}
}
The goal of this method is to call DatabaseTools.GetDataSet(true) until it hits an OutOfMemoryException, which happens during the 3rd or 4th load. However, as I understand it, this shouldn't actually ever happen- this is DatabaseTools.GetDataSet:
public static DataSet GetDataSet(bool setData, string sqlText = null, string strConnection = null)
{
sqlText = sqlText ?? FullFilemakerQuery;
if (setData)
{
Database = strConnection;
Data = new DataSet();
}
DataSet dataSet = new DataSet();
using (SqlConnection connection = GetConnectionString())
{
using (SqlDataAdapter dataAdapter = new SqlDataAdapter(sqlText, connection))
{
dataAdapter.SelectCommand.CommandType = System.Data.CommandType.Text;
if (setData)
{
dataAdapter.FillSchema(Data, SchemaType.Source);
DisableAutoIncrement(Data);
dataAdapter.Fill(Data);
NameTables(Data, sqlText);
BuildDataKeysAndRelations(Database);
dataSet = null;
}
else
{
dataAdapter.FillSchema(dataSet, SchemaType.Source);
DisableAutoIncrement(dataSet);
dataAdapter.Fill(dataSet);
NameTables(dataSet, sqlText);
}
}
connection.Close();
}
return dataSet ?? Data;
}
public static void NameTables(DataSet dataSet, string sqlText)
{
for (int i = 0; i < dataSet.Tables.Count; i++)
{
dataSet.Tables[i].TableName = sqlText.Split(';')[i].Split(Dividers, StringSplitOptions.None)[1].Trim();
}
}
public static void DisableAutoIncrement(DataSet data)
{
foreach (DataTable T in data.Tables)
{
T.PrimaryKey.Select(c => { c.AutoIncrement = false; return c; }).ToList();
}
}
When passing only 'true' to this function, it sets sqlText equal to a static FullFileMakerQuery which selects everything the program could use from the database, it then gets the default database name (Database has a custom setter such that when given a null or empty value, it sets itself to default), and sets the static Data to a new DataSet. We have tried setting it to null at this point (no change) and we have tried using Data.Dispose() which caused errors. Because this function can also just return the dataset without also setting the global Data to it, we initialize a new DataSet dataSet. Then we do the standard connect, data adapter, fillschema, load data.
The weirdness: by setting breakpoints in the memory test function and saving dumps, loading the dataset once takes some amount of memory, reloading it uses less memory (by about 36,000 bytes in System.Data.Common.StringStorage) and then reloading it again uses more memory (by about 120,000 bytes in the same place as before). If we reload it one more time, it uses even more memory and crashes due to an OutOfMemoryException, and at this point I have no idea what could be causing it.

You never call .Dispose() on your DataSet, neither on the return value, nor on your static variable before you call new again.
As you are using using blocks with other classes already, I guess you know what to do to actually call .Dispose() on it.
Turn on Visual Studio Static Code Analysis, it will give you warnings should you forget to do so.

C# Stops Executing When Opening A Connection To MySql

I am writing an application to transfer data from a simulated OPC server to a MySql Database. I am using some of the libraries from OPC Foundation - OPCNETAPI.DLL, OPCNETAPI.COM.DLL.
I have the program reading from the server happily, but it seems as soon as I open a connection inside the loop of reading data, it loops about 5 times and then stops executing.. There are no exceptions thrown and I have tried for hours to work it out and miserably failed.
I've also changed VisualStudio to break when an exception is thrown, and it doesn't
Hopefully some one may know whats going on, but there is nothing obvious, hopefully the following code will spark ideas!
Windows Forms Code
public void readplc()
{
Opc.URL url = new Opc.URL("opcda://localhost/Matrikon.OPC.Simulation.1");
Opc.Da.Server server = null;
OpcCom.Factory fact = new OpcCom.Factory();
server = new Opc.Da.Server(fact, null);
try
{
server.Connect(url, new Opc.ConnectData(new System.Net.NetworkCredential()));
}
catch (Exception ecy)
{
}
// Create a group
Opc.Da.Subscription group;
Opc.Da.SubscriptionState groupState = new Opc.Da.SubscriptionState();
groupState.Name = "Group";
groupState.Active = true;
group = (Opc.Da.Subscription)server.CreateSubscription(groupState);
// add items to the group.
Opc.Da.Item[] items = new Opc.Da.Item[2];
items[0] = new Opc.Da.Item();
items[0].ItemName = "Random.Int1";
items[1] = new Opc.Da.Item();
items[1].ItemName = "Random.Time";
items = group.AddItems(items);
group.DataChanged += new Opc.Da.DataChangedEventHandler(OnTransactionCompleted);
}
public void OnTransactionCompleted(object group, object hReq, Opc.Da.ItemValueResult[] items)
{
for (int i = 0; i < items.GetLength(0); i++)
{
try
{
string a = items[i].Value.ToString();
string b = items[i].ItemName;
string c = items[i].Key;
MySqlConnection conn = new MySqlConnection("server=localhost;user id=localhost;password=localhost;database=localhost;pooling=false");
conn.Open();
conn.Close();
}
catch (Exception ec)
{
MessageBox.Show(ec.Message);
}
}
}
I have set break points throughout the code, stepped through everything possible and still nothing obvious that's causing the issue.

How to test to see if mySql Database is working?

I am new to MySQL database, I am using Visual Studio C# to connect to my database. I have got a following select method. How can I run it to check if it is working?
EDITED The open and close connection methods
//Open connection to database
private bool OpenConnection()
{
try
{
// connection.open();
return true;
}
catch (MySqlException ex)
{
//When handling errors, your application's response based
//on the error number.
//The two most common error numbers when connecting are as follows:
//0: Cannot connect to server.
//1045: Invalid user name and/or password.
switch (ex.Number)
{
case 0:
MessageBox.Show("Cannot connect to server.");
break;
case 1045:
MessageBox.Show("Invalid username/password, please try again");
break;
}
return false;
}
}
//Close connection
private bool CloseConnection()
{
try
{
connection.Close();
return true;
}
catch (MySqlException ex)
{
MessageBox.Show(ex.Message);
return false;
}
}
Select method which is in the same class as the close and open connection as shown above
public List<string>[] Select()
{
string query = "SELECT * FROM Questions";
//Create a list to store the result
List<string>[] list = new List<string>[3];
list[0] = new List<string>();
list[1] = new List<string>();
list[2] = new List<string>();
list[3] = new List<string>();
list[4] = new List<string>();
list[5] = new List<string>();
list[6] = new List<string>();
list[7] = new List<string>();
//Open connection
if (this.OpenConnection() == true)
{
//Create Command
MySqlCommand cmd = new MySqlCommand(query, connection);
//Create a data reader and Execute the command
MySqlDataReader dataReader = cmd.ExecuteReader();
//Read the data and store them in the list
while (dataReader.Read())
{
list[0].Add(dataReader["id"] + "");
list[1].Add(dataReader["difficulty"] + "");
list[2].Add(dataReader["qustions"] + "");
list[3].Add(dataReader["c_answer"] + "");
list[4].Add(dataReader["choiceA"] + "");
list[5].Add(dataReader["choiceB"] + "");
list[6].Add(dataReader["choiceC"] + "");
list[7].Add(dataReader["choiceD"] + "");
}
//close Data Reader
dataReader.Close();
//close Connection
this.CloseConnection();
//return list to be displayed
return list;
}
else
{
return list;
}
}
This method is in a separate class which has got all the database connection settings. Now that I want to call this method from my main class to test it to see if it's working, how can I do this?

You should create an object instance of that DB class and then call the Select() method.
So, supposing that this DB class is named QuestionsDB you should write something like this:
QuestionDB questionDAL = new QuestionDB();
List<string>[] questions = questionDAL.Select();
However, before this, please correct this line
List<string>[] list = new List<string>[8]; // you need 8 lists for your db query
You could check if you have any record testing if the first list in your array list has more than zero elements.
if(questions[0].Count > 0)
... // you have read records.
However, said that, I will change your code adding a specific class for questions and using a list(of Question) instead of an array of list
So, for example, create a class like this
public class Question
{
public string ID;
public string Difficulty;
public string Question;
public string RightAnswer;
public string AnswerA;
public string AnswerB;
public string AnswerC;
public string AnswerD;
}
and change your select to return a List(of Question)
List<Question> list = new List<Question>;
......
while (dataReader.Read())
{
Question qst = new Question();
qst.ID = dataReader["id"] + "";
qst.Difficulty = dataReader["difficulty"] + "";
qst.Question = dataReader["qustions"] + "";
qst.RightAnswer = dataReader["c_answer"] + "";
qst.AnswerA = dataReader["choiceA"] + "";
qst.AnswerB = dataReader["choiceB"] + "";
qst.AnswerC = dataReader["choiceC"] + "";
qst.AnswerD = dataReader["choiceD"] + "";
list.Add(qst);
}
return list;

You can test whether the method works by writing a unit test for it. A good unit testing frame work is Nunit. Before you call this you must create and open a connection to the DB:
//Open connection
if (this.OpenConnection() == true)
{
as the other person said, you will want to fix the lists up.

Error in Calling SQL CLR function

I am writing a CLR function to parse a table column and write the results into another table. Basic requirement is parsing the Detail column, which contains a Time part and an ID part. Results will be the time difference between two Ids.
Ex: Time1,Id1;Time2,Id2;Time3,Id3... and so on
Time2-Time1 is time taken by Id1 in seconds.
Same function is working in normal console application but CLR function is throwing the exception when I call it from SQL server.
Error is
A .NET Framework error occurred during execution of user-defined routine or aggregate "Function1":
System.Security.HostProtectionException: Attempted to perform an operation that was forbidden by the CLR host.
The protected resources (only available with full trust) were: All
The demanded resources were: UI
System.Security.HostProtectionException:
at UserDefinedFunctions.Function1(String msisdn, String promptdetails)
My code is:
SqlConnection conn = new SqlConnection();
SqlCommand cmd = new SqlCommand();
int count = 0;
string PromptPart = string.Empty;
string PrevPromptPart = string.Empty;
DateTime TimePart;
TimePart = new DateTime();
DateTime PrevTimePart;
PrevTimePart = new DateTime();
TimeSpan difference;
try
{
count++;
conn.ConnectionString = "Context Connection=true";
cmd.Connection = conn;
String[] string1 = promptdetails.Split(";".ToCharArray());
foreach (var item1 in string1)
{
count++;
String[] string2 = item1.Split(",".ToCharArray());
PromptPart = string2[1];
TimePart = DateTime.ParseExact(string2[0], "M/d/yyyy h:mm:ss tt", System.Globalization.CultureInfo.InvariantCulture);
if (count > 1)
{
StringBuilder sbQuery = new StringBuilder(1024);
sbQuery.Append("INSERT INTO [Shami].[DBO].[data] (MSISDN,PromptID1,PromptID2,TimeDifference) VALUES");
sbQuery.Append("('");
sbQuery.Append(msisdn);
sbQuery.Append("',");
difference = TimePart.Subtract(PrevTimePart);
sbQuery.Append("'");
sbQuery.Append(PrevPromptPart);
sbQuery.Append("','");
sbQuery.Append(PromptPart);
sbQuery.Append("',");
sbQuery.Append(difference.Seconds);
sbQuery.Append(")");
string sub = string.Empty;
sub = sbQuery.ToString();
try
{
conn.Open();
cmd = new SqlCommand(sub);
SqlContext.Pipe.ExecuteAndSend(cmd);
}
catch (Exception ie)
{
Console.WriteLine("Error..");
}
}
if (count <= 1)
{
StringBuilder sbQuery = new StringBuilder(1024);
sbQuery.Append("INSERT INTO [Shami].[DBO].[data] (MSISDN,PromptID1,PromptID2,TimeDifference) VALUES");
sbQuery.Append("('");
sbQuery.Append(msisdn);
sbQuery.Append("',");
sbQuery.Append("'_'");
sbQuery.Append(",");
sbQuery.Append(PromptPart);
sbQuery.Append(",");
sbQuery.Append("'0'");
sbQuery.Append(")");
string sub = string.Empty;
sub = sbQuery.ToString();
try
{
conn.Open();
cmd = new SqlCommand(sub);
SqlContext.Pipe.ExecuteAndSend(cmd);
}
catch (Exception ie)
{
Console.WriteLine("Error..");
}
}
PrevPromptPart = PromptPart;
PrevTimePart = TimePart;
}
}
catch (Exception)
{ ;}
finally
{
conn.Close();
conn.Dispose();
cmd.Dispose();
}
return msisdn;
Please let me know where I'm going wrong. I can't debug in CLR.

You would just need to replace your line
Console.WriteLine("Error...");
with this line:
SqlContext.Pipe.Send("Error...");
This will do the same thing for you and it will run

Massive differences in reexecution

I wrote a little C# application that indexes a book and executes a boolean textretrieval algorithm on the index. The class at the end of the post showes the implementation of both, building the index and executing the algorithm on it.
The code is called via a GUI-Button in the following way:
private void Execute_Click(object sender, EventArgs e)
{
Stopwatch s;
String output = "-----------------------\r\n";
String sr = algoChoice.SelectedItem != null ? algoChoice.SelectedItem.ToString() : "";
switch(sr){
case "Naive search":
output += "Naive search\r\n";
algo = NaiveSearch.GetInstance();
break;
case "Boolean retrieval":
output += "boolean retrieval\r\n";
algo = BooleanRetrieval.GetInstance();
break;
default:
outputTextbox.Text = outputTextbox.Text + "Choose retrieval-algorithm!\r\n";
return;
}
output += algo.BuildIndex("../../DocumentCollection/PilzFuehrer.txt") + "\r\n";
postIndexMemory = GC.GetTotalMemory(true);
s = Stopwatch.StartNew();
output += algo.Start("../../DocumentCollection/PilzFuehrer.txt", new String[] { "Pilz", "blau", "giftig", "Pilze" });
s.Stop();
postQueryMemory = GC.GetTotalMemory(true);
output += "\r\nTime elapsed:" + s.ElapsedTicks/(double)Stopwatch.Frequency + "\r\n";
outputTextbox.Text = output + outputTextbox.Text;
}
The first execution of Start(...) runs about 700µs, every rerun only takes <10µs.
The application is compiled with Visual Studio 2010 and the default 'Debug' buildconfiguration.
I experimentad a lot to find the reason for that including profiling and different implementations but the effect always stays the same.
I'd be hyppy if anyone could give me some new ideas what I shall try or even an explanation.
class BooleanRetrieval:RetrievalAlgorithm
{
protected static RetrievalAlgorithm theInstance;
List<String> documentCollection;
Dictionary<String, BitArray> index;
private BooleanRetrieval()
: base("BooleanRetrieval")
{
}
public override String BuildIndex(string filepath)
{
documentCollection = new List<string>();
index = new Dictionary<string, BitArray>();
documentCollection.Add(filepath);
for(int i=0; i<documentCollection.Count; ++i)
{
StreamReader input = new StreamReader(documentCollection[i]);
var text = Regex.Split(input.ReadToEnd(), #"\W+").Distinct().ToArray();
foreach (String wordToIndex in text)
{
if (!index.ContainsKey(wordToIndex))
{
index.Add(wordToIndex, new BitArray(documentCollection.Count, false));
}
index[wordToIndex][i] = true;
}
}
return "Index " + index.Keys.Count + "words.";
}
public override String Start(String filepath, String[] search)
{
BitArray tempDecision = new BitArray(documentCollection.Count, true);
List<String> res = new List<string>();
foreach(String searchWord in search)
{
if (!index.ContainsKey(searchWord))
return "No documents found!";
tempDecision.And(index[searchWord]);
}
for (int i = 0; i < tempDecision.Count; ++i )
{
if (tempDecision[i] == true)
{
res.Add(documentCollection[i]);
}
}
return res.Count>0 ? res[0]: "Empty!";
}
public static RetrievalAlgorithm GetInstance()
{
Contract.Ensures(Contract.Result<RetrievalAlgorithm>() != null, "result is null.");
if (theInstance == null)
theInstance = new BooleanRetrieval();
theInstance.Executions++;
return theInstance;
}
}

Cold/warm start of .Net application is usually impacted by JIT time and disk access time to load assemblies.
For application that does a lot of disk IO very first access to data on disk will be much slower than on re-run for the same data due to caching content (also applies to assembly loading) if data is small enough to fit in memory cache for the disk.
First run of the task will be impacted by disk IO for assemblies and data, plus JIT time.
Second run of the same task without restart of application - just reading data from OS memory cache.
Second run of application - reading assemblies from OS memory cache and JIT again.