We Keep Coding

Where do I have memory leaks and how to fix it? Why memory consumption increases?

I am struggling a few days already with a problem of growing memory consumption by console application in .Net Core 2.2, and just now I ran out of ideas what else I could improve.
Im my application I have a method that triggers StartUpdatingAsync method:
public MenuViewModel()
{
if (File.Exists(_logFile))
File.Delete(_logFile);
try
{
StartUpdatingAsync("basic").GetAwaiter().GetResult();
}
catch (ArgumentException aex)
{
Console.WriteLine($"Caught ArgumentException: {aex.Message}");
}
Console.ReadKey();
}
StartUpdatingAsync creates 'repo' and instance is getting from DB a list of objects to be updated (around 200k):
private async Task StartUpdatingAsync(string dataType)
{
_repo = new DataRepository();
List<SomeModel> some_list = new List<SomeModel>();
some_list = _repo.GetAllToBeUpdated();
await IterateStepsAsync(some_list, _step, dataType);
}
And now, within IterateStepsAsync we are getting updates, parsing them with existing data and updating DB. Inside of each while I was creating new instances of all new classes and lists, to be sure that old ones are releasing memory, but it didnt help. Also I was GC.Collect() at the end of the method, what also is not helping. Please note, that method below triggers lots of parralel Tasks, but they supposed to be disposed within it, am I right?:
private async Task IterateStepsAsync(List<SomeModel> some_list, int step, string dataType)
{
List<Area> areas = _repo.GetAreas();
int counter = 0;
while (counter < some_list.Count)
{
_repo = new DataRepository();
_updates = new HttpUpdates();
List<Task> tasks = new List<Task>();
List<VesselModel> vessels = new List<VesselModel>();
SemaphoreSlim throttler = new SemaphoreSlim(_degreeOfParallelism);
for (int i = counter; i < step; i++)
{
int iteration = i;
bool skip = false;
if (dataType == "basic" && (some_list[iteration].Mmsi == 0 || !some_list[iteration].Speed.HasValue)) //if could not be parsed with "full"
skip = true;
tasks.Add(Task.Run(async () =>
{
string updated= "";
await throttler.WaitAsync();
try
{
if (!skip)
{
Model model= await _updates.ScrapeSingleModelAsync(some_list[iteration].Mmsi);
while (Updating)
{
await Task.Delay(1000);
}
if (model != null)
{
lock (((ICollection)vessels).SyncRoot)
{
vessels.Add(model);
scraped = BuildData(model);
}
}
}
else
{
//do nothing
}
}
catch (Exception ex)
{
Log("Scrape error: " + ex.Message);
}
finally
{
while (Updating)
{
await Task.Delay(1000);
}
Console.WriteLine("Updates for " + counter++ + " of " + some_list.Count + scraped);
throttler.Release();
}
}));
}
try
{
await Task.WhenAll(tasks);
}
catch (Exception ex)
{
Log("Critical error: " + ex.Message);
}
finally
{
_repo.UpdateModels(vessels, dataType, counter, some_list.Count, _step);
step = step + _step;
GC.Collect();
}
}
}
Inside of the method above, we are calling _repo.UpdateModels, where is updated DB. I tryed two approaches, with using EC Core and SqlConnection. Both with similar results. Below you can find both of them.
EF Core
internal List<VesselModel> UpdateModels(List<Model> vessels, string dataType, int counter, int total, int _step)
{
for (int i = 0; i < vessels.Count; i++)
{
Console.WriteLine("Parsing " + i + " of " + vessels.Count);
Model existing = _context.Vessels.Where(v => v.id == vessels[i].Id).FirstOrDefault();
if (vessels[i].LatestActivity.HasValue)
{
existing.LatestActivity = vessels[i].LatestActivity;
}
//and similar parsing several times, as above
}
Console.WriteLine("Saving ...");
_context.SaveChanges();
return new List<Model>(_step);
}
SqlConnection
internal List<VesselModel> UpdateModels(List<Model> vessels, string dataType, int counter, int total, int _step)
{
if (vessels.Count > 0)
{
using (SqlConnection connection = GetConnection(_connectionString))
using (SqlCommand command = connection.CreateCommand())
{
connection.Open();
StringBuilder querySb = new StringBuilder();
for (int i = 0; i < vessels.Count; i++)
{
Console.WriteLine("Updating " + i + " of " + vessels.Count);
//PARSE
VesselAisUpdateModel existing = new VesselAisUpdateModel();
if (vessels[i].Id > 0)
{
//find existing
}
if (existing != null)
{
//update for basic data
querySb.Append("UPDATE dbo." + _vesselsTableName + " SET Id = '" + vessels[i].Id+ "'");
if (existing.Mmsi == 0)
{
if (vessels[i].MMSI.HasValue)
{
querySb.Append(" , MMSI = '" + vessels[i].MMSI + "'");
}
}
//and similar parsing several times, as above
querySb.Append(" WHERE Id= " + existing.Id+ "; ");
querySb.AppendLine();
}
}
try
{
Console.WriteLine("Sending SQL query to " + counter);
command.CommandTimeout = 3000;
command.CommandType = CommandType.Text;
command.CommandText = querySb.ToString();
command.ExecuteNonQuery();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
connection.Close();
}
}
}
return new List<Model>(_step);
}
Main problem is, that after tenths/hundreds of thousands of updated models my console application memory consumption increases continuously. And I have no idea why.
SOLUTION my problem was inside of ScrapeSingleModelAsync method, where I was using incorrectly HtmlAgilityPack, what I could debug thanks to cassandrad.

Your code is messy, with huge amount of different objects with unknown lifetime. It's hardly possible to figure out the problem just looking at it.
Consider using profiling tools, for example Visual Studio's Diagnostic Tools, they will help you to find what objects are living too long in the heap. Here is overview of its functions related to memory profiling. Highly recomended to be read.
In short, you need to take two snapshots and look at what objects are taking the most memory. Let's look at simple example.
int[] first = new int[10000];
Console.WriteLine(first.Length);
int[] secod = new int[9999];
Console.WriteLine(secod.Length);
Console.ReadKey();
Take the first snapshot when your function works at least once. In my case, I took snapshot when the first huge space has been alocated.
After that, let your app be working some time so the difference in memory usage become noticeable, take the second memory snapshot.
You'll notice that another snapshot is added with info about how much is the difference. To get more specific info, click on one or another blue label of the latest snapshot to open snapshots comparison.
Following my example, we can see that there is change in count of int arrays. By default int[] wasn't visible in the table, so I had to uncheck Just My Code in filtration options.
So, this is what needs to be done. After you figure out what objects increase in count or size over time, you can locate where these objects are create and optimize this operation.

Display gstreamer-sharp video stream in WinForms

I am looking for an example/help for displaying a gstreamer-sharp feed in a WinForms application.
I am using VS 2012 and have the "glue" project built for this version of VS. I also have glib-sharp, gstreamer-sharp installed and referenced by my project. I have the gstreamer bin directory set as my project's working directory.
If have the following code in a button click handler I get the GStreamer D3D video sink test Window which pops-up on top of my Form.
Gst.Application.Init();
var pipeline = Parse.Launch(#"videotestsrc ! videoconvert ! autovideosink");
pipeline.SetState(State.Playing);
I would like to get the stream displaying within my application on a Panel or PictureBox I'm thinking.
Thanks!

The code below will get the test stream to display on a WinForm's panel control.
internal enum videosinktype { glimagesink, d3dvideosink, dshowvideosink, directdrawsink}
static Element mVideoTestSource, mVideoCaps, mVideoSink, mAppQueue, mVideoConv;
static Gst.App.AppSink mAppSink;
static System.Threading.Thread mMainGlibThread;
static GLib.MainLoop mMainLoop; // GLib's Main Loop
private const videosinktype mCfgVideosinkType = videosinktype.dshowvideosink;
private ulong mHandle;
private Gst.Video.VideoSink mGlImageSink;
private Gst.Pipeline mCurrentPipeline = null;
private void InitGStreamerPipeline()
{
//Assign Handle to prevent Cross-Threading Access
mHandle = (ulong)videoDisplay.Handle;
//Init Gstreamer
Gst.Application.Init();
GtkSharp.GstreamerSharp.ObjectManager.Initialize();
mMainLoop = new GLib.MainLoop();
mMainGlibThread = new System.Threading.Thread(mMainLoop.Run);
mMainGlibThread.Start();
#region BuildPipeline
switch (mCfgVideosinkType)
{
case videosinktype.glimagesink:
mGlImageSink = (Gst.Video.VideoSink)Gst.ElementFactory.Make("glimagesink", "glimagesink");
break;
case videosinktype.d3dvideosink:
mGlImageSink = (Gst.Video.VideoSink)Gst.ElementFactory.Make("d3dvideosink", "d3dvideosink");
//mGlImageSink = (Gst.Video.VideoSink)Gst.ElementFactory.Make("dshowvideosink", "dshowvideosink");
break;
case videosinktype.dshowvideosink:
mGlImageSink = (Gst.Video.VideoSink)Gst.ElementFactory.Make("dshowvideosink", "dshowvideosink");
break;
case videosinktype.directdrawsink:
mGlImageSink = (Gst.Video.VideoSink)Gst.ElementFactory.Make("directdrawsink", "directdrawsink");
break;
default:
break;
}
mVideoTestSource = ElementFactory.Make("videotestsrc", "videotestsrc0");
mVideoConv = ElementFactory.Make("videoconvert", "vidconvert0");
mVideoSink = ElementFactory.Make("autovideosink", "sink0");
mCurrentPipeline = new Gst.Pipeline("pipeline");
mCurrentPipeline.Add(mVideoTestSource, mVideoConv, mVideoSink, mGlImageSink);
if (!mVideoTestSource.Link(mVideoSink))
if (mUdpcSrc.Link(mVideoSink))
{
System.Diagnostics.Debug.WriteLine("Elements could not be linked");
}
#endregion
//subscribe to bus & bussync msgs
Bus bus = mCurrentPipeline.Bus;
bus.AddSignalWatch();
bus.Message += HandleMessage;
Bus bus = mCurrentPipeline.Bus;
bus.EnableSyncMessageEmission();
bus.SyncMessage += new SyncMessageHandler(bus_SyncMessage);
//play the stream
var setStateRet = mCurrentPipeline.SetState(State.Null);
System.Diagnostics.Debug.WriteLine("SetStateNULL returned: " + setStateRet.ToString());
setStateRet = mCurrentPipeline.SetState(State.Ready);
System.Diagnostics.Debug.WriteLine("SetStateReady returned: " + setStateRet.ToString());
setStateRet = mCurrentPipeline.SetState(Gst.State.Playing);
}
/// <summary>
///
/// </summary>
/// <remarks>
/// Indeed the application needs to set its Window identifier at the right time to avoid internal Window creation
/// from the video sink element. To solve this issue a GstMessage is posted on the bus to inform the application
/// that it should set the Window identifier immediately.
///
/// API: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideooverlay.html
/// </remarks>
/// <param name="o"></param>
/// <param name="args"></param>
private void bus_SyncMessage(object o, SyncMessageArgs args)
{
//Convenience function to check if the given message is a "prepare-window-handle" message from a GstVideoOverlay.
System.Diagnostics.Debug.WriteLine("bus_SyncMessage: " + args.Message.Type.ToString());
if (Gst.Video.Global.IsVideoOverlayPrepareWindowHandleMessage(args.Message))
{
Element src = (Gst.Element)args.Message.Src;
#if DEBUG
System.Diagnostics.Debug.WriteLine("Message'prepare-window-handle' received by: " + src.Name + " " + src.ToString());
#endif
if (src != null && (src is Gst.Video.VideoSink | src is Gst.Bin))
{
// Try to set Aspect Ratio
try
{
src["force-aspect-ratio"] = true;
}
catch (PropertyNotFoundException) { }
// Try to set Overlay
try
{
Gst.Video.VideoOverlayAdapter overlay_ = new Gst.Video.VideoOverlayAdapter(src.Handle);
overlay_.WindowHandle = mHandle;
overlay_.HandleEvents(true);
}
catch (Exception ex) { System.Diagnostics.Debug.WriteLine("Exception thrown: " + ex.Message); }
}
}
}
private void HandleMessage (object o, MessageArgs args)
{
var msg = args.Message;
//System.Diagnostics.Debug.WriteLine("HandleMessage received msg of type: {0}", msg.Type);
switch (msg.Type)
{
case MessageType.Error:
//
GLib.GException err;
string debug;
System.Diagnostics.Debug.WriteLine("Error received: " + msg.ToString());
//msg.ParseError (out err, out debug);
//if(debug == null) { debug = "none"; }
//System.Diagnostics.Debug.WriteLine ("Error received from element {0}: {1}", msg.Src, err.Message);
//System.Diagnostics.Debug.WriteLine ("Debugging information: "+ debug);
break;
case MessageType.StreamStatus:
Gst.StreamStatusType status;
Element theOwner;
msg.ParseStreamStatus(out status, out theOwner);
System.Diagnostics.Debug.WriteLine("Case SteamingStatus: status is: " + status + " ; Ownder is: " + theOwner.Name);
break;
case MessageType.StateChanged:
//System.Diagnostics.Debug.WriteLine("Case StateChanged: " + args.Message.ToString());
State oldState, newState, pendingState;
msg.ParseStateChanged(out oldState, out newState, out pendingState);
if (newState == State.Paused)
args.RetVal = false;
System.Diagnostics.Debug.WriteLine("Pipeline state changed from {0} to {1}: ; Pending: {2}", Element.StateGetName(oldState), Element.StateGetName(newState), Element.StateGetName(pendingState));
break;
case MessageType.Element:
System.Diagnostics.Debug.WriteLine("Element message: {0}", args.Message.ToString());
break;
default:
System.Diagnostics.Debug.WriteLine("HandleMessage received msg of type: {0}", msg.Type);
break;
}
args.RetVal = true;
}

Have a look at the GstVideoOverlay interface here:
https://developer.gnome.org/gst-plugins-libs/stable/gst-plugins-base-libs-gstvideooverlay.html
You will have to adapt to the C# syntax I guess, but there is a good amount of explaination and code samples for this.

You can have a look at https://cgit.freedesktop.org/gstreamer/gstreamer-sharp/tree/samples/VideoOverlay.cs You need to adapt this code to use the window Handle of WinForms.

Unable to receive data from PHP in unity

I have this script to interact with my database which is on mySql
string dataSubmitURL = "http://localhost/HighScore/AddScore.php?";
string dataGetURL = "http://localhost/HighScore/GetScore.php?";
// Use this for initialization
void Start () {
StartCoroutine(GetScores());
}
// Update is called once per frame
void Update () {
}
IEnumerator PostScores(string playerName, int score) {
string submitURL = dataSubmitURL + "name="+ playerName + "&score=" + score;
print(submitURL);
WWW submitData = new WWW(submitURL);
yield return submitData;
if (submitData.error != null)
{
Debug.LogError("Error occur when Submitting data : " + submitData.error);
Debug.LogError("Error occur when Submitting data : " + submitData.text);
Debug.LogError("Error occur when Submitting data : " + submitData.responseHeaders);
//submitData.text
}
else {
print(" Submitted");
}
IEnumerator GetScores() {
WWW getData = new WWW(dataGetURL);
yield return getData;
if (getData.error != null)
{
Debug.LogError("There was an error getting the high score: " + getData.error);
}
else {
print(getData.text);
}
}
}
But the problem is i am getting
There was an error getting the high score: Empty reply from server
Although these both urls
string dataSubmitURL = "http://localhost/HighScore/AddScore.php?";
string dataGetURL = "http://localhost/HighScore/GetScore.php?";
working fine in browser when i directly put it. I also added this crossDomain.xml in my root folder so that it can accessible to all. What I am doing wrong.
<?xml version="1.0"?>
<cross-domain-policy>
<allow-access-from domain="*"/>
</cross-domain-policy>

I don't know what was the problem all the code is correct and it works by changing the IP Address only.
string dataSubmitURL = "http://YouripAddress/HighScore/AddScore.php?";
string dataGetURL = "http://YouripAddress/HighScore/GetScore.php?";

Create PowerShell PSObject in C# cmdlet

New to C# but experienced in PowerShell. Taking over someone else's code. Writing a compiled PowerShell module, and trying to figure out how to create an object based on returned data. Right now, code returns a string:
ServerResponse<UCCSiteModel> requestModel = this.MakeRequest("/site/api/", "site", "GET", this.Credentials, queryString);
StringBuilder builder = new StringBuilder();
if (requestModel != null && requestModel.Response != null)
{
builder.AppendLine("SiteID: " + requestModel.Response.SiteID);
builder.AppendLine("Identity: " + requestModel.Response.SiteName);
builder.AppendLine("Site Code: " + requestModel.Response.SiteCode);
builder.AppendLine("Contact Name: " + requestModel.Response.ContactName);
builder.AppendLine("Contact Number: " + requestModel.Response.ContactNumber);
builder.AppendLine("Contact Email Address: " + requestModel.Response.ContactEmailAddress);
builder.AppendLine("Address: " + requestModel.Response.Address);
builder.AppendLine("City: " + requestModel.Response.City);
builder.AppendLine("State: " + requestModel.Response.State);
builder.AppendLine("Post Code: " + requestModel.Response.PostCode);
builder.AppendLine("Time Zone: " + requestModel.Response.Timezone);
builder.AppendLine("Longitude: " + requestModel.Response.longitude);
builder.AppendLine("Latitude: " + requestModel.Response.latitude);
this.WriteResponse(requestModel, builder.ToString());
}
How do I create an object from requestModel.Response to send back to PowerShell instead of the string? When writing PowerShell, I would normally use New-Object PsObject, and then Add-Member. Not sure how to do that in C#, or what it's called (so I can search). Anyone?

You can mirror the behavior of Add-Member simply by calling Add() on the Members property of your PSObject (I would change the property names to CamelCase for ease of accessibility in PowerShell):
if (requestModel != null && requestModel.Response != null)
{
PSObject responseObject = new PSObject();
responseObject.Members.Add(new PSNoteProperty("SiteID", requestModel.Response.SiteID));
responseObject.Members.Add(new PSNoteProperty("Identity", requestModel.Response.SiteName));
// and so on etc...
responseObject.Members.Add(new PSNoteProperty("Latitude", requestModel.Response.latitude));
this.WriteObject(responseObject);
}

Without knowing all the details I can't say this is the best plan, but here is what I would do. You could define a class and then return it. So you would create a new class such as below:
public class RequestResponse {
public int SiteID { get; set;}
public string Identity { get; set; }
other fields...
}
Next, in the code you posted, you would create the object and then fill the properties of the class.
var response = new RequestResponse();
if (requestModel != null && requestModel.Response != null)
{
response.SiteID = requestModel.Response.SiteID;
response.Identity = requestModel.Response.Identity ;
fill in other fields...
this.WriteResponse(requestModel, response);
}
I hope this gets you started in the right direction.
Wade

Massive differences in reexecution

I wrote a little C# application that indexes a book and executes a boolean textretrieval algorithm on the index. The class at the end of the post showes the implementation of both, building the index and executing the algorithm on it.
The code is called via a GUI-Button in the following way:
private void Execute_Click(object sender, EventArgs e)
{
Stopwatch s;
String output = "-----------------------\r\n";
String sr = algoChoice.SelectedItem != null ? algoChoice.SelectedItem.ToString() : "";
switch(sr){
case "Naive search":
output += "Naive search\r\n";
algo = NaiveSearch.GetInstance();
break;
case "Boolean retrieval":
output += "boolean retrieval\r\n";
algo = BooleanRetrieval.GetInstance();
break;
default:
outputTextbox.Text = outputTextbox.Text + "Choose retrieval-algorithm!\r\n";
return;
}
output += algo.BuildIndex("../../DocumentCollection/PilzFuehrer.txt") + "\r\n";
postIndexMemory = GC.GetTotalMemory(true);
s = Stopwatch.StartNew();
output += algo.Start("../../DocumentCollection/PilzFuehrer.txt", new String[] { "Pilz", "blau", "giftig", "Pilze" });
s.Stop();
postQueryMemory = GC.GetTotalMemory(true);
output += "\r\nTime elapsed:" + s.ElapsedTicks/(double)Stopwatch.Frequency + "\r\n";
outputTextbox.Text = output + outputTextbox.Text;
}
The first execution of Start(...) runs about 700µs, every rerun only takes <10µs.
The application is compiled with Visual Studio 2010 and the default 'Debug' buildconfiguration.
I experimentad a lot to find the reason for that including profiling and different implementations but the effect always stays the same.
I'd be hyppy if anyone could give me some new ideas what I shall try or even an explanation.
class BooleanRetrieval:RetrievalAlgorithm
{
protected static RetrievalAlgorithm theInstance;
List<String> documentCollection;
Dictionary<String, BitArray> index;
private BooleanRetrieval()
: base("BooleanRetrieval")
{
}
public override String BuildIndex(string filepath)
{
documentCollection = new List<string>();
index = new Dictionary<string, BitArray>();
documentCollection.Add(filepath);
for(int i=0; i<documentCollection.Count; ++i)
{
StreamReader input = new StreamReader(documentCollection[i]);
var text = Regex.Split(input.ReadToEnd(), #"\W+").Distinct().ToArray();
foreach (String wordToIndex in text)
{
if (!index.ContainsKey(wordToIndex))
{
index.Add(wordToIndex, new BitArray(documentCollection.Count, false));
}
index[wordToIndex][i] = true;
}
}
return "Index " + index.Keys.Count + "words.";
}
public override String Start(String filepath, String[] search)
{
BitArray tempDecision = new BitArray(documentCollection.Count, true);
List<String> res = new List<string>();
foreach(String searchWord in search)
{
if (!index.ContainsKey(searchWord))
return "No documents found!";
tempDecision.And(index[searchWord]);
}
for (int i = 0; i < tempDecision.Count; ++i )
{
if (tempDecision[i] == true)
{
res.Add(documentCollection[i]);
}
}
return res.Count>0 ? res[0]: "Empty!";
}
public static RetrievalAlgorithm GetInstance()
{
Contract.Ensures(Contract.Result<RetrievalAlgorithm>() != null, "result is null.");
if (theInstance == null)
theInstance = new BooleanRetrieval();
theInstance.Executions++;
return theInstance;
}
}

Cold/warm start of .Net application is usually impacted by JIT time and disk access time to load assemblies.
For application that does a lot of disk IO very first access to data on disk will be much slower than on re-run for the same data due to caching content (also applies to assembly loading) if data is small enough to fit in memory cache for the disk.
First run of the task will be impacted by disk IO for assemblies and data, plus JIT time.
Second run of the same task without restart of application - just reading data from OS memory cache.
Second run of application - reading assemblies from OS memory cache and JIT again.