We Keep Coding

C# Huge object initializer throws Stack Overflow error

I need to build an object which consists of almost 20000 nested objects (in multiple levels). Each object is a simple database entity with 1-5 fields or a list of entities. I am using inline object initializer to initiate my root object.
new OUTPUT() { XREF_CATALOG_MATERIALS = xrefCatalogMaterials.Find(x => x.MATERIAL.PART_NUM.Equals("xxxx")), FUNCTION = new FUNCTION() {...
I tried running on both x86 and x64 mode and in both cases I get stackoverflow exception. The same code and logic works fine on the other cases that my object is not that big (around 6000 nested objects)
Is there any way to increase .Net applicationheap size? any suggestion that I can use to solve that issue?

from that description you don't have a problem with heap size. you have a problem with stack size. looks like you're trying to invoke too many nested functions. every function call has an effect on stack.
Stack is much smaller than heap and it is relatively easy to overflow it. Easiest way is a recursion.
https://msdn.microsoft.com/en-us/library/system.stackoverflowexception(v=vs.110).aspx
StackOverflowException is thrown for execution stack overflow errors, typically in case of a very deep or unbounded recursion.

Push a stack onto another stack

In C#, is there a way to push one Stack onto another Stack without iterating through the stack elements? If not, is there a better data structure I should be using? In Java you can do:
stack1.addAll(stack2)
I was hoping to find the C# analogue...

0. Safe Solution - Extension Method
public static class Util {
public static void AddAll<T>(this Stack<T> stack1, Stack<T> stack2) {
T[] arr = new T[stack2.Count];
stack2.CopyTo(arr, 0);
for (int i = arr.Length - 1; i >= 0; i--) {
stack1.Push(arr[i]);
}
}
}
Probably the best is to create an extension method. Note that I am putting the first stack "on top" of the other stack so to speak by looping from arr.Length-1 to 0. So this code:
Stack<int> x = new Stack<int>();
x.Push(1);
x.Push(2);
Stack<int> y = new Stack<int>();
y.Push(3);
y.Push(4);
x.AddAll(y);
Will result in x being: 4,3,2,1. Which is what you would expect if you push 1,2,3,4. Of course, if you were to loop through your second stack and actually pop elements and then push those to the first stack, you would end up with 1,2,4,3. Again, modify the for loop as you see fit. Or you could add another parameter to specify which behavior you would like. I don't have Java handy, so I don't know what they do.
Having said that, you could do this, but I don't make any guarantees that it will continue to work. MS could always change the default behavior of how stack works when calling ToList. But, this is shorter, and on my machine with .NET 4.5 works the same as the extension method above:
1 Line Linq solution:
y.Reverse().ToList().ForEach(item => x.Push(item));

In your question, wanting to do this "without iterating through the stack elements" basically means a LinkedList-based stack where you would just join the first and last elements to combine stacks in constant time.
However, unless you've a very specific reason for using LinkedList, it's likely a better idea to just iterate over an array-based (List-based) stack elements.
As far as a specific implementation goes, you should probably clarify whether you want the second stack to be added to the first in the same stack order or to be reversed into the first stack by being popped out.

An addAll would just be a convenience method for a foreach loop that adds all of the items. There really isn't much you can do besides that:
foreach(var item in stack2)
stack1.Push(item);
If you do it particularly frequently you can add an extension method for it, for your own convenience.

This isn't meant to be done with the current .NET Stack implementation.
In order for the content of a Stack to be "grafted" onto the end of another Stack without iterating though its elements internal implementation details how the Stack class stores them in memory has to be known. Based on the principle of encapsulation this information is "officially" only know inside the Stack class itself. .NET's Stack does not expose methods to do this, so without using reflection there is no way to do it as the OP requested.
Conceivably you could use reflection to append to the internal array of one Stack the content of another Stack and also update the field that stores the stack length but this would be highly dependent on the implementation of the Stack class which could be changed in future versions of the framework without warning.
If you really need a Stack that can do this you could write your own Stack class from scratch or simply use another collection like ArrayList or LinkedList which have the method you want and add Push and Pop extension methods to them.

Is it possible to delete or insert frames into the C# call stack?

Using StackTrace I can get the stack frames. My question is whether it is possible to manipulate the call stack in C#? More specifically: Is it possible...
to insert a frame into the call stack? or
to delete a frame from it?

Stack is a very internal component of program's runtime. Having the ability to alter it would make it possible to:
Make almost unrestricted goto's. Something that is considered a very bad programming practice and is not possible even in PHP. Debugging such thing would be a complete mess.
Inject code on method returns. This may be useful, but is already covered by aspect oriented tools, like PostSharp or Spring.NET, which allow you to do this in clean and predictable way. And they cover much more cases than just method returns.
Thus, I do not think there is any programming language that allows you to manipulate stack explicitly. It would make things very messy and any benefits it may produce are already covered by Aspect-Oriented-Programming. I bet AOP will make you able to achieve what you want.

It is easy to manipulate the stack in Smalltalk, but I don't think it is possible in C#.

StackTrace class doesn't support this kind of manipulation. Frames are read only as per msdn (only GetFrame, GetFrames methods are available).
To see how the class is created by framework you can have a look directly into source code of the class.

No, it is not possible. The Stack Is An Implementation Detail.
Moreover, the ability to manipulate the stack will bring a LOT of security/consistency issues. This is why manipulating the stack is a bad practice/a security bug even when you can do it (in C, for example). For example,the infamous "buffer overflow" category of security issues spring exactly from this: the ability to insert a "goto", a new stack frame, into the code, so that upon return a malicious payload is executed instead of the legitimate calling function.
But then, why would you like to do it? Given a real goal, we could better reason how to do that, and suggest good alternatives

Yes, you can via Reflection.
Ultimately it's a string.
The following is an exception class for logging Javascript exceptions with the javascript stacktrace as a C# exception, preserving the original stack trace.
using System.Reflection;
public class JsException : Exception
{
static readonly FieldInfo _stackTraceString = typeof(Exception).GetField("_stackTraceString", BindingFlags.GetField | BindingFlags.NonPublic | BindingFlags.Instance);
public JsException(string error, string stack)
: base(error)
{
_stackTraceString.SetValue(this, stack);
}
}

"Magic" constants in C# like PHP has?

I am building a logging control for a C# project and would like to be able to call it with the name of the current source code File, Line, Class, Function, etc. PHP uses "magic constants" that have all of this info: http://php.net/manual/en/language.constants.predefined.php but I don't see anything like that in the C# compiler language.
Am I looking for something that doesn't exist?

Using the StackTrace/StackFrame classes, you can have your control find out where it's been called from, rather than passing it that information:
private static StringBuilder ListStack(out string sType)
{
StringBuilder sb = new StringBuilder();
sType = "";
StackTrace st = new StackTrace(true);
foreach (StackFrame f in st.GetFrames())
{
MethodBase m = f.GetMethod();
if (f.GetFileName() != null)
{
sb.AppendLine(string.Format("{0}:{1} {2}.{3}",
f.GetFileName(), f.GetFileLineNumber(),
m.DeclaringType.FullName, m.Name));
if (!string.IsNullOrEmpty(m.DeclaringType.Name))
sType = m.DeclaringType.Name;
}
}
return sb;
}
(I used this code to get the call stack of the currently executed method, so it does more than you asked for)

The StackTrace/StackFrame classes will give you quite a bit of this, though they can be quite expensive to construct.

You can ask the system for a stack trace, and you can use reflection. Details are coming.
__LINE__
__FILE__
__DIR__
__FUNCTION__ (does not really exist in C#)
__CLASS__
__METHOD__
__NAMESPACE__
This is a start:
http://www.csharp-examples.net/reflection-callstack/
http://www.csharp-examples.net/reflection-calling-method-name/
Assembly.GetExecutingAssembly().FullName
System.Reflection.MethodBase.GetCurrentMethod().Name
You will get better information in Debug (non-optimized) build. PhP might always have access to all that stuff, but it ain't the fastest gun on this planet. Play with it and let me know what is missing.

There are methods to get this type of data. It depends on what data you want.
__CLASS__ : If you want the current classname you'll need to use reflection.
__LINE__ : I'm not sure what "The current line number of the file" means, I'll take a guess and say it's how many lines in the file. That can be done by opening the file and doing a line count. This can be done via the File class, the FileInfo class may also work.
__DIR__ :Getting the directory of the file is done by using the DirectoryInfo class.
__FUNCTION__ and __METHOD__: Function name (method name), this can be retrieved via reflection.
__NAMESPACE__ :Namespace an be retrieved via reflection

Using Type, the best you can really do is get information about the current class. There is no means to get the file (though you should generally stick to one class per file), nor line number, nor function using Type.
Getting a type is simple, for example, this.getType(), or typeof(MyClass).
You can get the more specific details by generating a StackTrace object and retrieving a StackFrame from it, but doing so repeatedly is a bad idea.
I think a more important question is perhaps: why do you need them? For trace debugging, your output is supposedly temporary, so whether it reflects an accurate line number or not shouldn't matter (in fact, I rarely ever include a line number in trace debugging). Visual Studio is also very useful as a true step debugger. What do you really need File, Class, Function, and Line Number for?
Edit: For error checking, use exceptions like they're meant to be used: for exceptional (wrong) cases. The exception will generate a stack trace pointing you right at the problem.

Many of the previous responders have provided excellent information; however, I just wanted to point out that accessing the StackFrame is exorbitantly expensive and probably shouldn't be done except for special cases. Those cases being an extremely chatty verbose mode for debugging corner cases or error logging and for an error you probably already have an Exception instance which provides the StackTrace. Your best performance will be as Bring S suggested by using Type. Also as another design consideration logging to the console can slow your application down by several orders of magnitude depending on the volume of data to display. So if there is a console sink having the writer operating on a worker thread helps tremendously.

C#: How to test for StackOverflowException

Say you have a method that could potentially get stuck in an endless method-call loop and crash with a StackOverflowException. For example my naive RecursiveSelect method mentioned in this question.
Starting with the .NET Framework version 2.0, a StackOverflowException object cannot be caught by a try-catch block and the corresponding process is terminated by default. Consequently, users are advised to write their code to detect and prevent a stack overflow. For example, if your application depends on recursion, use a counter or a state condition to terminate the recursive loop.
Taking that information (from this answer) into account, since the exception can't be caught, is it even possible to write a test for something like this? Or would a test for this, if that failed, actually break the whole test-suite?
Note: I know I could just try it out and see what happens, but I am more interested in general information about it. Like, would different test frameworks and test runners handle this differently? Should I avoid a test like this even though it might be possible?

You would need to solve the Halting Problem! That would get you rich and famous :)

It's evil but you can spin it up in a new process. Launch the process from the unit test and wait for it to complete and check the result.

How about checking the number of frames on the stack in an assert statement?
const int MaxFrameCount = 100000;
Debug.Assert(new StackTrace().FrameCount < MaxFrameCount);
In your example from the related question this would be (The costly assert statement would be removed in the release build):
public static IEnumerable<T> SelectRecursive<T>(this IEnumerable<T> subjects, Func<T, IEnumerable<T>> selector)
{
const int MaxFrameCount = 100000;
Debug.Assert(new StackTrace().FrameCount < MaxFrameCount);
// Stop if subjects are null or empty
if(subjects == null || !subjects.Any())
yield break;
// For each subject
foreach(var subject in subjects)
{
// Yield it
yield return subject;
// Then yield all its decendants
foreach (var decendant in SelectRecursive(selector(subject), selector))
yield return decendant;
}
}
It's not a general test though, as you need to expect it to happen, plus you can only check the frame count and not the actual size of the stack. It is also not possible to check whether another call will exceed stack space, all that you can do is roughly estimate how many calls in total will fit on your stack.

The idea is to keep track of how deeply a recursive funcion is nested, so that it doesn't use too much stack space. Example:
string ProcessString(string s, int index) {
if (index > 1000) return "Too deeply nested";
s = s.Substring(0, index) + s.Substring(index, 1).ToUpper() + s.Substring(index + 1);
return ProcessString(s, index + 1);
}
This of course can't totally protect you from stack overflows, as the method can be called with too little stack space left to start with, but it makes sure that the method doesn't singelhandedly cause a stack overflow.

We cannot have a test for StackOverflow because this is the situation when there is no more stack left for allocation, the application would exit automatically in this situation.

If you are writing library code that somebody else is going to use, stack overflows tends to be a lot worse than other bugs because the other code can't just swallow the StackOverflowException; their entire process is going down.
There's no easy way to write a test that expects and catches a StackOverflowException, but that's not the behavior you want to be testing!
Here's some tips for testing your code doesn't stack overflow:
Parallelize your test runs. If you have a separate test suite for the stack overflow cases, then you'll still get results from the other tests if one test runner goes down. (Even if you don't separate your test runs, I'd consider stack overflows to be so bad that it's worth crashing the whole test runner if it happens. Your code shouldn't break in the first place!)
Threads may have different amounts of stack space, and if somebody is calling your code you can't control how much stack space is available. While the default for 32 bit CLR is 1MB stack size and 64 bit is 2MB stack size, be aware that web servers will default to a much smaller stack. Your test code could use one of the Thread constructors that takes a smaller stack size if you want to verify your code won't stack overflow with less avaliable space.
Test every different build flavor that you ship (Release and Debug? with or without debugging symbols? x86 and x64 and AnyCPU?) and platforms you'll support (64 bit? 32 bit? 64 bit OS running 32 bit? .NET 4.5? 3.5? mono?). The actual size of the stack frame in the generated native code could be different, so what breaks on one machine might not break on another.
Since your tests might pass on one build machine but fail on another, ensure that if it starts failing it doesn't block checkins for your entire project!
Once you measure how few iterations N cause your code to stack overflow, don't just test that number! I'd test a much larger number (50 N?) of iterations doesn't stack overflow (so you don't get tests that pass on one build server but fail on another).
Think about every possible code path where a function can eventually later call itself. Your product might prevent the X() -> X() -> X() -> ... recursive stack overflow, but what if there is a X() -> A() -> B() -> C() -> ... -> W() -> X() -> A() -> ... recursive case that is still broken?
PS: I don't have more details about this strategy, but apparently if your code hosts the CLR then you can specify that stack overflow only crashes the AppDomain?

First and foremost I think the method should handle this and make sure it does not recurse too deep.
When this check is done, I think the check should be tested - if anything - by exposing the maximum depth reached and assert it is never larger than the check allows.