We Keep Coding

Max parallel tasks in Console App, and how to handle "Error while copying content to stream" when calling API

I'm working on an integration that handles several large datasets (around 3M records or so, growing daily) which are pulling from a vendor OData API in chunks. When I run it/debug locally, the integration will run to completion without errors. However, as soon as I put it out on our PROD server, it will sometimes fail with "Error while copying content to stream" on a given chunk of data. I'm having trouble figuring out the reason, and could use some help.
Each request is wrapped in a using statement and should be disposing of itself cleanly. Usually, it does. Again, this all works perfectly on my local machine, but the server has a bit of trouble after it's into pulling millions of records. I have added extra logging at virtually every step to catch the specific error, but it's still not clear how to handle (or avoid) this error. It isn't consistent at all... totally intermittent and inconsistent.
The code where the error throws:
using (WebRequestHandler webRequestHandler = new WebRequestHandler())
{
using (HttpClient httpClient = new HttpClient(webRequestHandler))
{
httpClient.Timeout = TimeSpan.FromMinutes(15);
ConfigureJsonClient(httpClient, syncConfig.ApiEndpoint, syncConfig.ApiAuthKey);
apiCommand = "http://foo/with/filtering";
responseMessage = Task.Run(async () => await httpClient.GetAsync(apiCommand)
.ConfigureAwait(true)).Result;
}
}
The error:
API Error Occurred - RETRYING... Foo: 2 | API CALL: https://foo... | EXCEPTION: One or more errors occurred. | INNER EXCEPTION: Error while copying content to a stream. | STACKTRACE: at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at FooNamespace.Processor.GetDataFromApi(SyncConfig syncConfig, String collection, List`1 nonDatedCollectionList, Int32 skipIndex, Int32 retryCount, Int32 taskId)
I am also trying to run as many parallel tasks as I can to pull these large datasets in pre-defined chunks (data pages) using skip/take. No matter what I do, even locally, the max number of Tasks I can spin up in parallel is 40. Is there a reason for this limitation?
Is it possible the parallel tasks are conflicting with each other somehow when hitting the API? These should appear as completely separate, isolated calls...
Any and all help is much appreciated.

I've got a similar exception Error while copying content to a stream. On the inner exception I've got System.IO.IOException: The decryption operation failed. I found that the server is sending the content compressed without the header content-encoding: gzip. Change de default value of the AutomaticDecompression property on HttpClientHandler solve the problem.
HttpClientHandler httpClientHandler = new HttpClientHandler();
httpClientHandler.AutomaticDecompression = DecompressionMethods.All;
using (var client = new HttpClient(httpClientHandler))

How to throw a List<CustomExceptionObj> Exception at once

I have a wcf service, for which I now have made a change to keep compiling my exceptions when they occur into a list, and at the end, throw this list as an Error response. Now single errors is fine. I'm able to throw that and get the output. But throwing the list as an error is proving to be a little challenging.
-What I have tried till now: using AggregateException. I tried sniffing around for a proper implementation. So far, came up with this:
throw new AggregateException("Multiple Errors Occured", ExceptionsList)
Also,
List<CustomExceptionObject> ThrowException(List<CustomExceptionObject> ExceptionList)
{
AggregateException eggExp = new AggregateException(ExceptionList);
throw new eggExp;
}
I keep getting Unhandled Exception error using these method.
Any insight is appreciated.
UPDATE: The error message that I keep getting is -
An exception of type 'System.AggregateException' occurred in XYZ.dll but was not handled in user code.
Keep in mind that throwing a single object of my CustomExceptionObject throws a proper error response in SOAP UI. I can't seem to pull of a list of these exception.

An aggregate exception is the proper way to throw an exception that is a collection of exceptions.
throw new AggregateException("Multiple Errors Occured", ExceptionsList)

You could use this pattern:
public void DoStuff()
{
var ex = new List<Exception>();
try
{
DoSomethingThatThrowsFooException();
DoSomethingElseThatThrowsFooException();
DoSomethingThatThrowsBarException();
}
cath(FooException e)
{
ex.Add(e);
}
if (ex.Count>0)
throw new AggregateException(ex);
}
The BarException will not be caught and not be included in the AggregateException. Ultimately it could lead to an UnhandledException if not caught anywhere else.

So I overcame the issue with a hack. I'm not sure if it is the BEST way. But to get to the desirable outcome that I need, this works just fine.
STEP 1:
Create a class that has a property of List<CustomExceptionObject> type.
STEP 2:
Now when you have to throw an exception, Set this property with the incoming list,
and cast it as a FaultException type, complete with some dummy FaultReason and FaultCode.
throw new FaultException<ListOfExceptionsClass>(listofExceptions, new FaultReason("Here is the List of All exceptions"), Faultcode);
This will give a Fault Object in the response with a neat list of all exceptions occurred.

SerialPort.BaseStream.ReadAsync throws IOException

When I try to send data to the serial port, it works fine. But when I try to read one byte from the port, reading operation immediately throws IOException (Additional information: Reached the end of the file). I use the following code:
byte[] buff = new byte[1];
await port.BaseStream.ReadAsync(buff, 0, 1); // Exception arises here
I tried to set port.BaseStream.ReadTimeout = 1000 after configuring the port, but I'm still getting the same exception.
Call stack of main thread and another one. Btw, not sure if this is incorrect, but local variables values in non-main thread never appear in the debugging window, neither before exception arises nor after.

Trouble finding/fixing a NullReferenceException, Windows WCF Service Application

First of all, this is my first post so apologies in advance if I'm neglecting to include anything. Please let me know if I'm doing it wrong.
I'm developing a Windows 8 Store app and having it consume a WCF service I've made. My end goal is to pull data from an SQL database and present in graphs.
For the sake of isolating the problem, I dropped the sql portion (even though the problem exists regardless) and created a table from scratch, so my Service class is as follows:
public class Service1 : IService1
{
public DataSet querySql()
{
try
{
DataSet ds = new DataSet();
DataTable dt = ds.Tables.Add();
dt.Columns.Add("Test", typeof(string));
dt.Rows.Add(1, "Test1");
return ds;
}
From there I attempt to retrieve this data within the windows application like so:
private async void LoadChartContents()
{
// create proxy instance
DatabaseService.Service1Client serviceClient = new DatabaseService.Service1Client();
// async call to wcf method to get returned data
DatabaseService.querySqlResponse ds = await serviceClient.querySqlAsync();
There is more to that method, but it was breaking at the last line of the block I posted.
Here is where I also ran into problems with the debugger. I tried changing a few settings in hopes of getting closer to the problem, but I'm not sure if it helped. When it breaks now I get taken to App.g.i.cs where global::System.Diagnostics.Debugger.Break() is highlighted in
#if DEBUG && !DISABLE_XAML_GENERATED_BREAK_ON_UNHANDLED_EXCEPTION
UnhandledException += (sender, e) =>
{
if (global::System.Diagnostics.Debugger.IsAttached) global::System.Diagnostics.Debugger.Break();
};
#endif
The exception in the Locals tab says "Object reference not set to an instance of an object"
InnerException is null, Source is System.ServiceModel, and StackTrace is a hefty:
http://pastebin.com/QiXCPci1
What that looks like is a null pointer exception, but I'm having trouble figuring out where it's occurring
I have a feeling that the ds in DatabaseService.querySqlResponse ds = await serviceClient.querySqlAsync(); is null because that was where it was breaking earlier, but I can't be sure and I can't see why that would be null.
Again, sorry if this isn't clear. Please let me know what other information should be included. Thank you in advance for your help :)
EDIT:
Thanks to gunr2171, I was able to pinpoint the error further. Now it says:
Fatal Exception'1' occurred.
A first chance exception of type 'System.ServiceModel.FaultException'1' occurred in mscorlib.dll
EDIT:
Alas, I am still lost, but I will update this post if I make any progress.

IMethodCallMessage Args.Get() - FatalExecutionEngineError or AccessViolationException

This line of code causes the exceptions shown below:
object arg = methodCallMessage.Args[i];
private static List<ParameterInformation> GetParameterInfoList( IMethodCallMessage methodCallMessage )
{
List<ParameterInformation> parameterInformationList = new List<ParameterInformation>();
// Note: This works even if a parameter's value is null.
for( int i = 0 ; i < methodCallMessage.ArgCount ; i++ )
{
string argName = methodCallMessage.GetArgName(i);
object arg = methodCallMessage.Args[i];
var parameterInformation = new ParameterInformation(argName, arg);
parameterInformationList.Add(parameterInformation);
}
return parameterInformationList;
}
Exception:
FatalExecutionEngineError: The runtime has encountered a fatal error. The address of the error was at 0x71b97e8d, on thread 0x2ef4. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.
Or sometimes this exception:
AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
This happens on more than one machine.
In the call stack, when I show external code, this is at the top:
mscorlib.dll!System.Runtime.Remoting.Messaging.Message.Args.get() + 0x5 bytes.
Any ideas why this is happening, or how to fix it?
Note: The code that calls this method has a lock placed around it, so it shouldn't be a threading/timing issue.

This is more of a work-around than an actual fix, but it works for this situation. The problem only occurs when being done in an async way. The code that started the process had this wrapped around it:
Task.Factory.StartNew(() =>
When I removed that, and just processed things synchronously, the problem went away.