We Keep Coding

Correct usage of ReadOnlySequence<T> on implementing ReadOnlySequenceSegment<T>

I'm trying to implement a network library to complete my from scratch library Astron since a few days and I have a few questions. In order to handle Nagle correctly, I must have a frame parsing logic. So while doing that using the new pipeline API I noticed that output was provided as a read-only sequence, therefore I had to make a custom SequenceReader but I'm having some problems while testing it. I think I've misunderstood the new Span<T> and Memory<T> API :/
Is it relevant to mock the sequence segment?
In order to test as many behaviors as possible, I had to implement the ReadOnlySequenceSegment<T> abstract class, which I did using the Moq library to know when the next segment has been getting, here it is :
public class SequenceSegment : ReadOnlySequenceSegment<byte>
{
private readonly Mock<SequenceSegment> _mock;
private ReadOnlySequenceSegment<byte> _next;
public new ReadOnlySequenceSegment<byte> Next
{
get
{
var mockNext = _mock.Object.Next; // simulate get from the mock
return _next;
}
protected set => _next = value;
}
public SequenceSegment(ReadOnlyMemory<byte> memory)
{
Memory = memory;
_mock = new Mock<SequenceSegment>(memory);
}
public SequenceSegment Add(ReadOnlyMemory<byte> mem)
{
var segment = new SequenceSegment(mem)
{
RunningIndex = RunningIndex + Memory.Length
};
Next = segment;
return segment;
}
public void VerifyGetNext() => _mock.VerifyGet(ss => ss.Next);
}
As you can see I had to override the Next property with the new keyword which is a bad practice but I assume while testing it's fine?
Here is the test that doesn't pass :
[Fact]
public void TryRead_ShouldReturnTrue_OnSegmentSplitted()
{
var buffer = _createBuffer(); // int + int = 8bytes buffer
var firstSegment = new SequenceSegment(buffer.Slice(0, 3));
var secondSegment = firstSegment.Add(buffer.Slice(3, 5));
var input = new ReadOnlySequence<byte>(
firstSegment, 0,
secondSegment, 4);
var reader = new SequenceReader(input);
reader.TryRead(BinaryPrimitives.ReadInt32LittleEndian, out var firstValue); // throw here
reader.TryRead(BinaryPrimitives.ReadInt32LittleEndian, out var secondValue);
Assert.Equal(_firstValue, firstValue);
Assert.Equal(_secondValue, secondValue);
firstSegment.VerifyGetNext();
}
Test output :
Message: System.ArgumentOutOfRangeException : Specified argument was out of the range of valid values.
Parameter name: start
I commented the line that throws the exception on the test, so I assume my sequence logic is okay? Let's see the code of my SequenceReader with the line that throws commented :
public class SequenceReader
{
private const int _maxStackalloc = 128;
protected ReadOnlySequence<byte> _input;
public SequenceReader(ReadOnlySequence<byte> input) => _input = input;
public delegate T ReadDelegate<out T>(ReadOnlySpan<byte> src);
/// <summary>
/// Try to read a <see cref="T"/> with the <see cref="ReadDelegate{T}"/> specified as an arg.
/// The <see cref="SequenceReader"/> then advance the current position according to the size of <see cref="T"/>.
/// <see cref="T"/> must be a struct :
/// <see cref="byte"/>, <see cref="sbyte"/>, <see cref="bool"/>, <see cref="short"/>,
/// <see cref="ushort"/>, <see cref="int"/>, <see cref="uint"/>, <see cref="long"/>,
/// <see cref="ulong"/>, <see cref="float"/>, <see cref="double"/>, <see cref="decimal"/>,
/// </summary>
/// <typeparam name="T">The type to read.</typeparam>
/// <param name="read">The delegate to read the <see cref="T"/>. Must be a method from <see cref="BinaryPrimitives"/></param>
/// <param name="result">The result returned.</param>
/// <returns>Returns true if the read was successful, else returns false.</returns>
public unsafe bool TryRead<T>(ReadDelegate<T> read, out T result) where T : unmanaged
{
result = default;
var size = sizeof(T);
if (size > _maxStackalloc) return false;
if (size > _input.Length) return false;
if (_input.First.Length >= size)
result = read(_input.First.Span);
else
{
Span<byte> local = stackalloc byte[size];
_input.Slice(size).CopyTo(local); // throws at the slice
result = read(local);
}
_input = _input.Slice(size);
return true;
}
}
I already tried to change the line to _input.Slice(0, size) but nothing changed, also not on this successful test :
[Fact]
public void TryRead_ShouldReturnTrue_OnSegmentComplete()
{
var input = new ReadOnlySequence<byte>(_createBuffer());
var reader = new SequenceReader(input);
reader.TryRead(BinaryPrimitives.ReadInt32LittleEndian, out var firstValue);
reader.TryRead(BinaryPrimitives.ReadInt32LittleEndian, out var secondValue);
Assert.Equal(_firstValue, firstValue);
Assert.Equal(_secondValue, secondValue);
}
I'm wondering what I'm doing wrong, please if you have any idea it'd be awesome if you can help me!

C# - Odd Null Reference Exception during testing, why does this happen?

This references my last question which appears to have been abandoned. I am experiencing an odd "bug" if you will with C# and MS VS 2015. To reproduce the error, follow the steps:
Open console app project and copy paste code below.
Set a break point here:
First run code past break point, it works! :D
Then run code again but this time STOP at the break point and DRAG the executing statement cursor INTO the if statement from here:
to here:
Hit Continue and an NRE exception is thrown. Why does this happen? Is it just me? What is the technical explination for this?
CODE:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace testapp
{
class Program
{
static void Main(string[] args)
{
FILECollection randomCollection = new FILECollection();
// Fill with junk test data:
for(int i = 0; i<10; i++)
{
FILE junkfile = new FILE() { fileName = i.ToString(), folderName = i.ToString(), fileHashDigest = new byte[1] };
randomCollection.Add(junkfile);
}
if (true)
{
Console.WriteLine("testing this weird exception issue...");
FILE test;
test = new FILE();
test.fileName = "3";
test.folderName = "3";
test.fileHashDigest = new byte[1];
FILE exists = randomCollection.Where(f => f.fileName == test.fileName &&
f.fileHashDigest.SequenceEqual(test.fileHashDigest)).First();
}
}
}
public class FILE
{
public FILE() { _fileName = "";}
private string _fileName;
public string fileName
{
get
{
if (false)
return this._fileName.ToUpper();
else
return this._fileName;
}
set
{
if (false)
this._fileName = value.ToUpper();
else
this._fileName = value;
}
}
public string folderName { get; set; }
public byte[] fileHashDigest { get; set; }
}
public class FILECollection : IEnumerable<FILE>, ICollection<FILE>
{
private HashSet<FILE> svgHash;
private static List<FILE> PreallocationList;
public string FileName = "N/A";
/// <summary>
/// Default Constructor, will not
/// preallocate memory.
/// </summary>
/// <param name="PreallocationSize"></param>
public FILECollection()
{
this.svgHash = new HashSet<FILE>();
this.svgHash.Clear();
}
/// <summary>
/// Overload Constructor Preallocates
/// memory to be used for the new
/// FILE Collection.
/// </summary>
public FILECollection(int PreallocationSize, string fileName = "N/A", int fileHashDigestSize = 32)
{
FileName = fileName;
PreallocationList = new List<FILE>(PreallocationSize);
for (int i = 0; i <= PreallocationSize; i++)
{
byte[] buffer = new byte[fileHashDigestSize];
FILE preallocationSVG = new FILE()
{
fileName = "",
folderName = "",
fileHashDigest = buffer
};
PreallocationList.Add(preallocationSVG);
}
this.svgHash = new HashSet<FILE>(PreallocationList);
this.svgHash.Clear(); // Capacity remains unchanged until a call to TrimExcess is made.
}
/// <summary>
/// Add an FILE file to
/// the FILE Collection.
/// </summary>
/// <param name="svg"></param>
public void Add(FILE svg)
{
this.svgHash.Add(svg);
}
/// <summary>
/// Removes all elements
/// from the FILE Collection
/// </summary>
public void Clear()
{
svgHash.Clear();
}
/// <summary>
/// Determine if the FILE collection
/// contains the EXACT FILE file, folder,
/// and byte[] sequence. This guarantees
/// that the collection contains the EXACT
/// file you are looking for.
/// </summary>
/// <param name="item"></param>
/// <returns></returns>
public bool Contains(FILE item)
{
return svgHash.Any(f => f.fileHashDigest.SequenceEqual(item.fileHashDigest) &&
f.fileName == item.fileName &&
f.folderName == item.folderName);
}
/// <summary>
/// Determine if the FILE collection
/// contains the same file and folder name,
/// byte[] sequence is not compared. The file and folder
/// name may be the same but this does not guarantee the
/// file contents are exactly the same. Use Contains() instead.
/// </summary>
/// <param name="item"></param>
/// <returns></returns>
public bool ContainsPartially(FILE item)
{
return svgHash.Any(f => f.fileName == item.fileName &&
f.folderName == item.folderName);
}
/// <summary>
/// Returns the total number
/// of FILE files in the Collection.
/// </summary>
public int Count
{ get { return svgHash.Count(); } }
public bool IsReadOnly
{ get { return true; } }
public void CopyTo(FILE[] array, int arrayIndex)
{
svgHash.CopyTo(array, arrayIndex);
}
public bool Remove(FILE item)
{
return svgHash.Remove(item);
}
public IEnumerator<FILE> GetEnumerator()
{
return svgHash.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return svgHash.GetEnumerator();
}
}
}
I think either I am debugging in a terribly wrong way, or Microsoft should take a look at this. It's like future code is breaking current code...which is impossible!

OK here's my best guess..
First, as I mentioned in the comments, the exception doesn't occur if you comment out the line FILE exists = randomCollection.Where(f => f.fileName == test.fileName && f.fileHashDigest.SequenceEqual(test.fileHashDigest)).First()‌​;
Second, I noticed the same behavior can be reproduced with the following code:
if (true)
{
object o;
o = new object();
Func<bool> m = () => o == null;
}
i.e. the cause seems to be related to the variable being used in a lambda expression. So, looking at the same code snippet above in ILSpy I get the following:
Program.<>c__DisplayClass0_0 <>c__DisplayClass0_ = new Program.<>c__DisplayClass0_0();
<>c__DisplayClass0_.o = new object();
Func<bool> func = new Func<bool>(<>c__DisplayClass0_.<Main>b__0);
so my best guess is that the NullReferenceException refers to <>c__DisplayClass0_ intance being null - and I'm therefore inclined to believe that the stepping through the if(true) actually skipped the first line where <>c__DisplayClass0_ is instantiated

How to write a "filter" stream wrapper for XML?

I have some large XML feed files with illegal characters in them (0x1 etc). The files are third-party, I cannot change the process for writing them.
I would like to process these files using an XmlReader, but it blows up on these illegal characters.
I could read the files, filter out the bad characters, save them, then process them... but this is a lot of I/O, and it seems like it should be unnecessary.
What I would like to do is something like this:
using(var origStream = File.OpenRead(fileName))
using(var cleanStream = new CleansedXmlStream(origStream))
using(var streamReader = new StreamReader(cleanStream))
using(var xmlReader = XmlReader.Create(streamReader))
{
//do stuff with reader
}
I tried inheriting from Stream, but when I got to implementing the Read(byte[] buffer, int offset, int count) I lost some confidence. After all, I was planning on removing characters, so it seemed the count would be off, and I'd have to translate each byte to a char which seemed expensive (especially on large files) and I was unclear how this would work with a Unicode encoding, but the answers to my questions were not intuitively obvious.
When googling for "c# stream wrapper" or "c# filter stream" I am not getting satisfactory results. It's possible I'm using the wrong words or describing the wrong concept, so I'm hoping the SO community can square me away.
Using the example above, what would CleansedXmlStream look like?
Here's what my first attempt looked like:
public class CleansedXmlStream : Stream
{
private readonly Stream _baseStream;
public CleansedXmlStream(Stream stream)
{
this._baseStream = stream;
}
public new void Dispose()
{
if (this._baseStream != null)
{
this._baseStream.Dispose();
}
base.Dispose();
}
public override bool CanRead
{
get { return this._baseStream.CanRead; }
}
public override bool CanSeek
{
get { return this._baseStream.CanSeek; }
}
public override bool CanWrite
{
get { return this._baseStream.CanWrite; }
}
public override long Length
{
get { return this._baseStream.Length; }
}
public override long Position
{
get { return this._baseStream.Position; }
set { this._baseStream.Position = value; }
}
public override void Flush()
{
this._baseStream.Flush();
}
public override int Read(byte[] buffer, int offset, int count)
{
//what does this look like?
throw new NotImplementedException();
}
public override long Seek(long offset, SeekOrigin origin)
{
return this._baseStream.Seek(offset, origin);
}
public override void SetLength(long value)
{
this._baseStream.SetLength(value);
}
public override void Write(byte[] buffer, int offset, int count)
{
throw new NotSupportedException();
}
}

Inspired by #CharlesMager's comment, I ended up not making a Stream, but rather a StreamReader like so:
public class CleanTextReader : StreamReader
{
private readonly ILog _logger;
public CleanTextReader(Stream stream, ILog logger) : base(stream)
{
this._logger = logger;
}
public CleanTextReader(Stream stream) : this(stream, LogManager.GetLogger<CleanTextReader>())
{
//nothing to do here.
}
/// <summary>
/// Reads a specified maximum of characters from the current stream into a buffer, beginning at the specified index.
/// </summary>
/// <returns>
/// The number of characters that have been read, or 0 if at the end of the stream and no data was read. The number
/// will be less than or equal to the <paramref name="count" /> parameter, depending on whether the data is available
/// within the stream.
/// </returns>
/// <param name="buffer">
/// When this method returns, contains the specified character array with the values between
/// <paramref name="index" /> and (<paramref name="index + count - 1" />) replaced by the characters read from the
/// current source.
/// </param>
/// <param name="index">The index of <paramref name="buffer" /> at which to begin writing. </param>
/// <param name="count">The maximum number of characters to read. </param>
/// <exception cref="T:System.ArgumentException">
/// The buffer length minus <paramref name="index" /> is less than
/// <paramref name="count" />.
/// </exception>
/// <exception cref="T:System.ArgumentNullException"><paramref name="buffer" /> is null. </exception>
/// <exception cref="T:System.ArgumentOutOfRangeException">
/// <paramref name="index" /> or <paramref name="count" /> is
/// negative.
/// </exception>
/// <exception cref="T:System.IO.IOException">An I/O error occurs, such as the stream is closed. </exception>
public override int Read(char[] buffer, int index, int count)
{
try
{
var rVal = base.Read(buffer, index, count);
var filteredBuffer = buffer.Select(x => XmlConvert.IsXmlChar(x) ? x : ' ').ToArray();
Buffer.BlockCopy(filteredBuffer, 0, buffer, 0, count);
return rVal;
}
catch (Exception ex)
{
this._logger.Error("Read(char[], int, int)", ex);
throw;
}
}
/// <summary>
/// Reads a maximum of <paramref name="count" /> characters from the current stream, and writes the data to
/// <paramref name="buffer" />, beginning at <paramref name="index" />.
/// </summary>
/// <returns>
/// The position of the underlying stream is advanced by the number of characters that were read into
/// <paramref name="buffer" />.The number of characters that have been read. The number will be less than or equal to
/// <paramref name="count" />, depending on whether all input characters have been read.
/// </returns>
/// <param name="buffer">
/// When this method returns, this parameter contains the specified character array with the values
/// between <paramref name="index" /> and (<paramref name="index" /> + <paramref name="count" /> -1) replaced by the
/// characters read from the current source.
/// </param>
/// <param name="index">The position in <paramref name="buffer" /> at which to begin writing.</param>
/// <param name="count">The maximum number of characters to read. </param>
/// <exception cref="T:System.ArgumentNullException"><paramref name="buffer" /> is null. </exception>
/// <exception cref="T:System.ArgumentException">
/// The buffer length minus <paramref name="index" /> is less than
/// <paramref name="count" />.
/// </exception>
/// <exception cref="T:System.ArgumentOutOfRangeException">
/// <paramref name="index" /> or <paramref name="count" /> is
/// negative.
/// </exception>
/// <exception cref="T:System.ObjectDisposedException">The <see cref="T:System.IO.TextReader" /> is closed. </exception>
/// <exception cref="T:System.IO.IOException">An I/O error occurs. </exception>
public override int ReadBlock(char[] buffer, int index, int count)
{
try
{
var rVal = base.ReadBlock(buffer, index, count);
var filteredBuffer = buffer.Select(x => XmlConvert.IsXmlChar(x) ? x : ' ').ToArray();
Buffer.BlockCopy(filteredBuffer, 0, buffer, 0, count);
return rVal;
}
catch (Exception ex)
{
this._logger.Error("ReadBlock(char[], in, int)", ex);
throw;
}
}
/// <summary>
/// Reads the stream from the current position to the end of the stream.
/// </summary>
/// <returns>
/// The rest of the stream as a string, from the current position to the end. If the current position is at the end of
/// the stream, returns an empty string ("").
/// </returns>
/// <exception cref="T:System.OutOfMemoryException">
/// There is insufficient memory to allocate a buffer for the returned
/// string.
/// </exception>
/// <exception cref="T:System.IO.IOException">An I/O error occurs. </exception>
public override string ReadToEnd()
{
var chars = new char[4096];
int len;
var sb = new StringBuilder(4096);
while ((len = Read(chars, 0, chars.Length)) != 0)
{
sb.Append(chars, 0, len);
}
return sb.ToString();
}
}
My unit test looks like this:
[TestMethod]
public void CleanTextReaderCleans()
{
//arrange
var originalString = "The quick brown fox jumped over the lazy dog.";
var badChars = new string(new[] {(char) 0x1});
var concatenated = string.Concat(badChars, originalString);
//act
using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(concatenated)))
{
using (var reader = new CleanTextReader(stream))
{
var newString = reader.ReadToEnd().Trim();
//assert
Assert.IsTrue(originalString.Equals(newString));
}
}
}
... and usage looks like this:
using(var origStream = File.OpenRead(fileName))
using(var streamReader = new CleanTextReader(origStream))
using(var xmlReader = XmlReader.Create(streamReader))
{
//do stuff with reader
}
If anyone has suggestions for improvements, I'd be happy to hear them.

I tried #JeremyHolovacs stream implementation but it still was not sufficient for my use case:
using (var fstream = File.OpenRead(dlpath))
{
using (var zstream = new GZipStream(fstream, CompressionMode.Decompress))
{
using (var xstream = new CleanTextReader(zstream))
{
var ser = new XmlSerializer(typeof(MyType));
prods = ser.Deserialize(XmlReader.Create(xstream, new XmlReaderSettings() { CheckCharacters = false })) as MyType;
}
}
}
Somehow not all relevant overloads must have been implemented.
I adapted the class as follows and it works as expected:
public class CleanTextReader : StreamReader
{
public CleanTextReader(Stream stream) : base(stream)
{
}
public override int Read()
{
var val = base.Read();
return XmlConvert.IsXmlChar((char)val) ? val : (char)' ';
}
public override int Read(char[] buffer, int index, int count)
{
var ret = base.Read(buffer, index, count);
for (int i=0; i<ret; i++)
{
int idx = index + i;
if (!XmlConvert.IsXmlChar(buffer[idx]))
buffer[idx] = ' ';
}
return ret;
}
public override int ReadBlock(char[] buffer, int index, int count)
{
var ret = base.ReadBlock(buffer, index, count);
for (int i = 0; i < ret; i++)
{
int idx = index + i;
if (!XmlConvert.IsXmlChar(buffer[idx]))
buffer[idx] = ' ';
}
return ret;
}
}

The best overloaded method match for has some invalid arguments

I am using VS 2010 C#. My code is as follows:
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using MarketplaceWebServiceOrders;
using MarketplaceWebServiceOrders.Model;
namespace FetchNewOrdersJob
{
public class MarketplaceWebServiceOrders
{
private volatile bool isRunning;
private OrderFetcher orderFetcher;
private TimeSpan _checkOrdersInterval = TimeSpan.FromMinutes(15.0);
/// <summary>
/// Gets or sets the order check interval. Defaults to 15 minutes.
/// </summary>
public TimeSpan CheckOrdersInterval
{
get { return _checkOrdersInterval; }
set { _checkOrdersInterval = value; }
}
/// <summary>
/// Internal method to handle an order.
/// </summary>
protected virtual void HandleOrder(Order order)
{
Console.WriteLine("Processing Order:");
Console.WriteLine("---------------------------------------------------");
Console.WriteLine(order.ToString());
// Fetch the order items in each order
orderFetcher.FetchOrderItems(order.AmazonOrderId, delegate(OrderItem item)
{
Console.WriteLine("\tProcessing Order Item");
Console.WriteLine("\t---------------------------------------------------"); // Process order item here.
Console.WriteLine("\t" + item.ToString().Replace("\n", "\n\t"));
});
Console.WriteLine("=================================================");
Console.WriteLine();
}
/// <summary>
/// Method to continuously check orders over an interval, and list OrderItems for those Orders.
/// </summary>
private void OrdersJobThread(object obj)
{
orderFetcher.ProcessOrder += HandleOrder;
if (this.CheckOrdersInterval == TimeSpan.MinValue)
{
throw new ArgumentException("The CheckOrdersInterval TimeSpan cannot be zero.", "CheckOrdersInterval");
}
DateTime startCheckInterval = DateTime.Now.Subtract(CheckOrdersInterval);
// Continue forever until the isRunning flag is cleared.
while (isRunning)
{
try
{
// Check the orders for this interval.
DateTime checkInterval = startCheckInterval;
startCheckInterval = DateTime.Now.Subtract(TimeSpan.FromMinutes(3.0));
Console.WriteLine("Fetching orders from " + checkInterval.ToString() + " to " + startCheckInterval.ToString());
orderFetcher.FetchOrders(checkInterval, startCheckInterval);
// Wait for the next interval.
Console.WriteLine("Fetch complete. Sleeping until next interval.");
while (isRunning && DateTime.Now.Subtract(startCheckInterval) < CheckOrdersInterval)
{
Thread.Sleep(1000);
}
}
catch(Exception err)
{
Console.WriteLine("Error: " + err.Message + ". Orders job thread is exiting.");
isRunning = false;
}
}
}
/// <summary>
/// Sample code to invoke the OrderFetcher.
/// </summary>
/// <param name="service">MarketplaceWebServiceOrders object.</param>
/// <param name="sellerId">The seller Id.</param>
/// <param name="marketplaceIdList">List of marketplaces passed in to the GetOrders call.</param>
public static void InvokeOrderFetcherSample(
MarketplaceWebServiceOrders service,
string sellerId,
string [] marketplaceIdList
)
{
// Create a FetchOrderUpdates job with the default time span.
MarketplaceWebServiceOrders job = new MarketplaceWebServiceOrders();
job.isRunning = true;
job.orderFetcher = new OrderFetcher(service, sellerId, marketplaceIdList);
Thread jobThread = new Thread(job.OrdersJobThread);
jobThread.IsBackground = true;
jobThread.Start();
// Pause on the main thread for one hour or until the thread exits, then end the job.
jobThread.Join(1000 * 60 * 60);
job.isRunning = false;
// Block until the thread terminates to prevent any requests in progress from being aborted.
while (jobThread.IsAlive)
{
Thread.Sleep(1000);
}
}
}
}
I get two errors and can't seem to figure this out
Argument 1: cannot convert from 'FetchNewOrdersJob.MarketplaceWebServiceOrders' to 'MarketplaceWebServiceOrders.MarketplaceWebServiceOrders'
The best overloaded method match for 'MarketplaceWebServiceOrders.OrderFetcher.OrderFetcher(MarketplaceWebServiceOrders.MarketplaceWebServiceOrders, string, string[])' has some invalid arguments
The Line number of the error is 112: job.orderFetcher = new OrderFetcher(service, sellerId, marketplaceIdList);
OrderFetcher.cs
using System;
using System.Collections.Generic;
using System.Text;
using MarketplaceWebServiceOrders.Model;
namespace MarketplaceWebServiceOrders
{
/// <summary>
/// Sample helper class to Fetch Orders and OrderItems using the Amazon MWS Orders API.
/// </summary>
public class OrderFetcher
{
public delegate void RetriableMethodCall();
public delegate void ProcessOrderHandler(Order order);
public delegate void ProcessOrderItemHandler(OrderItem orderItem);
/// <summary>
/// Default amount of time, in milliseconds, to sleep if a request is throttled; default to 1 per 10 minutes.
/// </summary>
public const int DEFAULT_THROTTLED_WAIT_TIMEOUT = 10 * 60 * 1000;
/// <summary>
/// Default throttling limit for ListOrders calls; default to 1 per 12 seconds.
/// </summary>
private const int LIST_ORDERS_DEFAULT_THROTTLE_LIMIT = 12 * 1000;
/// <summary>
/// Default throttling limit for ListOrderItems calls; default to 1 per 100 minutes.
/// </summary>
private const int LIST_ORDER_ITEMS_DEFAULT_THROTTLE_LIMIT = 10 * 60 * 1000;
private MarketplaceWebServiceOrders mwsService;
private string mwsSellerId;
private string[] mwsMarketplaceIdList;
private DateTime lastServiceCall = DateTime.MinValue;
private ProcessOrderHandler _processOrder;
/// <summary>
/// Event called when an order is received for processing.
/// </summary>
public event ProcessOrderHandler ProcessOrder
{
add { _processOrder += value; }
remove { _processOrder -= value; }
}
/// <summary>
/// Creates a new instance of the OrderFetcherSample class.
/// </summary>
/// <param name="service"></param>
public OrderFetcher(MarketplaceWebServiceOrders service, string sellerId, string[] marketplaceIdList)
{
mwsService = service;
mwsSellerId = sellerId;
mwsMarketplaceIdList = marketplaceIdList;
}
/// <summary>
/// Fetches all orders created between the starting time and the server's
/// local system time minus two minutes.
/// <param name="startTime">The starting time period of orders to fetch.</param>
public void FetchOrders(DateTime startTime)
{
FetchOrders(startTime, DateTime.MinValue);
}
/// <summary>
/// Fetches all orders created in the given time period and processes them locally.
/// <param name="startTime">The starting time period of orders to fetch.</param>
/// <param name="endTime">The ending time period of orders to fetch.</param>
public void FetchOrders(DateTime startTime, DateTime endTime)
{
ListOrdersRequest request = new ListOrdersRequest();
request.CreatedAfter = startTime;
if (endTime != DateTime.MinValue)
{
request.CreatedBefore = endTime;
}
request.SellerId = mwsSellerId;
request.MarketplaceId = new MarketplaceIdList();
request.MarketplaceId.Id = new List<string>();
foreach (string marketplaceId in mwsMarketplaceIdList)
{
request.MarketplaceId.Id.Add(marketplaceId);
}
List<Order> orderList = new List<Order>();
ListOrdersResponse response = null;
OrderFetcher.InvokeRetriable(LIST_ORDERS_DEFAULT_THROTTLE_LIMIT, delegate()
{
response = mwsService.ListOrders(request);
ProcessOrders(response.ListOrdersResult.Orders.Order);
});
String nextTokenString = response.ListOrdersResult.NextToken;
while (!string.IsNullOrEmpty(nextTokenString))
{
// If NextToken is set, continue looping through the orders.
ListOrdersByNextTokenRequest nextRequest = new ListOrdersByNextTokenRequest();
nextRequest.NextToken = nextTokenString;
nextRequest.SellerId = mwsSellerId;
ListOrdersByNextTokenResponse nextResponse = null;
OrderFetcher.InvokeRetriable(LIST_ORDERS_DEFAULT_THROTTLE_LIMIT, delegate()
{
nextResponse = mwsService.ListOrdersByNextToken(nextRequest);
ProcessOrders(nextResponse.ListOrdersByNextTokenResult.Orders.Order);
});
nextTokenString = nextResponse.ListOrdersByNextTokenResult.NextToken;
}
}
/// <summary>
/// Method called by the FetchOrders method to process the orders.
/// </summary>
/// <param name="orders">List of orders returned by FetchOrders</param>
protected virtual void ProcessOrders(List<Order> orders)
{
foreach (Order order in orders)
{
if (_processOrder != null)
{
_processOrder(order);
}
}
}
/// <summary>
/// Fetches the OrderItems for the specified orderId.
/// </summary>
public void FetchOrderItems(string orderId, ProcessOrderItemHandler handler)
{
if (handler == null) throw new ArgumentNullException("handler");
ListOrderItemsRequest request = new ListOrderItemsRequest();
request.SellerId = mwsSellerId;
request.AmazonOrderId = orderId;
ListOrderItemsResponse response = null;
OrderFetcher.InvokeRetriable(LIST_ORDER_ITEMS_DEFAULT_THROTTLE_LIMIT, delegate()
{
response = mwsService.ListOrderItems(request);
foreach (OrderItem orderItem in response.ListOrderItemsResult.OrderItems.OrderItem)
{
handler(orderItem);
}
});
String nextTokenString = response.ListOrderItemsResult.NextToken;
while (!string.IsNullOrEmpty(nextTokenString))
{
// If NextToken is set, continue looping through the orders.
ListOrderItemsByNextTokenRequest nextRequest = new ListOrderItemsByNextTokenRequest();
nextRequest.NextToken = nextTokenString;
nextRequest.SellerId = mwsSellerId;
ListOrderItemsByNextTokenResponse nextResponse = null;
OrderFetcher.InvokeRetriable(LIST_ORDER_ITEMS_DEFAULT_THROTTLE_LIMIT, delegate()
{
nextResponse = mwsService.ListOrderItemsByNextToken(nextRequest);
foreach (OrderItem orderItem in nextResponse.ListOrderItemsByNextTokenResult.OrderItems.OrderItem)
{
handler(orderItem);
}
});
nextTokenString = nextResponse.ListOrderItemsByNextTokenResult.NextToken;
}
}
/// <summary>
/// Invokes a method in a retriable fashion.
/// </summary>
/// <param name="throttledWaitTime">The amount of time to wait if the request is throttled.</param>
/// <param name="method">The method to invoke.</param>
public static void InvokeRetriable(RetriableMethodCall method)
{
InvokeRetriable(DEFAULT_THROTTLED_WAIT_TIMEOUT, method);
}
/// <summary>
/// Invokes a method in a retriable fashion.
/// </summary>
/// <param name="throttledWaitTime">The amount of time to wait if the request is throttled.</param>
/// <param name="method">The method to invoke.</param>
public static void InvokeRetriable(int throttledWaitTime, RetriableMethodCall method)
{
bool retryRequest = false;
do
{
retryRequest = false;
try
{
// Perform some action
method.Invoke();
}
catch (MarketplaceWebServiceOrdersException ordersErr)
{
// If the request is throttled, wait and try again.
if (ordersErr.ErrorCode == "RequestThrottled")
{
Console.WriteLine("Request is throttled; waiting...");
retryRequest = true;
System.Threading.Thread.Sleep(throttledWaitTime);
}
else
{
// On any other error, re-throw the exception to be handled by the caller
throw;
}
}
} while (retryRequest);
}
}
}

I think this may have to do with the way in which you imported (I assume) your web service objects. If it's a WCF service, make sure your "Model" is referenced along the different projects, and when adding the service reference, make sure to check the option "Reuse existing Types".

You have classes MarketplaceWebServiceOrders both in current FetchNewOrdersJob and in MarketplaceWebServiceOrders namespaces (also last namespace have same name as your class). Looks like your OrderFetcher class is declared in MarketplaceWebServiceOrders namespace. And it expects as argument MarketplaceWebServiceOrders which is also declared in that namespace. But you are trying to pass MarketplaceWebServiceOrders class, which is declared in current FetchNewOrdersJob namespace.
Rename your FetchNewOrdersJob.MarketplaceWebServiceOrders class to avoid these conflicts.

Converting Directed Acyclic Graph (DAG) to tree

I'm trying to implement algoritm to convert Directed Acyclic Graph to Tree (for fun, learining, kata, name it). So I come up with the data structure Node:
/// <summary>
/// Represeting a node in DAG or Tree
/// </summary>
/// <typeparam name="T">Value of the node</typeparam>
public class Node<T>
{
/// <summary>
/// creats a node with no child nodes
/// </summary>
/// <param name="value">Value of the node</param>
public Node(T value)
{
Value = value;
ChildNodes = new List<Node<T>>();
}
/// <summary>
/// Creates a node with given value and copy the collection of child nodes
/// </summary>
/// <param name="value">value of the node</param>
/// <param name="childNodes">collection of child nodes</param>
public Node(T value, IEnumerable<Node<T>> childNodes)
{
if (childNodes == null)
{
throw new ArgumentNullException("childNodes");
}
ChildNodes = new List<Node<T>>(childNodes);
Value = value;
}
/// <summary>
/// Determines if the node has any child node
/// </summary>
/// <returns>true if has any</returns>
public bool HasChildNodes
{
get { return this.ChildNodes.Count != 0; }
}
/// <summary>
/// Travearse the Graph recursively
/// </summary>
/// <param name="root">root node</param>
/// <param name="visitor">visitor for each node</param>
public void Traverse(Node<T> root, Action<Node<T>> visitor)
{
if (root == null)
{
throw new ArgumentNullException("root");
}
if (visitor == null)
{
throw new ArgumentNullException("visitor");
}
visitor(root);
foreach (var node in root.ChildNodes)
{
Traverse(node, visitor);
}
}
/// <summary>
/// Value of the node
/// </summary>
public T Value { get; private set; }
/// <summary>
/// List of all child nodes
/// </summary>
public List<Node<T>> ChildNodes { get; private set; }
}
It's pretty straightforward. Methods:
/// <summary>
/// Helper class for Node
/// </summary>
/// <typeparam name="T">Value of a node</typeparam>
public static class NodeHelper
{
/// <summary>
/// Converts Directed Acyclic Graph to Tree data structure using recursion.
/// </summary>
/// <param name="root">root of DAG</param>
/// <param name="seenNodes">keep track of child elements to find multiple connections (f.e. A connects with B and C and B also connects with C)</param>
/// <returns>root node of the tree</returns>
public static Node<T> DAG2TreeRec<T>(this Node<T> root, HashSet<Node<T>> seenNodes)
{
if (root == null)
{
throw new ArgumentNullException("root");
}
if (seenNodes == null)
{
throw new ArgumentNullException("seenNodes");
}
var length = root.ChildNodes.Count;
for (int i = 0; i < length; ++i)
{
var node = root.ChildNodes[i];
if (seenNodes.Contains(node))
{
var nodeClone = new Node<T>(node.Value, node.ChildNodes);
node = nodeClone;
}
else
{
seenNodes.Add(node);
}
DAG2TreeRec(node, seenNodes);
}
return root;
}
/// <summary>
/// Converts Directed Acyclic Graph to Tree data structure using explicite stack.
/// </summary>
/// <param name="root">root of DAG</param>
/// <param name="seenNodes">keep track of child elements to find multiple connections (f.e. A connects with B and C and B also connects with C)</param>
/// <returns>root node of the tree</returns>
public static Node<T> DAG2Tree<T>(this Node<T> root, HashSet<Node<T>> seenNodes)
{
if (root == null)
{
throw new ArgumentNullException("root");
}
if (seenNodes == null)
{
throw new ArgumentNullException("seenNodes");
}
var stack = new Stack<Node<T>>();
stack.Push(root);
while (stack.Count > 0)
{
var tempNode = stack.Pop();
var length = tempNode.ChildNodes.Count;
for (int i = 0; i < length; ++i)
{
var node = tempNode.ChildNodes[i];
if (seenNodes.Contains(node))
{
var nodeClone = new Node<T>(node.Value, node.ChildNodes);
node = nodeClone;
}
else
{
seenNodes.Add(node);
}
stack.Push(node);
}
}
return root;
}
}
and test:
static void Main(string[] args)
{
// Jitter preheat
Dag2TreeTest();
Dag2TreeRecTest();
Console.WriteLine("Running time ");
Dag2TreeTest();
Dag2TreeRecTest();
Console.ReadKey();
}
public static void Dag2TreeTest()
{
HashSet<Node<int>> hashSet = new HashSet<Node<int>>();
Node<int> root = BulidDummyDAG();
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
var treeNode = root.DAG2Tree<int>(hashSet);
stopwatch.Stop();
Console.WriteLine(string.Format("Dag 2 Tree = {0}ms",stopwatch.ElapsedMilliseconds));
}
private static Node<int> BulidDummyDAG()
{
Node<int> node2 = new Node<int>(2);
Node<int> node4 = new Node<int>(4);
Node<int> node3 = new Node<int>(3);
Node<int> node5 = new Node<int>(5);
Node<int> node6 = new Node<int>(6);
Node<int> node7 = new Node<int>(7);
Node<int> node8 = new Node<int>(8);
Node<int> node9 = new Node<int>(9);
Node<int> node10 = new Node<int>(10);
Node<int> root = new Node<int>(1);
//making DAG
root.ChildNodes.Add(node2);
root.ChildNodes.Add(node3);
node3.ChildNodes.Add(node2);
node3.ChildNodes.Add(node4);
root.ChildNodes.Add(node5);
node4.ChildNodes.Add(node6);
node4.ChildNodes.Add(node7);
node5.ChildNodes.Add(node8);
node2.ChildNodes.Add(node9);
node9.ChildNodes.Add(node8);
node9.ChildNodes.Add(node10);
var length = 10000;
Node<int> tempRoot = node10;
for (int i = 0; i < length; i++)
{
var nextChildNode = new Node<int>(11 + i);
tempRoot.ChildNodes.Add(nextChildNode);
tempRoot = nextChildNode;
}
return root;
}
public static void Dag2TreeRecTest()
{
HashSet<Node<int>> hashSet = new HashSet<Node<int>>();
Node<int> root = BulidDummyDAG();
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
var treeNode = root.DAG2TreeRec<int>(hashSet);
stopwatch.Stop();
Console.WriteLine(string.Format("Dag 2 Tree Rec = {0}ms",stopwatch.ElapsedMilliseconds));
}
What is more, data structure need some improvment:
Overriding GetHash, toString, Equals, == operator
implementing IComparable
LinkedList is probably a better choice
Also, before the conversion there are certian thigs that need to be checked:
Multigraphs
If it's DAG (Cycles)
Diamnods in DAG
Multiple roots in DAG
All in all, it narrows down to a few questions:
How can I improve the conversion? Since this is a recurion it's possible to blow up the stack. I can add stack to memorize it. If I do continuation-passing style, will I be more efficient?
I feel that immutable data structure in this case would be better. Is it correct?
Is Childs the right name ? :)

Algorithm:
As you observed, some nodes appear twice in the output. If the node 2 had children, the whole subtree would appear twice. If you want each node to appear just once, replace
if (hashSet.Contains(node))
{
var nodeClone = new Node<T>(node.Value, node.Childs);
node = nodeClone;
}
with
if (hashSet.Contains(node))
{
// node already seen -> do nothing
}
I wouldn't be too worried about the size of the stack or performance of recursion. However, you could replace your Depth-first-search with Breadth-first-search which would result in nodes closer to the root being visited earlier, thus yielding a more "natural" tree (in your picture you already numbered the nodes in BFS order).
var seenNodes = new HashSet<Node>();
var q = new Queue<Node>();
q.Enqueue(root);
seenNodes.Add(root);
while (q.Count > 0) {
var node = q.Dequeue();
foreach (var child in node.Childs) {
if (!seenNodes.Contains(child )) {
seenNodes.Add(child);
q.Enqueue(child);
}
}
The algorithm handles diamonds and cycles.
Multiple roots
Just declare a class Graph which will contain all the vertices
class Graph
{
public List<Node> Nodes { get; private set; }
public Graph()
{
Nodes = new List<Node>();
}
}
Code:
the hashSet could be named seenNodes.
Instead of
var length = root.Childs.Count;
for (int i = 0; i < length; ++i)
{
var node = root.Childs[i];
write
foreach (var child in root.Childs)
In Traverse, the visitor is quite unnecessary. You could rather have a method which yields all the nodes of the tree (in the same order traverse does) and it is up to user to do whatever with the nodes:
foreach(var node in root.TraverseRecursive())
{
Console.WriteLine(node.Value);
}
If you override GetHashCode and Equals, the algorithm will no more be able to distinguish between two different Nodes with same value, which is probably not what you want.
I don't see any reason why LinkedList would be better here than List, except for the reallocations (Capacity 2,4,8,16,...) which List does when adding nodes.

you had better posted in CodeReview
Childs is wrong => Children
you don't have to use a HashSet, you could have easily used a List>, because checking references only is enough here. (and so no GetHashCode, Equals and operators overriding is needed)
easeier way is Serializing your class and then Deserializing it again into second objectwith XmlSerializer.
while Serialized and Deserialized, 1 object referenced 2 times will become 2 objects with different references.