We Keep Coding

Duplicate dictionary key on task list

I'm trying to generate a zip file of pdfs asynchronously to speed things up as follows:
var files = new Dictionary<string, byte[]>();
var fileTasks = new List<Task<Library.Models.Helpers.File>>();
foreach (var i in groups)
{
var task = Task.Run(async () =>
{
var fileName = $"{i.Key.Title.Replace('/', '-')} - Records.pdf";
ViewBag.GroupName= i.Key.Title;
var html = await this.RenderViewAsync("~/Views/Report/_UserRecordsReport.cshtml", i.ToList(), true);
return await _fileUtilityService.HtmlToPDF2(html, null, fileName);
});
fileTasks.Add(task);
}
var completedTaskFiles = await Task.WhenAll(fileTasks);
foreach(var item in completedTaskFiles)
{
files.Add($"{item.FileName}", item.FileResult);
}
return _fileUtilityService.GenerateZIP(files);
I'm generating all my html to pdf file tasks and waiting for them to be completed - then trying to synchronously loop through the completed tasks and add them to my dictionary for zipping but I keep getting the following error:
An item with the same key has already been added
There is no duplicate key in the list of items being added.
EDIT - so the current idea is that because its a scoped service, thats why i'm running into thread issues (attached the file utility service for information)
public class FileUtilityService : IFileUtilityService
{
private readonly IHttpClientFactory _clientFactory;
public FileUtilityService(IHttpClientFactory clientFactory)
{
public async Task<byte[]> HtmlToPDF(string html = null, string url = null)
{
try
{
byte[] res = null;
if (html is null && url != null)
{
var client = _clientFactory.CreateClient();
var requestResp = await client.GetAsync(url);
using var sr = new StreamReader(await requestResp.Content.ReadAsStreamAsync());
html = HttpUtility.HtmlDecode(await sr.ReadToEndAsync());
}
using(var ms = new MemoryStream())
{
HtmlConverter.ConvertToPdf(html, ms);
res = ms.ToArray();
}
return res;
}catch(Exception ex)
{
throw ex;
}
}
public async Task<Library.Models.Helpers.File> HtmlToPDF(string html = null, string url = null, string fileName = "")
{
return new Library.Models.Helpers.File() { FileName = fileName, FileResult = await HtmlToPDF(html, url) };
}

Handling Parallel POST to API

I have a API Post method that takes is a string which represents a Bae64 string of bytes from a word document that the API converts to PDF. My test client sends multiple documents, each on its own task, to the API to be converted. The problem is with concurrency and writing the files. I end up with a file in use since the calls are parallel. I have tried a lot of different way to block the conversion process until a document is converted but none of it has worked. Everything works fine if it's jsut a single file being converted but as soon as it's 2 or more, the problem happens. Can anyone guide me in the correct direction to solve this issue?
API:
[HttpPost]
public async Task<SimpleResponse> Post([FromBody]string request)
{
var response = new SimpleResponse();
Task t = Task.Factory.StartNew(async () =>
{
try
{
Converter convert = new Converter();
var result = await convert.CovertDocToPDF(request, WebConfigurationManager.AppSettings["tempDocPath"], WebConfigurationManager.AppSettings["tempPdfPath"]);
response.Result = result;
response.Success = true;
}
catch (Exception ex)
{
response.Exception = ex;
response.Success = false;
response.Errors = new List<string>();
response.Errors.Add(string.Format("{0}, {1}", ex.Message, ex.InnerException?.Message ?? ""));
}
});
t.Wait();
return response;
}
Conversion code
public Task<string> CovertDocToPDF(string blob, string tempDocPath, string tempPdfPath)
{
try
{
// Convert blob back to bytes
byte[] bte = Convert.FromBase64String(blob);
// Process and return blob
return Process(bte, tempDocPath, tempPdfPath);
}
catch (Exception Ex)
{
throw Ex;
}
}
private async Task<string> Process(byte[] bytes, string tempDocPath, string tempPdfPath)
{
try
{
string rs = RandomString(16, true);
tempDocPath = tempDocPath + rs + ".docx";
tempPdfPath = tempPdfPath + rs + ".pdf";
// This is where the problem happens with concurrent calls. I added
// the try catch when the file is in use to generate a new
// filename but the error still happens.
try
{
// Create a temp file
File.WriteAllBytes(tempDocPath, bytes);
}
catch (Exception Ex)
{
rs = RandomString(16, true);
tempDocPath = tempDocPath + rs + ".docx";
tempPdfPath = tempPdfPath + rs + ".pdf";
File.WriteAllBytes(tempDocPath, bytes);
}
word.Application app = new word.Application();
word.Document doc = app.Documents.Open(tempDocPath);
doc.SaveAs2(tempPdfPath, word.WdSaveFormat.wdFormatPDF);
doc.Close();
app.Quit(); // Clean up the word instance.
// Need the bytes to return the blob
byte[] pdfFileBytes = File.ReadAllBytes(tempPdfPath);
// Delete temp files
File.Delete(tempDocPath);
File.Delete(tempPdfPath);
// return blob
return Convert.ToBase64String(pdfFileBytes);
}
catch (Exception Ex)
{
throw Ex;
}
}
Client:
public async void btnConvert_Click(object sender, EventArgs e)
{
var response = await StartConvert();
foreach (SimpleResponse sr in response)
{
if (sr.Success)
{
byte[] bte = Convert.FromBase64String(sr.Result.ToString());
string rs = RandomString(16, true);
string pdfFileName = tempPdfPath + rs + ".pdf";
if (File.Exists(pdfFileName))
{
File.Delete(pdfFileName);
}
System.IO.File.WriteAllBytes(pdfFileName, bte);
}
else
{
}
}
}
private async Task<IEnumerable<SimpleResponse>> StartConvert()
{
var tasks = new List<Task<SimpleResponse>>();
foreach (string s in docPaths)
{
byte[] bte = File.ReadAllBytes(s);
tasks.Add(ConvertDocuments(Convert.ToBase64String(bte)));
}
return (await Task.WhenAll(tasks));
}
private async Task<SimpleResponse> ConvertDocuments(string requests)
{
using (var client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true }))
{
client.BaseAddress = new Uri(BaseApiUrl);
client.DefaultRequestHeaders.Add("Accept", "application/json");
// Add an Accept header for JSON format.
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));//application/json
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, BaseApiUrl + ApiUrl);
var data = JsonConvert.SerializeObject(requests);
request.Content = new StringContent(data, Encoding.UTF8, "application/json");
HttpResponseMessage response1 = await client.PostAsync(BaseApiUrl + ApiUrl, request.Content).ConfigureAwait(false);
var response = JsonConvert.DeserializeObject<SimpleResponse>(await response1.Content.ReadAsStringAsync());
return response;
}
}
Random String Generator
public string RandomString(int size, bool lowerCase = false)
{
var builder = new StringBuilder(size);
// Unicode/ASCII Letters are divided into two blocks
// (Letters 65–90 / 97–122):
// The first group containing the uppercase letters and
// the second group containing the lowercase.
// char is a single Unicode character
char offset = lowerCase ? 'a' : 'A';
const int lettersOffset = 26; // A...Z or a..z: length = 26
for (var i = 0; i < size; i++)
{
var #char = (char)_random.Next(offset, offset + lettersOffset);
builder.Append(#char);
}
return lowerCase ? builder.ToString().ToLower() : builder.ToString();
}

First, get rid of Task.Factory.StartNew ... t.Wait() - you don't need an additional task, the root level method is async and your blocking Wait just spoils the benefits of async by blocking synchronously.
Second, like a comment suggested above, the file name random string generator is most likely to be not really random. Either do not supply anything to the seed value of your pseudo-random gen, or use something like Environment.TickCount which should be sufficient for this. Guid.NewGuid() will work too.
Another good option for temp files is Path.GetTempFileName (also generates an empty file for you): https://learn.microsoft.com/en-us/dotnet/api/system.io.path.gettempfilename?view=netstandard-2.0
[HttpPost]
public async Task<SimpleResponse> Post([FromBody]string request)
{
var response = new SimpleResponse();
try
{
...
var result = await convert.CovertDocToPDF(...);
...
}
catch (Exception ex)
{
...
}
return response;
}

Based on your code it seems that you have a "faulty" random string generator for file name (I would say _random.Next is a suspect, possibly some locking and/or "app wide" instance could fix the issue). You can use Guid.NewGuid to create random part of file name (which in theory can have collisions also but in most practical cases should be fine) or Path.GetTempFileName:
var rs = Guid.NewGuid().ToString("N");
tempDocPath = tempDocPath + rs + ".docx";
tempPdfPath = tempPdfPath + rs + ".pdf";

C# ClientWebSocket not receiving all packets?

I've built a windows service that subscribes to around 10,000 stock tickers in real-time using ClientWebSocket. If I subscribe to 1,000 tickers I receive all the data points as I should (receiving few hundred messages a second), as soon as I get up to 2,000 tickers I don't seem to be receiving the data I should be, 10,000 (receiving thousands of messages a second) its even worse. I've run comparison reports and it looks like I'm losing up to 60% of the packets. I've talked to polygon (the provider of the real-time data) about this issue and they claim their Socket is a firehose and everything that should go out, goes out, and that none of their other clients are complaining. So the only logical thing here would to be to assume its my code, or some limitation. Maybe it's the Task portion of the Receive method? Maybe window's has a max task limitation and I'm exceeding it.
I've also tested this on a high powered dedicated server with 10gb connection so it doesnt seem to be a connection or hardware limitation.
I've also by passed my BlockingCollection cache and the problem still persisted.
Hopefully one of you has some insight, thank you!
Here's my code:
public static ConcurrentDictionary<string, TradeObj> TradeFeed = new ConcurrentDictionary<string, TradeObj>();
public static ConcurrentDictionary<string, QuoteObj> QuoteFeed = new ConcurrentDictionary<string, QuoteObj>();
public static ConcurrentDictionary<string, AggObj> AggFeed = new ConcurrentDictionary<string, AggObj>();
public static BlockingCollection<byte[]> packets = new BlockingCollection<byte[]>();
private static void Start(string[] args)
{
try
{
Polygon.StartSub();
int HowManyConsumers = 2;
for (int i = 0; i < HowManyConsumers; i++)
{
Task.Factory.StartNew(Polygon.ConsumePackets);
}
} catch(Exception e)
{
Console.WriteLine(e.Message);
}
Console.ReadKey();
}
public static async Task StartSub()
{
do
{
using (var socket = new ClientWebSocket())
try
{
// socket.Options.KeepAliveInterval = TimeSpan.Zero;
var Connection = "wss://socket.polygon.io/stocks";
await socket.ConnectAsync(new Uri(Connection), CancellationToken.None);
Console.WriteLine("Websocket opened to Polygon.");
await Send(socket, "{\"action\":\"auth\",\"params\":\""+ConfigurationManager.AppSettings["PolygonAPIToken"]+"\"}");
List<List<string>> batches = new List<List<string>>();
for (int i = 0; i < FeedCache.Tickers.Count(); i += 500)
{
var tempList = new List<string>();
tempList.AddRange(FeedCache.Tickers.Skip(i).Take(500));
batches.Add(tempList);
}
int bNum = 0;
string[] quoteStrings = new string[batches.Count()];
foreach (var tList in batches)
{
var tQuery = "";
tQuery = tQuery + "T." + string.Join(",T.", tList.ToArray());
tQuery = tQuery + ",A." + string.Join(",A.", tList.ToArray());
tQuery = tQuery + ",Q." + string.Join(",Q.", tList.ToArray());
quoteStrings[bNum] = tQuery;
bNum++;
}
for (int i = 0; i < quoteStrings.Count(); i++)
{
string SubscribeString = "{\"action\":\"subscribe\",\"params\":\"" + quoteStrings[i] + "\"}";
await Send(socket, SubscribeString);
}
await Receive(socket);
}
catch (Exception ex)
{
Console.WriteLine($"ERROR - {ex.Message}");
Console.WriteLine(ex.ToString());
}
} while (true);
}
static async Task Send(ClientWebSocket socket, string data)
{
var segment = new ArraySegment<byte>(Encoding.UTF8.GetBytes(data));
await socket.SendAsync(segment, WebSocketMessageType.Text, true, CancellationToken.None);
}
static async Task Receive(ClientWebSocket socket)
{
do {
WebSocketReceiveResult result;
var buffer = new ArraySegment<byte>(new byte[2000]);
using (var ms = new MemoryStream())
{
do
{
result = await socket.ReceiveAsync(buffer, CancellationToken.None);
ms.Write(buffer.Array, buffer.Offset, result.Count);
} while (!result.EndOfMessage);
if (result.MessageType == WebSocketMessageType.Close)
{
await socket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "Closed in server by the client", CancellationToken.None);
Console.WriteLine("Socket disconnecting, trying to reconnect.");
await StartSub();
}
else
{
packets.Add(ms.ToArray());
}
}
} while (true);
}
public static async void ConsumePackets()
{
foreach (var buffer in packets.GetConsumingEnumerable())
{
using (var ms = new MemoryStream(buffer))
{
ms.Seek(0, SeekOrigin.Begin);
using (var reader = new StreamReader(ms, Encoding.UTF8))
{
var data = await reader.ReadToEndAsync();
try
{
var j = JArray.Parse(data);
if (j != null)
{
string id = (string)j[0]["ev"];
switch (id)
{
case "T":
AddOrUpdateTrade((string)j[0]["sym"], j);
break;
case "Q":
AddOrUpdateQuote((string)j[0]["sym"], j);
break;
case "A":
AddOrUpdateAgg((string)j[0]["sym"], j);
break;
}
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
}
}
}
public static void AddOrUpdateTrade(string ticker, JArray data)
{
TradeFeed.AddOrUpdate(ticker, new TradeObj {
LastPrice = (double)data[0]["p"],
TradeCount = 1
}, (key, existingVal) =>
{
return new TradeObj {
LastPrice = (double)data[0]["p"],
TradeCount = existingVal.TradeCount + 1,
PriceDirection = (double)data[0]["p"] < existingVal.LastPrice ? "D" : "U"
};
});
}
public static void AddOrUpdateAgg(string ticker, JArray data)
{
AggFeed.AddOrUpdate(ticker, new AggObj
{
TickVolume = (long)data[0]["v"],
VolumeShare = (long)data[0]["av"],
OpenPrice = (double)data[0]["op"],
TickAverage = (double)data[0]["a"],
VWAP = (double)data[0]["vw"],
TickClosePrice = (double)data[0]["c"],
TickHighPrice = (double)data[0]["h"],
TickLowPrice = (double)data[0]["l"],
TickOpenPrice = (double)data[0]["o"]
}, (key, existingVal) =>
{
return new AggObj
{
TickVolume = (long)data[0]["v"],
VolumeShare = (long)data[0]["av"],
OpenPrice = (double)data[0]["op"],
TickAverage = (double)data[0]["a"],
VWAP = (double)data[0]["vw"],
TickClosePrice = (double)data[0]["c"],
TickHighPrice = (double)data[0]["h"],
TickLowPrice = (double)data[0]["l"],
TickOpenPrice = (double)data[0]["o"]
};
});
}
public static void AddOrUpdateQuote(string ticker, JArray data)
{
QuoteFeed.AddOrUpdate(ticker, new QuoteObj
{
BidPrice = (double)data[0]["bp"],
BidSize = (double)data[0]["bs"],
AskPrice = (double)data[0]["ap"],
AskSize = (double)data[0]["as"]
}, (key, existingVal) =>
{
return new QuoteObj
{
BidPrice = (double)data[0]["bp"],
BidSize = (double)data[0]["bs"],
AskPrice = (double)data[0]["ap"],
AskSize = (double)data[0]["as"]
};
});
}

HTTP CONNECT stream.ReadToEnd takes infinite time

I'm trying to build a class which should be able to use HTTP CONNECT proxy tunnels.
My current code hangs at DoHttpConnect() - "var retvar = await sr.ReadToEndAsync();" and never reaches Console.WriteLine(retvar);
Connection class:
public class HttpConnect : BaseProxyModule
{
public HttpConnect(HostPortCollection proxyInformation) : base(proxyInformation)
{
}
public override async Task<Stream> OpenStreamAsync(HostPortCollection dstSrv)
{
var socket = new TcpClient();
if (await SwallowExceptionUtils.TryExec(() => socket.ConnectAsync(_proxyInformation.Addr, _proxyInformation.Port)) && socket.Connected)
{
var nStream = socket.GetStream();
var cmd = ProxyCommandBuildUtils.GenerateHttpConnectCommand(dstSrv.Host, dstSrv.Port);
await nStream.WriteAsync(cmd, 0, cmd.Length);
var sr = new StreamReader(nStream);
var cLine = await sr.ReadLineAsync();
var fLineElem = cLine.Split(' ');
if (fLineElem.Length >= 1 && fLineElem[1] == "200")
{
await sr.ReadLineAsync();
return nStream;
}
}
return null;
}
internal class ProxyCommandBuildUtils
{
private const string HTTP_PROXY_CONNECT_CMD = "CONNECT {0}:{1} HTTP/1.1\r\nHost: {0}\r\n\r\n";
public static byte[] GenerateHttpConnectCommand(string host, int port)
{
string connectCmd = String.Format(CultureInfo.InvariantCulture, HTTP_PROXY_CONNECT_CMD, host, port.ToString(CultureInfo.InvariantCulture));
return connectCmd.GetBytes();
}
}
Usage:
public static void Main(string[] args)
{
DoHttpConnect().Wait();
Debugger.Break();
}
public static async Task DoHttpConnect()
{
//var hCon = new HttpConnect(new HostPortCollection("5.135.195.166", 3128)); //Doesn't work..
var hCon = new HttpConnect(new HostPortCollection("109.75.213.146", 53281)); //Doesn't work either :/
var pStream = await hCon.OpenStreamAsync(new HostPortCollection("www.myip.ch", 80));
var request = FormatHttpRequest(new Uri("http://www.myip.ch/"));
using (var sw = new StreamWriter(pStream))
using (var sr = new StreamReader(pStream))
{
await sw.WriteLineAsync(request);
var retvar = await sr.ReadToEndAsync(); //Takes till the end of time itself
Console.WriteLine(retvar);
}
Debugger.Break();
}
private static string FormatHttpRequest(Uri url)
{
return $"GET {url.LocalPath} HTTP/1.0\r\nHost: {url.DnsSafeHost}\r\n\r\n";
}

The WriteLineAsync() call doesn't actually write out to the network immediately. Instead, the StreamWriter holds it in a buffer. Because of this, the server never actually gets your request, so won't send you a reply.
You should force the buffer to write out to the network by doing one of the following:
Turn on AutoFlush (reference) for the StreamWriter, which will make it flush the buffer with every call to WriteLineAsync()
Explicitly call FlushAsync() (reference)
Close the stream (which will flush the buffer) by rewriting the code like this:
using (var sr = new StreamReader(pStream))
{
using (var sw = new StreamWriter(pStream))
{
await sw.WriteLineAsync(request);
}
var retvar = await sr.ReadToEndAsync();
Console.WriteLine(retvar);
}

How to wait for multiple async http request

I'd like to ask about how to wait for multiple async http requests.
My code is like this :
public void Convert(XDocument input, out XDocument output)
{
var ns = input.Root.Name.Namespace;
foreach (var element in input.Root.Descendants(ns + "a"))
{
Uri uri = new Uri((string)element.Attribute("href"));
var wc = new WebClient();
wc.OpenReadCompleted += ((sender, e) =>
{
element.Attribute("href").Value = e.Result.ToString();
}
);
wc.OpenReadAsync(uri);
}
//I'd like to wait here until above async requests are all completed.
output = input;
}
Dose anyone know a solution for this?

There is an article by Scott Hanselman in which he describes how to do non blocking requests. Scrolling to the end of it, there is a public Task<bool> ValidateUrlAsync(string url) method.
You could modify it like this (could be more robust about response reading)
public Task<string> GetAsync(string url)
{
var tcs = new TaskCompletionSource<string>();
var request = (HttpWebRequest)WebRequest.Create(url);
try
{
request.BeginGetResponse(iar =>
{
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.EndGetResponse(iar);
using(var reader = new StreamReader(response.GetResponseStream()))
{
tcs.SetResult(reader.ReadToEnd());
}
}
catch(Exception exc) { tcs.SetException(exc); }
finally { if (response != null) response.Close(); }
}, null);
}
catch(Exception exc) { tcs.SetException(exc); }
return tsc.Task;
}
So with this in hand, you could then use it like this
var urls=new[]{"url1","url2"};
var tasks = urls.Select(GetAsync).ToArray();
var completed = Task.Factory.ContinueWhenAll(tasks,
completedTasks =>{
foreach(var result in completedTasks.Select(t=>t.Result))
{
Console.WriteLine(result);
}
});
completed.Wait();
//anything that follows gets executed after all urls have finished downloading
Hope this puts you in the right direction.
PS. this is probably as clear as it can get without using async/await

Consider using continuation passing style. If you can restructure your Convert method like this,
public void ConvertAndContinueWith(XDocument input, Action<XDocument> continueWith)
{
var ns = input.Root.Name.Namespace;
var elements = input.Root.Descendants(ns + "a");
int incompleteCount = input.Root.Descendants(ns + "a").Count;
foreach (var element in elements)
{
Uri uri = new Uri((string)element.Attribute("href"));
var wc = new WebClient();
wc.OpenReadCompleted += ((sender, e) =>
{
element.Attribute("href").Value = e.Result.ToString();
if (interlocked.Decrement(ref incompleteCount) == 0)
// This is the final callback, so we can continue executing.
continueWith(input);
}
);
wc.OpenReadAsync(uri);
}
}
You then run that code like this:
XDocument doc = something;
ConvertAndContinueWith(doc, (finishedDocument) => {
// send the completed document to the web client, or whatever you need to do
});