In our production server, we create thousands of pdf documents by wkhtmltopdf and sign them via iTextSharp using HSM. Most of the operation performed at night. But after we performed an update for multithreading we have been stack into a serious situation. After working a long time, our WCF service starts to give exception for every request:
System.IndexOutOfRangeException: Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader's or TextWriter's Synchronized methods. This also applies to classes like StreamWriter and StreamReader.
at System.Buffer.InternalBlockCopy(Array src, Int32 srcOffsetBytes, Array dst, Int32 dstOffsetBytes, Int32 byteCount
at System.IO.StreamWriter.Write(Char[] buffer, Int32 index, Int32 count
at System.IO.TextWriter.SyncTextWriter.WriteLine(String value
at iTextSharp.text.pdf.PdfReader..ctor(IRandomAccessSource byteSource, Boolean partialRead, Byte[] ownerPassword, X509Certificate certificate, ICipherParameters certificateKey, Boolean closeSourceOnConstructorError
at iTextSharp.text.pdf.PdfReader..ctor(RandomAccessFileOrArray raf, Byte[] ownerPassword, Boolean partial
at Turkkep.Daemons.Crypto.PDFSignerService.sign(Byte[] sourceData, Stream destination
at Turkkep.Daemons.Crypto.PDFSignerService.Sign(Byte[] sourceData) Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type
at Turkkep.Core.Signature.IPDFSigner.Sign(Byte[] sourceData
at Turkkep.Shared.Signature.PDFSigner.Sign(Byte[] sourceData)
--- End of inner exception stack trace --- at Turkkep.Shared.Signature.PDFSigner.Sign(Byte[] sourceData
at Turkkep.Shared.Daemon.InvoiceWorker.getPdfDocument(InvoiceType12 ublInvoice, Int64 organization
at Turkkep.Shared.Daemon.InvoiceWorker.processEArchive(Invoice invoice, String signedInvoice, InvoiceType12 ublInvoice
at Turkkep.Shared.Daemon.InvoiceWorker.sendInvoice12(Invoice invoice)
Our signing code for WCF service is;
private void sign(byte[] sourceData, Stream destination)
{
RandomAccessFileOrArray source = new RandomAccessFileOrArray(sourceData);
using (var reader = new PdfReader(source, null))
{
using (var stamper = PdfStamper.CreateSignature(reader, destination, '\0'))
{
var appearance = stamper.SignatureAppearance;
appearance.Reason = _reason;
appearance.Location = _location;
appearance.SetVisibleSignature(new Rectangle(0, 0, 0, 0), 1, String.Format("TurkKEP-{0}-{1}-{2}", DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day));
var externalSigner = new PDFExternalSigner("SHA-256");
ICollection<X509Certificate> chain = getChain(externalSigner.GetActiveConnection());
MakeSignature.SignDetached(appearance, externalSigner, chain, null, null, null, _estimatedSize, CryptoStandard.CMS);
}
}
}
public byte[] Sign(byte[] sourceData)
{
try
{
using (MemoryStream destination = new MemoryStream())
{
sign(sourceData, destination);
return destination.ToArray();
}
}
catch (FaultException)
{
throw;
}
catch (Exception ex)
{
_logger.Error("PDFSignerService.Sign", ex);
throw new Exception(ex.ToString());
}
}
I think locking the PdfReader can solve the problem but I don't want to weak multithread power. Getting an exception for some messages is not a problem. In this situation service never works again until we restart it.
Note: Production server is up to date. We've just updated.
Edit: Last night we have caught similar exception with different stacktrace:
19.12.2017 04:03:08
System.Exception: PDF sign failed. Crypto service communication exception occured.
---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: System.IndexOutOfRangeException: Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader's or TextWriter's Synchronized methods. This also applies to classes like StreamWriter and StreamReader.
at System.Buffer.InternalBlockCopy(Array src, Int32 srcOffsetBytes, Array dst, Int32 dstOffsetBytes, Int32 byteCount
at System.IO.StreamWriter.Write(Char[] buffer, Int32 index, Int32 count
at System.IO.TextWriter.SyncTextWriter.WriteLine(String value
at iTextSharp.text.pdf.PdfStamperImp.Close(PdfIndirectReference info, Int32 skipInfo
at iTextSharp.text.pdf.PdfStamperImp.Close(IDictionary`2 moreInfo
at iTextSharp.text.pdf.PdfSignatureAppearance.PreClose(Dictionary`2 exclusionSizes
at iTextSharp.text.pdf.security.MakeSignature.SignDetached(PdfSignatureAppearance sap, IExternalSignature externalSignature, ICollection`1 chain, ICollection`1 crlList, IOcspClient ocspClient, ITSAClient tsaClient, Int32 estimatedSize, CryptoStandard sigtype, SignaturePolicyIdentifier signaturePolicy
at Turkkep.Daemons.Crypto.PDFSignerService.sign(Byte[] sourceData, Stream destination
at Turkkep.Daemons.Crypto.PDFSignerService.Sign(Byte[] sourceData) Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc
After this exception service throws same exception for every call.
Now we are going to change code as following for moving the PdfStamper out of the using block.
private void sign(byte[] sourceData, Stream destination)
{
RandomAccessFileOrArray source = new RandomAccessFileOrArray(sourceData);
using (var reader = new PdfReader(source, null))
{
try
{
var stamper = PdfStamper.CreateSignature(reader, destination, '\0');
var appearance = stamper.SignatureAppearance;
appearance.Reason = _reason;
appearance.Location = _location;
appearance.SetVisibleSignature(new Rectangle(0, 0, 0, 0), 1, String.Format("TurkKEP-{0}-{1}-{2}", DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day));
var externalSigner = new PDFExternalSigner("SHA-256");
ICollection<X509Certificate> chain = getChain(externalSigner.GetActiveConnection());
MakeSignature.SignDetached(appearance, externalSigner, chain, null, null, null, _estimatedSize, CryptoStandard.CMS);
stamper.Dispose();
}
catch (Exception ex)
{
_logger.Error("Document sign failed unexpectedly.", ex);
throw new FaultException("Document sign failed unexpectedly. Exception: " + ex.ToString());
}
}
}
Related
I am having trouble catching and handling exceptions thrown by websocketsharp
When I connect to the server with the client, and I close the client just by ALT+F4 or X button, the server receives the exception of connection forcibly closed.
Even though, these exceptions won't crash the program, this will cause the console to be spammed by those exceptions, important messages and logs will be pushed down and nothing can be read (there is a lot of clients).
This is the exception:
7/28/2021 6:41:56 AM|Fatal|<>c__DisplayClass71_0.<receiveRequest>b__0:0|WebSocketSharp.WebSocketException: An exception has occurred while reading an HTTP request/response.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
at System.Net.Sockets.NetworkStream.ReadByte()
at WebSocketSharp.HttpBase.readHeaders(Stream stream, Int32 maxLength)
at WebSocketSharp.HttpBase.Read[T](Stream stream, Func`2 parser, Int32 millisecondsTimeout)
--- End of inner exception stack trace ---
at WebSocketSharp.HttpBase.Read[T](Stream stream, Func`2 parser, Int32 millisecondsTimeout)
at WebSocketSharp.HttpRequest.Read(Stream stream, Int32 millisecondsTimeout)
at WebSocketSharp.Net.WebSockets.TcpListenerWebSocketContext..ctor(TcpClient tcpClient, String protocol, Boolean secure, ServerSslConfiguration sslConfig, Logger logger)
at WebSocketSharp.Ext.GetWebSocketContext(TcpClient tcpClient, String protocol, Boolean secure, ServerSslConfiguration sslConfig, Logger logger)
at WebSocketSharp.Server.WebSocketServer.<>c__DisplayClass71_0.<receiveRequest>b__0(Object state)
I tried using
public class Echo : WebSocketBehavior
{
protected override void OnError(ErrorEventArgs e)
{
//handle
}
I tried wrapping the websocket server in a try catch
try
{
string path = "ws://localhost:111";
WebSocketServer wssv = new WebSocketServer(path);
wssv.Start();
while (Console.ReadKey(true).Key != ConsoleKey.Enter) { }
wssv.Stop();
} catch { }
Nothing works, the exception will the thrown and shown on the console whatever I tried.
So the question is, how do you catch those exceptions?
I was able to solve the issue disabling/hiding completely the logs output.
public static void Disable(this Logger logger)
{
var field = logger.GetType().GetField("_output", BindingFlags.NonPublic | BindingFlags.Instance);
field?.SetValue(logger, new Action<LogData, string>((d, s) => { }));
}
I have an aws Lambda function that I am making using .NET core. When running the lambda function I get the following error:
(Unable to load DLL 'BCrypt.dll': The specified module could not be found.
(Exception from HRESULT: 0x8007007E)): AggregateException
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at lambda_method(Closure , Stream , Stream , ContextInfo )
at Interop.BCrypt.BCryptGenRandom(IntPtr hAlgorithm, Byte* pbBuffer, Int32 cbBuffer, Int32 dwFlags)
at Interop.BCrypt.BCryptGenRandom(Byte[] pbBuffer, Int32 cbBuffer)
at System.Security.Cryptography.RNGCryptoServiceProvider.GetBytes(Byte[] data)
at User.Util.GetSalt(ILambdaContext context)
at User.Functions.d__1.MoveNext()
It is being thrown from the following code:
public static string GetSalt()
{
byte[] bytes = new byte[128 / 8];
using(var keyGenerator = RandomNumberGenerator.Create())
{
keyGenerator.GetBytes(bytes);
return BitConverter.ToString(bytes).Replace("-", "").ToLower();
}
}
The code is breaking on this line:
keyGenerator.GetBytes(bytes);
It was my belief that .net core should work well cross platform (As aws uses unix machines).
From everything I can find online this seems to be the best way to do this.
We have an ASP.Net WebAPI 2.0 application that publishes messages to our RabbitMQ Server. In 99% of the cases, all is well... but randomly the application terminates with a System.AccessViolationException for no apparent reason.
How can I prevent this failure? Could it be related to our recent upgrade to the 3.6.6 C# driver (it worked fine before the upgrade)?
Things I have already eliminated:
A new IModel is used for each publish (I know IModel is not thread
safe)
A call is made to CreateConnection for each call as well (I
know I could reuse the connection, but we do not currently). The connection is AutoClose = true;
The Channel is used in a using block... so it is disposed each time
Here is a sample stack trace of where it explodes:
Exception Details
System.AccessViolationException
Attempted to read or write protected
memory. This is often an indication that other memory is corrupt.
at System.Net.UnsafeNclNativeMethods.SafeNetHandlesXPOrLater.GetAddrInfoW(String
nodename, String servicename, AddressInfo& hints, SafeFreeAddrInfo&
handle)
at System.Net.Dns.TryGetAddrInfo(String name, AddressInfoHints flags, IPHostEntry& hostinfo)
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)
at RabbitMQ.Client.TcpClientAdapter.BeginConnect(String host, Int32 port, AsyncCallback requestCallback, Object state)
at RabbitMQ.Client.Impl.SocketFrameHandler.Connect(ITcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout)
at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, Int32 connectionTimeout, Int32
readTimeout, Int32 writeTimeout)
at RabbitMQ.Client.Framing.Impl.ProtocolBase.CreateFrameHandler(AmqpTcpEndpoint
endpoint, Func'2 socketFactory, Int32 connectionTimeout, Int32
readTimeout, Int32 writeTimeout)
at RabbitMQ.Client.ConnectionFactory.CreateConnection(IList'1 endpoints, String clientProvidedName)
And another
System.Net.UnsafeNclNativeMethods+SafeNetHandlesXPOrLater.GetAddrInfoW(System.String,
System.String, System.Net.AddressInfo ByRef,
System.Net.SafeFreeAddrInfo ByRef)
System.Net.Dns.TryGetAddrInfo(System.String, System.Net.AddressInfoHints, System.Net.IPHostEntry ByRef)
System.Net.Dns.InternalGetHostByName(System.String, Boolean)
System.Net.Dns.GetHostAddresses(System.String)
RabbitMQ.Client.TcpClientAdapter.BeginConnect(System.String, Int32, System.AsyncCallback, System.Object)
RabbitMQ.Client.Impl.SocketFrameHandler.Connect(RabbitMQ.Client.ITcpClient,
RabbitMQ.Client.AmqpTcpEndpoint, Int32)
RabbitMQ.Client.Impl.SocketFrameHandler..ctor(RabbitMQ.Client.AmqpTcpEndpoint,
System.Func'2,
Int32, Int32, Int32)
RabbitMQ.Client.Framing.Impl.ProtocolBase.CreateFrameHandler(RabbitMQ.Client.AmqpTcpEndpoint,
System.Func'2,
Int32, Int32, Int32)
RabbitMQ.Client.ConnectionFactory.CreateConnection(System.Collections.Generic.IList'1,
System.String)
I'm not sure exactly why your error is occurring I would need to see some of your code for that, but I use RabbitMQ quite a bit and to publish I use a class like so:
(Changed some parts as they are not relevant for your situation such as encryption, compression etc. But this would be the basic format of it)
using System;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Framing;
namespace Messaging
{
public class MessageSender : IDisposable
{
private const string EXCHANGE_NAME = "MY_EXCHANGE";
private readonly ConnectionFactory factory;
private readonly IConnection connection;
private readonly IModel channel;
public MessageSender(string address, string username, string password)
{
factory = new ConnectionFactory {UserName = username, Password = password, HostName = address};
connection = factory.CreateConnection();
channel = connection.CreateModel();
channel.ExchangeDeclare(EXCHANGE_NAME, "topic");
}
public void Send(string payload, string topic)
{
var prop = new BasicProperties();
var data = Encoding.ASCII.GetBytes(payload);
channel.BasicPublish(EXCHANGE_NAME, topic.ToUpper(), prop, data);
}
public void Dispose()
{
try
{
channel.Dispose();
connection.Dispose();
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
}
}
The idea is to let you issue multiple calls or a single one and dispose of the class when you wish. Wrap it with a using statement and your set.
Never had a single problem with this in about 3-4 years of using it, so you might be able to contrast this vs your code to find the differences.
Managed to get a (partial) resolution on this from RabbitMQ users group.
Details here: Rabbit MQ Users Group
Basic idea is that the IConnection object should be shared, as it is heavy to open and close. Only the IModel should be opened fresh for each thread
I am trying to use Memcached.ClientLibrary. I was able to make it work and everything but after a few hits (even before I get to see a page for the first time), I get this weird error about which I couldn't find any info when searching for it.
Error message:
Cannot write to a BufferedStream while the read buffer is not empty if the underlying stream is not seekable. Ensure that the stream underlying this BufferedStream can seek or avoid interleaving read and write operations on this BufferedStream.
Stack trace:
[NotSupportedException: Cannot write to a BufferedStream while the read buffer is not empty if the underlying stream is not seekable. Ensure that the stream underlying this BufferedStream can seek or avoid interleaving read and write operations on this BufferedStream.]
System.IO.BufferedStream.ClearReadBufferBeforeWrite() +10447571
System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count) +163
Memcached.ClientLibrary.SockIO.Write(Byte[] bytes, Int32 offset, Int32 count) in C:\devroot\memcacheddotnet\trunk\clientlib\src\clientlib\SockIO.cs:411
Memcached.ClientLibrary.SockIO.Write(Byte[] bytes) in C:\devroot\memcacheddotnet\trunk\clientlib\src\clientlib\SockIO.cs:391
Memcached.ClientLibrary.MemcachedClient.Set(String cmdname, String key, Object obj, DateTime expiry, Object hashCode, Boolean asString) in C:\devroot\memcacheddotnet\trunk\clientlib\src\clientlib\MemCachedClient.cs:766
Memcached.ClientLibrary.MemcachedClient.Set(String key, Object value, DateTime expiry) in C:\devroot\memcacheddotnet\trunk\clientlib\src\clientlib\MemCachedClient.cs:465
Yuusoft.Julian.Server.Models.Utils.Caching.CacheWrapper.Add(CacheKey key, T o, CacheDependency dependencies, Nullable`1 expirationTime, CacheItemRemovedCallback callBack)
My code to initialize (static constructor):
SockIOPool pool = SockIOPool.GetInstance();
pool.SetServers(CacheWrapper.Servers);
pool.InitConnections = 3;
pool.MinConnections = 1;
pool.MaxConnections = 50;
pool.SocketConnectTimeout = 1000;
pool.SocketTimeout = 3000;
pool.MaintenanceSleep = 30;
pool.Failover = true;
pool.Nagle = false;
pool.Initialize();
// Code to set (the second is the one erroing - but not at the first hit?!)
MemcachedClient mc = new MemcachedClient();
mc.Set(key, o, expirationTime.Value);
// Code to get
MemcachedClient mc = new MemcachedClient();
object o = mc.Get(key);
In addition to this exception, following two exceptions were also present in my memcached log4net logs of Memcached.ClientLibrary (Error storing data in cache for key:<key with spaces> and Exception thrown while trying to get object from cache for key:<key with spaces>) I was able to resolve all these
three exceptions by ensuring that memcached key doesn't contain any whitespace.
Reference:https://groups.google.com/forum/#!topic/memcached/4WMcTbL8ZZY
Memcached Version: memcached-win32-1.4.4-14
I've been struggling with a problem when downloading very big files (>2GB) on Silverlight. My application is an out-of-browser Download Manager running with elevated permissions.
When the file reaches a certain ammount of data (2GB), it throws the following exception:
System.ArgumentOutOfRangeException was caught
Message=Specified argument was out of the range of valid values.
Parameter name: count
StackTrace:
in MS.Internal.InternalNetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback callback, Object state)
in MS.Internal.InternalNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
in MySolution.DM.Download.BeginResponseCallback(IAsyncResult ar)
InnerException:
Null
The only clue I have is this site, who shows the BeginCode implementation. This exception only occurs when count is < then 0.
My code
/* "Target" is a File object. "source" is a Stream object */
var buffer = new byte[64 * 1024];
int bytesRead;
Target.Seek(0, SeekOrigin.End); // The file might exists when resuming a download
/* The exception throws from inside "source.Read" */
while ((bytesRead = source.Read(buffer, 0, buffer.Length)) > 0)
{
Target.Write(buffer, 0, bytesRead);
_fileBytes = Target.Length;
Deployment.Current.Dispatcher.BeginInvoke(() => { DownloadPercentual = Double.Parse(Math.Round((decimal)(_fileBytes / (_totalSize / 100)), 5).ToString()); });
}
Target.Close();
logFile.Close();
The error occurs with different kind of files, and they come from public buckets on Amazon S3. (with regular http requests).
I searched a bit and it looks like this is a known limitation in Silverlight. One possible workaround is to perform the download in multiple sections, each smaller than 2GB, using the Range header.