We Keep Coding

split binary file into chunks or Parts upload / download

I am using couchdb for some reason as a content management to upload files as binary data, there is no GridFs support like mongoDB to upload large files, so I need to upload files as chunks then retrieve them as one file.
here is my code
public string InsertDataToCouchDb(string dbName, string id, string filename, byte[] image)
{
var connection = System.Configuration.ConfigurationManager.ConnectionStrings["CouchDb"].ConnectionString;
using (var db = new MyCouchClient(connection, dbName))
{
// HERE I NEED TO UPLOAD MY IMAGE BYTE[] AS CHUNKS
var artist = new couchdb
{
_id = id,
filename = filename,
Image = image
};
var response = db.Entities.PutAsync(artist);
return response.Result.Content._id;
}
}
public byte[] FetchDataFromCouchDb(string dbName, string id)
{
var connection = System.Configuration.ConfigurationManager.ConnectionStrings["CouchDb"].ConnectionString;
using (var db = new MyCouchClient(connection, dbName))
{
//HERE I NEED TO RETRIVE MY FULL IMAGE[] FROM CHUNKS
var test = db.Documents.GetAsync(id, null);
var doc = db.Serializer.Deserialize<couchdb>(test.Result.Content);
return doc.Image;
}
}
THANK YOU

Putting image data in a CouchDB document is a terrible idea. Just don't. This is the purpose of CouchDB attachments.
The potential of bloating the database with redundant blob data via document updates alone will surely have major, negative consequences for anything other than a toy database.
Further there seems to be a lack of understanding how async/await works as the code in the OP is invoking async methods, e.g. db.Entities.PutAsync(artist), without an await - the call surely will fail every time (if the compiler even allows the code). I highly recommend grok'ing the Microsoft document Asynchronous programming with async and await.
Now as for "chunking": If the image data is so large that it needs to be otherwise streamed, the business of passing it around via a byte array looks bad. If the images are relatively small, just use Attachment.PutAsync as it stands.
Although Attachment.PutAsync at MyCouch v7.6 does not support streams (effectively chunking) there exists the Support Streams for attachments #177 PR, which does, and it looks pretty good.
Here's a one page C# .Net Core console app that uploads a given file as an attachment to a specific document using the very efficient streaming provided by PR 177. Although the code uses PR 177, it most importantly uses Attachments for blob data. Replacing a stream with a byte array is rather straightforward.
MyCouch + PR 177
In a console get MyCouch sources and then apply PR 177
$ git clone https://github.com/danielwertheim/mycouch.git
$ cd mycouch
$ git pull origin 15a1079502a1728acfbfea89a7e255d0c8725e07
(I don't know git so there's probably a far better way to get a PR)
MyCouchUploader
With VS2019
Create a new .Net Core console app project and solution named "MyCouchUploader"
Add the MyCouch project pulled with PR 177 to the solution
Add the MyCouch project as MyCouchUploader dependency
Add the Nuget package "Microsoft.AspNetCore.StaticFiles" as a MyCouchUploader dependency
Replace the content of Program.cs with the following code:
using Microsoft.AspNetCore.StaticFiles;
using MyCouch;
using MyCouch.Requests;
using MyCouch.Responses;
using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Security.Cryptography;
using System.Threading.Tasks;
namespace MyCouchUploader
{
class Program
{
static async Task Main(string[] args)
{
// args: scheme, database, file path of asset to upload.
if (args.Length < 3)
{
Console.WriteLine("\nUsage: MyCouchUploader scheme dbname filepath\n");
return;
}
var opts = new
{
scheme = args[0],
dbName = args[1],
filePath = args[2]
};
Action<Response> check = (response) =>
{
if (!response.IsSuccess) throw new Exception(response.Reason);
};
try
{
// canned doc id for this app
const string docId = "SO-68998781";
const string attachmentName = "Image";
DbConnectionInfo cnxn = new DbConnectionInfo(opts.scheme, opts.dbName)
{ // timely fail if scheme is bad
Timeout = TimeSpan.FromMilliseconds(3000)
};
MyCouchClient client = new MyCouchClient(cnxn);
// ensure db is there
GetDatabaseResponse info = await client.Database.GetAsync();
check(info);
// delete doc for succcessive program runs
DocumentResponse doc = await client.Documents.GetAsync(docId);
if (doc.StatusCode == HttpStatusCode.OK)
{
DocumentHeaderResponse del = await client.Documents.DeleteAsync(docId, doc.Rev);
check(del);
}
// sniff file for content type
FileExtensionContentTypeProvider provider = new FileExtensionContentTypeProvider();
if (!provider.TryGetContentType(opts.filePath, out string contentType))
{
contentType = "application/octet-stream";
}
// create a hash for silly verification
using var md5 = MD5.Create();
using Stream stream = File.OpenRead(opts.filePath);
byte[] fileHash = md5.ComputeHash(stream);
stream.Position = 0;
// Use PR 177, sea-locks:stream-attachments.
DocumentHeaderResponse put = await client.Attachments.PutAsync(new PutAttachmentStreamRequest(
docId,
attachmentName,
contentType,
stream // :-D
));
check(put);
// verify
AttachmentResponse verify = await client.Attachments.GetAsync(docId, attachmentName);
check(verify);
if (fileHash.SequenceEqual(md5.ComputeHash(verify.Content)))
{
Console.WriteLine("Atttachment verified.");
}
else
{
throw new Exception(String.Format("Attachment failed verification with status code {0}", verify.StatusCode));
}
}
catch (Exception e)
{
Console.WriteLine("Fail! {0}", e.Message);
}
}
}
}
To run:
$ MyCouchdbUploader http://name:password#localhost:5984 dbname path-to-local-image-file
Use Fauxton to visually verify the attachment for the doc.

Zlib compression imcompatible between Ionic.Zip, System.IO.Compression and SharpCompress

I am working on porting a file format (OMF) into C#. Part of the storage in the file is an array of zlib compressed data. An existing version of the file formatter uses a static method from Ionic.Zip to read the file, as follows:
public static byte[] Uncompress(this byte[] value)
{
// Uncompress
return ZlibStream.UncompressBuffer(value);
}
The project I am working on currently already uses SharpCompress and using 2 different compression libraries seemed wasteful, so I figured I would rewrite it to use SharpCompress. SharpCompress does not have the UncompressBuffer static function that Ionic does, so I implemented it as follows which seemed to be a pretty standard approach in my reading:
using (var originalStream = new MemoryStream(value))
{
using (var decompressedStream = new MemoryStream())
{
using (var decompressor = new ZlibStream(originalStream, CompressionMode.Decompress))
{
decompressor.CopyTo(decompressedStream);
return decompressedStream.ToArray();
}
}
}
I have also tried a similiar approach using the System.IO.Compression.DeflateStream class, following the MSDocs provided pattern. However, in both cases, at the CopyTo function call, I get an exception indicating there is an issue with the data:
For Zlib: 'Zlib Exception: Bad state (incorrect data check)'
For Windows: 'Block length does not match with its complement'
It could be something I am missing that differentiates the function of the UncompressBuffer function from this method of decompression, but it seems like the UncompressBuffer function works with internal portions of the Zlib Class.
What am I doing wrong here? Is there a difference between the implementations of the 2 zip libraries that makes them incompatible?

The code below runs, which suggests that there are at least baseline round trip capabilities with Ionic.Zip and SharpCompress. This suggests that there might be some specific subtleties with the payload you are trying to decompress.
class Program
{
static void Main(string[] args)
{
var rnd = new Random(0);
var raw = new byte[1024];
rnd.NextBytes(raw);
raw = System.Text.Encoding.UTF8.GetBytes(System.Convert.ToBase64String(raw));
var ionicCompressed = Ionic.Zlib.ZlibStream.CompressBuffer(raw);
var sharpCompressed = DoSharpCompress(raw);
var ionicDecompressIonic = Ionic.Zlib.ZlibStream.UncompressBuffer(sharpCompressed);
var ionicDecompressSharp = Ionic.Zlib.ZlibStream.UncompressBuffer(ionicCompressed);
var sharpDecompressSharp = DoSharpDecompress(sharpCompressed);
var sharpDecompressIonic = DoSharpDecompress(ionicCompressed);
AssertEqual(ionicDecompressIonic, ionicDecompressSharp);
AssertEqual(sharpDecompressSharp, sharpDecompressIonic);
AssertEqual(ionicDecompressSharp, sharpDecompressIonic);
AssertEqual(raw, sharpDecompressIonic);
Console.WriteLine(System.Text.Encoding.UTF8.GetString(raw));
Console.WriteLine(System.Text.Encoding.UTF8.GetString(sharpDecompressIonic));
Console.WriteLine(System.Text.Encoding.UTF8.GetString(raw) == System.Text.Encoding.UTF8.GetString(sharpDecompressIonic));
Console.ReadLine();
}
static byte[] DoSharpCompress(byte[] uncompressed)
{
var sc1 = new SharpCompress.Compressors.Deflate.ZlibStream(new MemoryStream(uncompressed), SharpCompress.Compressors.CompressionMode.Compress);
var sc2 = new MemoryStream();
sc1.CopyTo(sc2);
return sc2.ToArray();
}
static byte[] DoSharpDecompress(byte[] compressed)
{
var sc1 = new SharpCompress.Compressors.Deflate.ZlibStream(new MemoryStream(compressed), SharpCompress.Compressors.CompressionMode.Decompress);
var sc2 = new MemoryStream();
sc1.CopyTo(sc2);
return sc2.ToArray();
}
static bool AssertEqual(byte[] a, byte[] b)
{
if (!a.SequenceEqual(b))
throw new Exception();
return true;
}
}

Is it possible to extract the content of a PKCS#7 signed file using System.Security.Cryptography.Pkcs.SignedCms?

I found this post related to pkcs#7 decrypting which suggest that there is no out of the box way to decrypt a signed pkcs#7 file. Decrypting PKCS#7 encrypted data in C#
However, I also saw this post and I managed to validate using the SignedCms class. http://blogs.msdn.com/b/shawnfa/archive/2006/02/27/539990.aspx
At the end of the post though, extracting the data using cms.ContentInfo.Content seems to return the signed file (.zip.p7) instead of the actual content (.zip). Am I missing something here? Using bouncy castle, I was able to get the actual data but would rather not use a third party if it's not necessary.
static void Main(string[] args)
{
var encodedFile = File.ReadAllBytes(InPath);
var signedData = new SignedCms();
signedData.Decode(encodedFile);
signedData.CheckSignature(true);
if (!Verify(signedData))
throw new Exception("No valid cert was found");
var trueContent = new CmsSignedData(File.ReadAllBytes(InPath)).SignedContent;
using (var str = new MemoryStream())
{
trueContent.Write(str);
var zip = new ZipArchive(str, ZipArchiveMode.Read);
zip.ExtractToDirectory(OutPath);
}
//using (var str = new MemoryStream(signedData.ContentInfo.Content))
//{
// var zip = new ZipArchive(str, ZipArchiveMode.Read);
// zip.ExtractToDirectory(OutPath);
//}
}
static bool Verify(SignedCms signedData)
{
var myCetificates = new X509Store(StoreName.My, StoreLocation.LocalMachine);
myCetificates.Open(OpenFlags.ReadOnly);
var certs = signedData.Certificates;
return (from X509Certificate2 cert in certs
select myCetificates.Certificates.Cast<X509Certificate2>()
.Any(crt => crt.Thumbprint == cert.Thumbprint))
.Any();
}
UPDATE: Will get back with the base64 sample file. Need to come up with one that doesn't contain sensitive information.
UPDATE: Here is the base 64 version of a file I can extract using bouncy castle but not using the ContentInfo.Content property. I've added as a comment on the code how I would extract SignedCms directly.
MIIFfQYJKoZIhvcNAQcCoIIFbjCCBWoCAQExCzAJBgUrDgMCGgUAMIHOBgEAoIHIBIHFUEsDBBQAAAAIAGRuWEggPN74JwAAADsAAAAMAAAAdGVzdGRhdGEudHh0bcixDQAgCEXB3sQd/gq6EVFaLHz7hwHgysM/uobpvMCDOUppdbkTUEsBAj8AFAAAAAgAZG5YSCA83vgnAAAAOwAAAAwAJAAAAAAAAAAgAAAAAAAAAHRlc3RkYXRhLnR4dAoAIAAAAAAAAQAYAOAo0XGdbtEBktqpaZ1u0QGS2qlpnW7RAVBLBQYAAAAAAQABAF4AAABRAAAAAACgggL7MIIC9zCCAd+gAwIBAgIQN3SCtKd9Hp1BDqeyqVr+tjANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwdmb28uYmFyMB4XDTE2MDIyNDAwNDg0MFoXDTM5MTIzMTIzNTk1OVowFDESMBAGA1UEAxMJcGtjczdzaWduMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAh5KDyFhPcxueQ1vJ0yQpcAyrSHLKw/Y2K0qhZfa7W3A6q33/i8VLl0XOFCCJs+iwcJcC+iCOIw+fFkRUEj9d7Z1sKwBCcZMolkZtNvtdXOgphV6R3g6/QNZgiQ3FFqHgp7+5OAdtBEsfsoowOClnVqZAmXnXW3Pl6Lsx0wxI8A7huytqboKRqrbRz02xO9rR1Ism0g5uQBtB2rpqGQk6/1ti/UYCp9fx7pxvWhe+a+oNIq7+ijAHPNLC+bwQTGd3LhwkzMSdONpY/utdbqCqToq/IbQ7KM0NRExZL/egPCsyJ2GGBQOVCveKkzdsW5V8p2XljcI5Sq7V8lVtqR3unwIDAQABo0cwRTBDBgNVHQEEPDA6gBBZ/6xjvqs3BLMBnQj5XmOJoRQwEjEQMA4GA1UEAxMHZm9vLmJhcoIQCiUdhpq5qrlA3FTAf3hpsTANBgkqhkiG9w0BAQsFAAOCAQEAcYl//1mzJa62K5uR+8CrpsrPWHEJ1H8l25CUQGL3u4g+ANOvJsJR6nryo4us9nI7XPocJNnFCoj26qfNmWgIF5giGkQpU2gRV5bPvYc6nPj4jZaQ7RcxJ4FT1BN/3ue7NLREodO/E1J2YWKE3uOLreBikn5B6tPKPEzA8XTjVB0ZZzfu7LMvsltB2fcaEbsifVCt4hlbtWlgFuNCbidIRWMpg8NdwcWydR1C7kuKGh1LJDG0v3ZPRgytkie8l/9zqvki4wt/kWXmDwba0lCWoyrfyahGMq5u2cqLG45pk/+1L89nw3BfR1U+5b786iXgNXKmYRWchwMQQ9r1xEa5FTGCAYYwggGCAgEBMCYwEjEQMA4GA1UEAxMHZm9vLmJhcgIQN3SCtKd9Hp1BDqeyqVr+tjAJBgUrDgMCGgUAoDcwEAYJKoZIhvcNAQkDMQMGAQAwIwYJKoZIhvcNAQkEMRYEFGmRdgvd3g6DeMqXK0QfUmJq7UnFMA0GCSqGSIb3DQEBAQUABIIBAEKdfeAfyc5TAei/GWW6Ns6aAvOrVXqNkkOJessd9P2ZYSxG4CTOo300jea7o2DYruiC4FJnSPqH7NoI0Q8EAod/E7HevZnrUq2Rtga4hSZSJfgnQuJqrOJksWfysRqt86cfwQYqmlSd94e7CgmT293rGTbG8SdXxRA8qi2J+2OULVSBFi3Z1x0hQlf31ioVBl5WMchsM8ri/q9IBBwFGqdEKVqxcmLkEtVv3czCq1z6rqkXkDk/qZ7qlhDzAqn8uguoXqhOR075Hv3Qnz6j1R+220aCOq5CmZIzdk8o8arEA9siXUASjbQpzULpG5fAenlCrgCnAXBkzkJKsBaTbYY=

Streaming files from amazon s3 with seek possibility in C#

I need to work with huge files in Amazon S3. How can I get part of huge file from S3? Best way would be get stream with the seek possibility.
Unfortunately, CanSeek property of response.ResponseStream is false:
GetObjectRequest request = new GetObjectRequest();
request.BucketName = BUCKET_NAME;
request.Key = NumIdToAmazonKey(numID);
GetObjectResponse response = client.GetObject(request);

You could do following to read a certain part of your file
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucketName,
Key = keyName,
ByteRange = new ByteRange(0, 10)
};
See the documentation

I know this isn't exactly what OP is asking for but I needed a seekable s3 stream so I could read Parquet files without downloading them so I gave this a shot here: https://github.com/mukunku/RandomHelpers/blob/master/SeekableS3Stream.cs
Performance wasn't as bad as I expected. You can use the TimeWastedSeeking property to see how much time is being wasted by allowing Seek() on an s3 stream.
Here's an example on how to use it:
using (var client = new AmazonS3Client(credentials, Amazon.RegionEndpoint.USEast1))
{
using (var stream = SeekableS3Stream.OpenFile(client, "myBucket", "path/to/myfile.txt", true))
{
//stream is seekable!
}
}

After a frustrating afternoon with the same problem I found the static class AmazonS3Util
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/TS3Util.html
Which has a MakeStreamSeekable method.

Way late for the OP, but I've just posted an article and code demonstration of a SeekableS3Stream that performs reasonably well in real-world use cases.
https://github.com/mlhpdx/seekable-s3-stream
Specifically, I demonstrate reading a single small file from a much larger ISO disk image using the DiscUtils library unmodified by implementing a random-access stream that uses Range requests to pull sections of the file as-needed and maintains them in an MRU list to prevent re-downloading ranges for hot data structures in the file (e.g. zip central directory records).
The use is similarly simple:
using System;
using System.IO;
using System.Threading.Tasks;
using Amazon.S3;
using DiscUtils.Iso9660;
namespace Seekable_S3_Stream
{
class Program
{
const string BUCKET = "rds.nsrl.nist.gov";
const string KEY = "RDS/current/RDS_ios.iso"; // "RDS/current/RDS_modern.iso";
const string FILENAME = "READ_ME.TXT";
static async Task Main(string[] args)
{
var s3 = new AmazonS3Client();
using var stream = new Cppl.Utilities.AWS.SeekableS3Stream(s3, BUCKET, KEY, 1 * 1024 * 1024, 4);
using var iso = new CDReader(stream, true);
using var file = iso.OpenFile(FILENAME, FileMode.Open, FileAccess.Read);
using var reader = new StreamReader(file);
var content = await reader.ReadToEndAsync();
await Console.Out.WriteLineAsync($"{stream.TotalRead / (float)stream.Length * 100}% read, {stream.TotalLoaded / (float)stream.Length * 100}% loaded");
}
}
}

C# RSA Decryption using Bouncy Castle

I have been given a Base64 Encoded encrypted string, which was encrypted in Java using Bouncy Castle. Example Java snippet below:
Cipher cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
cipher.init(Cipher.ENCRYPT_MODE, key.getPublic());
byte[] encryptedText = cipher.doFinal("xxxxx|xxxxx".getBytes("UTF-8"));
String encodedText = new BASE64Encoder().encode(encryptedText);
I need to decrypt the resulting string using Bouncy Castle, but in C#
I have been given a code snippet on how to do this in Java, but I can't convert this for C# (reasons is we are building a .net site, and is going to be an iFrame within a Java site. The Java site is going to passing in the RSA Encrypted string to the .NET site). Example Java code to decrypt below:
Cipher cipherDec = Cipher.getInstance("RSA/ECB/PKCS1Padding");
cipherDec.init(Cipher.DECRYPT_MODE, key.getPrivate());
byte[] decodedText = new BASE64Decoder().decodeBuffer(encodedText);
byte[] decryptedText = cipherDec.doFinal(decodedText);
String finalValue = new String(decryptedText, "UTF-8");
I have downloaded the examples from http://www.bouncycastle.org/csharp/ but there doesn't seem to be an example of inputting a string value to get encrypted, and it then going though the encrypt/decrypt process.
I have been given values for modulus, public exponent, private expontent, prime P, prime q, prime exponent p, prime exponent q and crt coefficient.
I have seen that I can use the following:
IAsymmetricBlockCipher signer = new Pkcs1Encoding(new RsaEngine());
signer.Init(true, pubParameters);
But the signer object doesn't seem to have the same methods as the Java examples above.
Only method I can use is
ProcessBlock(byte[] inbuf, int inOff, int inLen);
But I can't see how to use this in my context.
Any help here would be most appreciated.

To Help others, the final code to convert is as follows:
RsaKeyParameters privParameters = new RsaPrivateCrtKeyParameters(mod, pubExp, privExp, p, q, pExp, qExp, crtCoef);
RsaKeyParameters pubParameters = new RsaKeyParameters(false, mod, pubExp);
IAsymmetricBlockCipher eng = new Pkcs1Encoding(new RsaEngine());
eng.Init(false, privParameters);
byte[] encdata = System.Convert.FromBase64String("{the enc string}");
encdata = eng.ProcessBlock(encdata, 0, encdata.Length);
string result = Encoding.UTF8.GetString(encdata);
mod, pubExp etc etc are all BigInteger values:
static BigInteger mod = new BigInteger("big int value");
The Following using directives are required:
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Crypto.Parameters;
using Org.BouncyCastle.Crypto.Engines;
using Org.BouncyCastle.Crypto.Encodings;
using Org.BouncyCastle.Math;
Which can be obtained from the bouncycastle site. http://www.bouncycastle.org/csharp/

Have you tried converting the base 64 string to a byte array and then using the process block method? There may be more to it than that but it's definitely the first step I would take.
Here's an example of how to do this: http://msdn.microsoft.com/en-us/library/system.convert.frombase64string.aspx

I'm not sure I understand why you must use Bouncycastle. The following small code snippet shows and RSA encryption/decryption example using only .NET classes:
using System;
using System.Text;
using System.Security.Cryptography;
namespace RsaForDotNet
{
class Program
{
static void Main(string[] args)
{
RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(512);
var encrypted_msg = rsa.Encrypt(Encoding.UTF8.GetBytes("Secret Data"), false);
var encoded_msg = Convert.ToBase64String(encrypted_msg);
Console.WriteLine(encoded_msg);
var decoded_msg = Convert.FromBase64String(encoded_msg);
var decrypted_msg = Encoding.UTF8.GetString(rsa.Decrypt(decoded_msg, false));
Console.WriteLine(decrypted_msg);
}
}
}