We Keep Coding

split binary file into chunks or Parts upload / download

I am using couchdb for some reason as a content management to upload files as binary data, there is no GridFs support like mongoDB to upload large files, so I need to upload files as chunks then retrieve them as one file.
here is my code
public string InsertDataToCouchDb(string dbName, string id, string filename, byte[] image)
{
var connection = System.Configuration.ConfigurationManager.ConnectionStrings["CouchDb"].ConnectionString;
using (var db = new MyCouchClient(connection, dbName))
{
// HERE I NEED TO UPLOAD MY IMAGE BYTE[] AS CHUNKS
var artist = new couchdb
{
_id = id,
filename = filename,
Image = image
};
var response = db.Entities.PutAsync(artist);
return response.Result.Content._id;
}
}
public byte[] FetchDataFromCouchDb(string dbName, string id)
{
var connection = System.Configuration.ConfigurationManager.ConnectionStrings["CouchDb"].ConnectionString;
using (var db = new MyCouchClient(connection, dbName))
{
//HERE I NEED TO RETRIVE MY FULL IMAGE[] FROM CHUNKS
var test = db.Documents.GetAsync(id, null);
var doc = db.Serializer.Deserialize<couchdb>(test.Result.Content);
return doc.Image;
}
}
THANK YOU

Putting image data in a CouchDB document is a terrible idea. Just don't. This is the purpose of CouchDB attachments.
The potential of bloating the database with redundant blob data via document updates alone will surely have major, negative consequences for anything other than a toy database.
Further there seems to be a lack of understanding how async/await works as the code in the OP is invoking async methods, e.g. db.Entities.PutAsync(artist), without an await - the call surely will fail every time (if the compiler even allows the code). I highly recommend grok'ing the Microsoft document Asynchronous programming with async and await.
Now as for "chunking": If the image data is so large that it needs to be otherwise streamed, the business of passing it around via a byte array looks bad. If the images are relatively small, just use Attachment.PutAsync as it stands.
Although Attachment.PutAsync at MyCouch v7.6 does not support streams (effectively chunking) there exists the Support Streams for attachments #177 PR, which does, and it looks pretty good.
Here's a one page C# .Net Core console app that uploads a given file as an attachment to a specific document using the very efficient streaming provided by PR 177. Although the code uses PR 177, it most importantly uses Attachments for blob data. Replacing a stream with a byte array is rather straightforward.
MyCouch + PR 177
In a console get MyCouch sources and then apply PR 177
$ git clone https://github.com/danielwertheim/mycouch.git
$ cd mycouch
$ git pull origin 15a1079502a1728acfbfea89a7e255d0c8725e07
(I don't know git so there's probably a far better way to get a PR)
MyCouchUploader
With VS2019
Create a new .Net Core console app project and solution named "MyCouchUploader"
Add the MyCouch project pulled with PR 177 to the solution
Add the MyCouch project as MyCouchUploader dependency
Add the Nuget package "Microsoft.AspNetCore.StaticFiles" as a MyCouchUploader dependency
Replace the content of Program.cs with the following code:
using Microsoft.AspNetCore.StaticFiles;
using MyCouch;
using MyCouch.Requests;
using MyCouch.Responses;
using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Security.Cryptography;
using System.Threading.Tasks;
namespace MyCouchUploader
{
class Program
{
static async Task Main(string[] args)
{
// args: scheme, database, file path of asset to upload.
if (args.Length < 3)
{
Console.WriteLine("\nUsage: MyCouchUploader scheme dbname filepath\n");
return;
}
var opts = new
{
scheme = args[0],
dbName = args[1],
filePath = args[2]
};
Action<Response> check = (response) =>
{
if (!response.IsSuccess) throw new Exception(response.Reason);
};
try
{
// canned doc id for this app
const string docId = "SO-68998781";
const string attachmentName = "Image";
DbConnectionInfo cnxn = new DbConnectionInfo(opts.scheme, opts.dbName)
{ // timely fail if scheme is bad
Timeout = TimeSpan.FromMilliseconds(3000)
};
MyCouchClient client = new MyCouchClient(cnxn);
// ensure db is there
GetDatabaseResponse info = await client.Database.GetAsync();
check(info);
// delete doc for succcessive program runs
DocumentResponse doc = await client.Documents.GetAsync(docId);
if (doc.StatusCode == HttpStatusCode.OK)
{
DocumentHeaderResponse del = await client.Documents.DeleteAsync(docId, doc.Rev);
check(del);
}
// sniff file for content type
FileExtensionContentTypeProvider provider = new FileExtensionContentTypeProvider();
if (!provider.TryGetContentType(opts.filePath, out string contentType))
{
contentType = "application/octet-stream";
}
// create a hash for silly verification
using var md5 = MD5.Create();
using Stream stream = File.OpenRead(opts.filePath);
byte[] fileHash = md5.ComputeHash(stream);
stream.Position = 0;
// Use PR 177, sea-locks:stream-attachments.
DocumentHeaderResponse put = await client.Attachments.PutAsync(new PutAttachmentStreamRequest(
docId,
attachmentName,
contentType,
stream // :-D
));
check(put);
// verify
AttachmentResponse verify = await client.Attachments.GetAsync(docId, attachmentName);
check(verify);
if (fileHash.SequenceEqual(md5.ComputeHash(verify.Content)))
{
Console.WriteLine("Atttachment verified.");
}
else
{
throw new Exception(String.Format("Attachment failed verification with status code {0}", verify.StatusCode));
}
}
catch (Exception e)
{
Console.WriteLine("Fail! {0}", e.Message);
}
}
}
}
To run:
$ MyCouchdbUploader http://name:password#localhost:5984 dbname path-to-local-image-file
Use Fauxton to visually verify the attachment for the doc.

Xml validation failed with RSA256with rsa Algorithm

I am trying to validate offline aadhar ekyc XML file with the sample code. But the validation keeps failing, I am not sure if it's the code error or error in the .cer file.
I am using SHA256withRSA algorithm to validate the XML file as mention in the documentation and sample code.
Same thing happening with Python Code.
Is there any special encoding feed into the validation function?
using System;
using System.Security.Cryptography.X509Certificates;
using System.Xml;
namespace test
{
class MainClass
{
public static void Main(string[] args)
{
// link ->
//https://drive.google.com/file/d/1aSv3HJUFf5_42Z-FqpdVHEk5b3VA3T3D/view
string XMLFilePath = "offlineaadhaar.xml"; //Get the XML file
// link ->
//https://drive.google.com/file/d/1FW4ciIhZqJuelOcGF2x6VaBCSDO9J-gM/view
string KeyFilePath = "okyc-publickey.cer"; //Get the public key certificate file
XmlDocument ObjXmlDocument = new XmlDocument();
ObjXmlDocument.Load(XMLFilePath); //Load the XML
XmlAttributeCollection SignatureElement = ObjXmlDocument.DocumentElement.Attributes; //Get the all XML attribute
string SignatureValue = SignatureElement.GetNamedItem("s").InnerXml; // Get Signature value
SignatureElement.RemoveNamedItem("s");//Remove the signature "s" attribute from XML and get the new XML to validate
//Console.WriteLine(SignatureElement);
/*----------------Read and parse the public key as string-----------------------*/
X509Certificate2 ObjX509Certificate2 = new X509Certificate2(KeyFilePath, "public"); //Initialize the public ket certificate file
Org.BouncyCastle.X509.X509Certificate objX509Certificate;
Org.BouncyCastle.X509.X509CertificateParser objX509CertificateParser = new Org.BouncyCastle.X509.X509CertificateParser();
objX509Certificate = objX509CertificateParser.ReadCertificate(ObjX509Certificate2.GetRawCertData());
/*----------------End-----------------------*/
//Console.WriteLine(objX509Certificate);
/* Init alg */
Org.BouncyCastle.Crypto.ISigner signer = Org.BouncyCastle.Security.SignerUtilities.GetSigner("SHA256withRSA");
//Console.WriteLine(signer);
/* Populate key */
signer.Init(false, objX509Certificate.GetPublicKey());
/* Get the signature into bytes */
var expectedSig = Convert.FromBase64String(SignatureValue);
/* Get the bytes to be signed from the string */
var msgBytes = System.Text.Encoding.UTF8.GetBytes(ObjXmlDocument.InnerXml);
/* Calculate the signature and see if it matches */
signer.BlockUpdate(msgBytes, 0, msgBytes.Length);
Console.WriteLine(msgBytes.Length);
bool Flag = signer.VerifySignature(expectedSig);
if (Flag)
{
Console.WriteLine("XML Validate Successfully");
}
else
{
Console.WriteLine("XML Validation Failed");
}
}
}
}

Is it possible to extract the content of a PKCS#7 signed file using System.Security.Cryptography.Pkcs.SignedCms?

I found this post related to pkcs#7 decrypting which suggest that there is no out of the box way to decrypt a signed pkcs#7 file. Decrypting PKCS#7 encrypted data in C#
However, I also saw this post and I managed to validate using the SignedCms class. http://blogs.msdn.com/b/shawnfa/archive/2006/02/27/539990.aspx
At the end of the post though, extracting the data using cms.ContentInfo.Content seems to return the signed file (.zip.p7) instead of the actual content (.zip). Am I missing something here? Using bouncy castle, I was able to get the actual data but would rather not use a third party if it's not necessary.
static void Main(string[] args)
{
var encodedFile = File.ReadAllBytes(InPath);
var signedData = new SignedCms();
signedData.Decode(encodedFile);
signedData.CheckSignature(true);
if (!Verify(signedData))
throw new Exception("No valid cert was found");
var trueContent = new CmsSignedData(File.ReadAllBytes(InPath)).SignedContent;
using (var str = new MemoryStream())
{
trueContent.Write(str);
var zip = new ZipArchive(str, ZipArchiveMode.Read);
zip.ExtractToDirectory(OutPath);
}
//using (var str = new MemoryStream(signedData.ContentInfo.Content))
//{
// var zip = new ZipArchive(str, ZipArchiveMode.Read);
// zip.ExtractToDirectory(OutPath);
//}
}
static bool Verify(SignedCms signedData)
{
var myCetificates = new X509Store(StoreName.My, StoreLocation.LocalMachine);
myCetificates.Open(OpenFlags.ReadOnly);
var certs = signedData.Certificates;
return (from X509Certificate2 cert in certs
select myCetificates.Certificates.Cast<X509Certificate2>()
.Any(crt => crt.Thumbprint == cert.Thumbprint))
.Any();
}
UPDATE: Will get back with the base64 sample file. Need to come up with one that doesn't contain sensitive information.
UPDATE: Here is the base 64 version of a file I can extract using bouncy castle but not using the ContentInfo.Content property. I've added as a comment on the code how I would extract SignedCms directly.
MIIFfQYJKoZIhvcNAQcCoIIFbjCCBWoCAQExCzAJBgUrDgMCGgUAMIHOBgEAoIHIBIHFUEsDBBQAAAAIAGRuWEggPN74JwAAADsAAAAMAAAAdGVzdGRhdGEudHh0bcixDQAgCEXB3sQd/gq6EVFaLHz7hwHgysM/uobpvMCDOUppdbkTUEsBAj8AFAAAAAgAZG5YSCA83vgnAAAAOwAAAAwAJAAAAAAAAAAgAAAAAAAAAHRlc3RkYXRhLnR4dAoAIAAAAAAAAQAYAOAo0XGdbtEBktqpaZ1u0QGS2qlpnW7RAVBLBQYAAAAAAQABAF4AAABRAAAAAACgggL7MIIC9zCCAd+gAwIBAgIQN3SCtKd9Hp1BDqeyqVr+tjANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwdmb28uYmFyMB4XDTE2MDIyNDAwNDg0MFoXDTM5MTIzMTIzNTk1OVowFDESMBAGA1UEAxMJcGtjczdzaWduMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAh5KDyFhPcxueQ1vJ0yQpcAyrSHLKw/Y2K0qhZfa7W3A6q33/i8VLl0XOFCCJs+iwcJcC+iCOIw+fFkRUEj9d7Z1sKwBCcZMolkZtNvtdXOgphV6R3g6/QNZgiQ3FFqHgp7+5OAdtBEsfsoowOClnVqZAmXnXW3Pl6Lsx0wxI8A7huytqboKRqrbRz02xO9rR1Ism0g5uQBtB2rpqGQk6/1ti/UYCp9fx7pxvWhe+a+oNIq7+ijAHPNLC+bwQTGd3LhwkzMSdONpY/utdbqCqToq/IbQ7KM0NRExZL/egPCsyJ2GGBQOVCveKkzdsW5V8p2XljcI5Sq7V8lVtqR3unwIDAQABo0cwRTBDBgNVHQEEPDA6gBBZ/6xjvqs3BLMBnQj5XmOJoRQwEjEQMA4GA1UEAxMHZm9vLmJhcoIQCiUdhpq5qrlA3FTAf3hpsTANBgkqhkiG9w0BAQsFAAOCAQEAcYl//1mzJa62K5uR+8CrpsrPWHEJ1H8l25CUQGL3u4g+ANOvJsJR6nryo4us9nI7XPocJNnFCoj26qfNmWgIF5giGkQpU2gRV5bPvYc6nPj4jZaQ7RcxJ4FT1BN/3ue7NLREodO/E1J2YWKE3uOLreBikn5B6tPKPEzA8XTjVB0ZZzfu7LMvsltB2fcaEbsifVCt4hlbtWlgFuNCbidIRWMpg8NdwcWydR1C7kuKGh1LJDG0v3ZPRgytkie8l/9zqvki4wt/kWXmDwba0lCWoyrfyahGMq5u2cqLG45pk/+1L89nw3BfR1U+5b786iXgNXKmYRWchwMQQ9r1xEa5FTGCAYYwggGCAgEBMCYwEjEQMA4GA1UEAxMHZm9vLmJhcgIQN3SCtKd9Hp1BDqeyqVr+tjAJBgUrDgMCGgUAoDcwEAYJKoZIhvcNAQkDMQMGAQAwIwYJKoZIhvcNAQkEMRYEFGmRdgvd3g6DeMqXK0QfUmJq7UnFMA0GCSqGSIb3DQEBAQUABIIBAEKdfeAfyc5TAei/GWW6Ns6aAvOrVXqNkkOJessd9P2ZYSxG4CTOo300jea7o2DYruiC4FJnSPqH7NoI0Q8EAod/E7HevZnrUq2Rtga4hSZSJfgnQuJqrOJksWfysRqt86cfwQYqmlSd94e7CgmT293rGTbG8SdXxRA8qi2J+2OULVSBFi3Z1x0hQlf31ioVBl5WMchsM8ri/q9IBBwFGqdEKVqxcmLkEtVv3czCq1z6rqkXkDk/qZ7qlhDzAqn8uguoXqhOR075Hv3Qnz6j1R+220aCOq5CmZIzdk8o8arEA9siXUASjbQpzULpG5fAenlCrgCnAXBkzkJKsBaTbYY=

Encrypting a class + serializing it

I'm trying to encrypt a class before serializing it. The only way I found is to encrypt it and return a String or a stream, is it possible to return the original class ?
public static Options Encrypt(Options Settings)
{
Options sk = null;
try
{
using (var stream = new MemoryStream())
{
RuntimeTypeModel.Default.Serialize(stream, Settings);
byte[] data = encryptWithPadding(stream.ToArray(), 0);
String base64EncryptedString = Convert.ToBase64String(data);
// needs to return an option instance
}
}
catch (Exception e)
{
Global.LogError("Serialization failed", e);
}
return sk;
}

You can't "encrypt" a protobuf without serializing it. Encryption generally works on bytes, not data structures. If you want to encrypt it, and then embed the encrypted version as a field in some other protobuf, make that other field have type bytes instead of Options.
(In theory you could encrypt each of the fields of Options individually, but this would not help you much and would very likely be less secure than encrypting the whole serialized blob.)

C# Encrypt an XML File

I need two methods one to encrypt and one to decrypt an xml file with a key= "hello world",the key hello world should be used to encrypt and decrypt the xml file.These methods should work on all machines!!! Any encryption methods will do. XML File contents below:
<root>
<lic>
<number>19834209</number>
<expiry>02/02/2002</expiry>
</lic>
</root>
Can some give me a sample?The issue is the msdn sample encyptions make a xml file encypted but when I decrypt on another machine it doesn't work.For example
I tried this sample:
How to: Encrypt XML Elements with Asymmetric Keys,
but here there is some kinda session and on another machine it says bad data phewf!

If you want the same key for encrypting and decrypting you should use a symmetric method (that's the definition, really). Here's the closest one to your sample (same source).
http://msdn.microsoft.com/en-us/library/sb7w85t6.aspx
The posted sample isn't working because they aren't using the same keys. Not only on different machines: running the program on the same machine twice should not work either (didn't work for me), because they use different random keys every time.
try adding this code after creating your key:
key = new RijndaelManaged();
string password = "Password1234"; //password here
byte[] saltBytes = Encoding.UTF8.GetBytes("Salt"); // salt here (another string)
var p = new Rfc2898DeriveBytes(password, saltBytes); //TODO: think about number of iterations (third parameter)
// sizes are devided by 8 because [ 1 byte = 8 bits ]
key.IV = p.GetBytes(key.BlockSize / 8);
key.Key = p.GetBytes(key.KeySize / 8);
Now the program is using the same key and initial vector, and Encrypt and Decrypt should work on all machines.
Also, consider renaming key to algorithm, otherwise this is very misleading. I'd say it's a bad, not-working-well example from MSDN.
NOTE: PasswordDeriveBytes.GetBytes() has been deprecated because of serious (security) issues within the PasswordDeriveBytes class. The code above has been rewritten to use the safer Rfc2898DeriveBytes class instead (PBKDF2 instead of PBKDF1). Code generated with the above using PasswordDeriveBytes may be compromised.
See also: Recommended # of iterations when using PKBDF2-SHA256?

First of all, if you want to use the same key for encrypting and decrypting, you should look at symmetric cryptography. Asymmetric cryptography is when the keys for encrypting and decrypting are different. Just so that you know - RSA is asymmetric, TripleDES and Rijndael are symmetric. There are others too, but .NET does not have default implementations for them.
I'd advise studying the System.Security.Cryptography namespace. And learning a bit about all that stuff. It has all you need to encrypt and decrypt files, as well as generate a password. In particular, you might be interested in these classes:
CryptoStream
PasswordDeriveBytes
RijndaelManaged
There are also examples for usage in MSDN for each of them. You can use these classes to encrypt any file, not just XML. If however you want to encrypt just a select few elements, you can take a look at System.Security.Cryptography.Xml namespace. I see you've already found one article about it. Keep following the links on that page and you will learn more about those classes.

Would be cooler if you used a private key to sign the <lic> element and added the result to the file (in a <hash> element perhaps). This would make it possibly for everyone to read the xml file in case your support needs to know the license number, or the date of expiry, but they can not change any values without the private key.
The public key needed to verify the signature would be common knowledge.
Clarification
Signing your code will only protect it against changes, it will not keep any information in it hidden. Your original question mentions encryption, but I am not sure that it is a requirement to hide the data, or just protect it from modification.
Example code: (Never publish PrivateKey.key. ServerMethods are only needed when signing the xml file, ClientMethods are only needed when verifying the xml file.)
using System;
using System.Diagnostics;
using System.IO;
using System.Security.Cryptography;
using System.Text;
using System.Xml;
public static class Program {
public static void Main() {
if (!File.Exists("PublicKey.key")) {
// Assume first run, generate keys and sign document.
ServerMethods.GenerateKeyPair();
var input = new XmlDocument();
input.Load("input.xml");
Debug.Assert(input.DocumentElement != null);
var licNode = input.DocumentElement["lic"];
Debug.Assert(licNode != null);
var licNodeXml = licNode.OuterXml;
var signedNode = input.CreateElement("signature");
signedNode.InnerText = ServerMethods.CalculateSignature(licNodeXml);
input.DocumentElement.AppendChild(signedNode);
input.Save("output.xml");
}
if (ClientMethods.IsValidLicense("output.xml")) {
Console.WriteLine("VALID");
} else {
Console.WriteLine("INVALID");
}
}
public static class ServerMethods {
public static void GenerateKeyPair() {
var rsa = SharedInformation.CryptoProvider;
using (var keyWriter = File.CreateText("PublicKey.key"))
keyWriter.Write(rsa.ToXmlString(false));
using (var keyWriter = File.CreateText("PrivateKey.key"))
keyWriter.Write(rsa.ToXmlString(true));
}
public static string CalculateSignature(string data) {
var rsa = SharedInformation.CryptoProvider;
rsa.FromXmlString(File.ReadAllText("PrivateKey.key"));
var dataBytes = Encoding.UTF8.GetBytes(data);
var signatureBytes = rsa.SignData(dataBytes, SharedInformation.HashAlgorithm);
return Convert.ToBase64String(signatureBytes);
}
}
public static class ClientMethods {
public static bool IsValid(string data, string signature) {
var rsa = SharedInformation.CryptoProvider;
rsa.FromXmlString(File.ReadAllText("PublicKey.key"));
var dataBytes = Encoding.UTF8.GetBytes(data);
var signatureBytes = Convert.FromBase64String(signature);
return rsa.VerifyData(dataBytes, SharedInformation.HashAlgorithm, signatureBytes);
}
public static bool IsValidLicense(string filename) {
var doc = new XmlDocument();
doc.Load(filename);
var licNode = doc.SelectSingleNode("/root/lic") as XmlElement;
var signatureNode = doc.SelectSingleNode("/root/signature") as XmlElement;
if (licNode == null || signatureNode == null) return false;
return IsValid(licNode.OuterXml, signatureNode.InnerText);
}
}
public static class SharedInformation {
public static int KeySize {
get { return 1024; }
}
public static string HashAlgorithm {
get { return "SHA512"; }
}
public static RSACryptoServiceProvider CryptoProvider {
get { return new RSACryptoServiceProvider(KeySize, new CspParameters()); }
}
}
}

this is how you digitally sign and verify XML documents Sign XML Documents