We Keep Coding

Can I use part of a Shell Object parsing name in the windows portable device (WPD) api to get a file stream?

Is it always valid to use the first part of the last guid from a windows api codepack ShellObject parsing name representing a path to a USB-connected android or iphone device when transferring a file to local storage in the windows portable device (WPD) api?
Eg. an example parsing name from a ShellObject get parsing name call:
::{20D04FE0-3AEA-1069-A2D8-08002B30309D}\\\?\usb#vid_22b8&pid_2e82#zy322kldjb#{6ac27878-a6fa-4155-ba85-f98f491d4f33}\{00000009-0001-0001-0000-000000000000}\{00000009-0001-0001-0000-000000000000}\{00005461-0001-0001-0000-000000000000}
As far as I can tell, this represents:
- My Computer ::{20D04FE0-3AEA-1069-A2D8-08002B30309D}\
- USB device \\?\usb#vid_22b8&pid_2e82#zy322kldjb#{6ac27878-a6fa-4155-ba85-f98f491d4f33}\
- Root folder SID-{10001,,26401026048}\
- Folder 1 {00000009-0001-0001-0000-000000000000}\
- Folder 2 {00000075-0001-0001-0000-000000000000}\
- File {00005461-0001-0001-0000-000000000000}
Enumerating the contents of Folder 2 on the device using the WPD api, I can see the same file has a file id like o5461 - this looks like part of the File part above.
Is this assumption always correct - can I always use it like in the following code, to copy one file from an android phone to local storage on the c-drive? Or is there sometimes a mismatch? Also, what does the "o" mean in the file id?
This seems to work, but I'm hesitant to use it in production-ready code.
using System.IO;
using PortableDeviceApiLib;
public class Copier
{
public void PerformCopy()
{
var deviceId = #"\\?\usb#vid_22b8&pid_2e82#zy322kldjb#{6ac27878-a6fa-4155-ba85-f98f491d4f33}";
var sourceItemGuidString = "{00005461-0001-0001-0000-000000000000}";
var destinationPath = #"C:\Test\";
var fileName = "Testing.jpg";
var size = 3738545;
// get "o5461" from the parsing path from the Shell libraries
var fileId = "o" + sourceItemGuidString.Replace("{", string.Empty).Replace("}", string.Empty).Split('-')[0].TrimStart('0');
this.TransferContentFromDevice(deviceId, fileId, size, destinationPath, fileName);
}
private void TransferContentFromDevice(string deviceId, string fileId, long length, string saveToPath, string fileName)
{
PortableDeviceClass device;
try
{
device = SomehowGetOnePortableDeviceAndConnectToIt(deviceId);
// Make sure that the target dir exists.
Directory.CreateDirectory(saveToPath);
device.Content(out var content);
content.Transfer(out var resources);
var property = new _tagpropertykey
{
fmtid = new Guid(0xE81E79BE, 0x34F0, 0x41BF, 0xB5, 0x3F, 0xF1, 0xA0, 0x6A, 0xE8, 0x78, 0x42),
pid = 0
};
uint optimalTransferSize = 0;
resources.GetStream(fileId, ref property, 0, ref optimalTransferSize, out IStream wpdStream);
var sourceStream = (System.Runtime.InteropServices.ComTypes.IStream)wpdStream;
using (var targetStream = new FileStream(Path.Combine(saveToPath, fileName), FileMode.Create, FileAccess.Write))
{
// Get the total size.
long written = 0;
long lPCt = 0;
unsafe
{
var buffer = new byte[1024];
int bytesRead;
do
{
sourceStream.Read(buffer, 1024, new IntPtr(&bytesRead));
targetStream.Write(buffer, 0, bytesRead);
written += 1024;
long PCt = length > 0 ? (100 * written) / length : 100;
if (PCt != lPCt)
{
lPCt = PCt;
Console.WriteLine("Progress: " + lPCt);
}
} while (bytesRead > 0);
}
}
}
finally
{
Disconnect(device);
}
}
}

If item has a parsing name like below, I was after the 5461 part from the final guid:
::{20D04FE0-3AEA-1069-A2D8-08002B30309D}\\\?\usb#vid_22b8&pid_2e82#zy322kldjb#{6ac27878-a6fa-4155-ba85-f98f491d4f33}\SID-{10001,,26401026048}\{00000009-0001-0001-0000-000000000000}\{00000075-0001-0001-0000-000000000000}\{00005461-0001-0001-0000-000000000000}
We can use the following code which will get the WPD_OBJECT_ID property from a ShellObject. More info on this and related properties here:
string GetFileId(ShellObject item)
{
const int WPD_OBJECT_ID = 2;
var property = item.Properties.GetProperty(new PropertyKey(new Guid("{ef6b490d-5cd8-437a-affc-da8b60ee4a3c}"), WPD_OBJECT_ID));
return property.ValueAsObject as string;
}
This returns the file id o5641, which can then be used in my TransferContentFromDevice method directly.
Thanks to #SimonMourier for pointing me in the right direction.

Storing double values in binary file and reading it

I need to store double values in binary files and read them. My data is stored in an array. I have tried the following code but apparently I am storing more values than the array size and I am reading totally wrong data. Like if I am storing 0.26 from array[0], I can see the very few first values in the binary file to be A4 70 3D... I don't get how it is converting 0.26 to these values and on what basis.
This code is for writing to binary file:
double [] DataCollection_array = new double[10000];
public void store_data()
{
Binary_filename = folder_path + "\\" + "Binary1.bin";
stream = new FileStream(folder_path + "\\" + "Binary1.bin", FileMode.Create);
binary_writer = new BinaryWriter(stream);
writetoBinary(DataCollection_array.size);
}
public void writetoBinary(int size)
{
for (int i = 0; i < size; i++)
{
binary_writer.Write(DataCollection_array[i]);
}
}
This code for reading the double values from a folder that contains binary files:
int bytes_counter1 = 0;
Channels = new List<double>[File_size];
public void read_data ()
{
path2 = Directory2.folder_path + "\\" + "Binary" + file_number + ".bin";
file_stream = new FileStream(path2, FileMode.Open, FileAccess.Read);
using (reader = new BinaryReader(file_stream))
{
if (bytes_counter1 < reader.BaseStream.Length)
{
reader.BaseStream.Seek((count + offset1), SeekOrigin.Begin);
Channels.Add((double)reader.ReadByte());
bytes_counter1++;
}
}
}

You are writing doubles:
binary_writer.Write(DataCollection_array[i]);
But you are only reading bytes:
Channels.Add((double)reader.ReadByte()); // Read one byte
Change it to:
Channels.Add(reader.ReadDouble()); // Read one double

SSIS 2015 Script task convert text file to UTF8 in C# or VB

I want to convert my resulting txt file into a UTF8 formatted file so I can load it into my Azure SQL DW via Polybase. It is required the source file be in UTF8.
MSDN has an "IO Streaming example" HERE works perfectly for a single job. I am trying to architect an SSIS solution for around 30 tables though. I believe using this method would cause a race condition where the PS script will be locked by 1 SSIS package when another SSIS package needs it.
I am a sql dev, not a .NET dev so please forgive me. How would one convert the above to an SSIS C# Script task assuming I know how to pass parameters into the Script task?
PowerShell Code from MSDN
#Static variables
$ascii = [System.Text.Encoding]::ASCII
$utf16le = [System.Text.Encoding]::Unicode
$utf8 = [System.Text.Encoding]::UTF8
$ansi = [System.Text.Encoding]::Default
$append = $False
#Set source file path and file name
$src = [System.IO.Path]::Combine("<MySrcFolder>","<MyUtf8stage>.txt")
#Set source file encoding (using list above)
$src_enc = $ascii
#Set target file path and file name
$tgt = [System.IO.Path]::Combine("<MyDestFolder>","<MyFinalstage>.txt")
#Set target file encoding (using list above)
$tgt_enc = $utf8
$read = New-Object System.IO.StreamReader($src,$src_enc)
$write = New-Object System.IO.StreamWriter($tgt,$append,$tgt_enc)
while ($read.Peek() -ne -1)
{
$line = $read.ReadLine();
$write.WriteLine($line);
}
$read.Close()
$read.Dispose()
$write.Close()
$write.Dispose()
Update
I found a similar post which I was able to tweak to my needs, I swear I searched high and low before posting. Anyway here is what IS working for me. If you see anyway to improve it please share:
public void Main()
{
//$Package::SourceSQLObject = tablename
//$Package::StageFile_DestinationFolderPath = rootpath eg "C:\temp\"
string path = (string)Dts.Variables["$Package::StageFile_DestinationFolderPath"].Value;
string name = (string)Dts.Variables["$Package::SourceSQLObject"].Value;
string from = Path.Combine(path, name) + ".csv";
string to = Path.ChangeExtension(from, "txt");
Dts.Log("Starting " + to.ToUpper(), 0, null);
using (StreamReader reader = new StreamReader(from, Encoding.ASCII, false, 10))
using (StreamWriter writer = new StreamWriter(to, false, Encoding.UTF8, 10))
{
while (reader.Peek() >= 0)
{
writer.WriteLine(reader.ReadLine());
}
}
Dts.TaskResult = (int)ScriptResults.Success;

Your code indicates that your are trying to convert an ASCII file to UTF-8 however that article also states the following:
As UTF-8 uses the same character encoding as ASCII PolyBase will also
support loading data that is ASCII encoded.
So my advice to you is to try the file first with Polybase, check for any conversion issues before you spend any time trying to convert the files.

var mySrcFolder = ""; // something from user variables?
var myUtf8stage = ""; // something from user variables?
var myFinalstage = ""; // something from user variables?
// Static variables
var ascii = System.Text.Encoding.ASCII;
var utf16le = System.Text.Encoding.Unicode;
var utf8 = System.Text.Encoding.UTF8;
var ansi = System.Text.Encoding.Default;
var append = false;
// Set source file path and file name
var src = System.IO.Path.Combine(
mySrcFolder,
String.Format("{0}.txt", myUtf8stage));
// Set source file encoding (using list above)
var src_enc = ascii;
// Set target file path and file name
var tgt = System.IO.Path.Combine(
mySrcFolder,
String.Format("{0}.txt", myFinalstage));
// Set target file encoding (using list above)
var tgt_enc = utf8;
using (var read = new System.IO.StreamReader(src, src_enc))
using (var write = new System.IO.StreamWriter(tgt, append, tgt_enc))
{
while (read.Peek() != -1)
{
var line = read.ReadLine();
write.WriteLine(line);
}
}

public void Main()
{
//$Package::SourceSQLObject = tablename
//$Package::StageFile_DestinationFolderPath = rootpath eg "C:\temp\"
string path = (string)Dts.Variables["$Package::StageFile_DestinationFolderPath"].Value;
string name = (string)Dts.Variables["$Package::SourceSQLObject"].Value;
string from = Path.Combine(path, name) + ".csv";
string to = Path.ChangeExtension(from, "txt");
Dts.Log("Starting " + to.ToUpper(), 0, null);
using (StreamReader reader = new StreamReader(from, Encoding.ASCII, false, 10))
using (StreamWriter writer = new StreamWriter(to, false, Encoding.UTF8, 10))
{
while (reader.Peek() >= 0)
{
writer.WriteLine(reader.ReadLine());
}
}
Dts.TaskResult = (int)ScriptResults.Success;

Transfer file through text on sockets. [C# and Python]

I'm currently working on a project that include file transferring. The way I originally planned it was to read it in python via
f = open(filename)
f.read()
and write it in C# using the File object interface, and the same from C#, I'm reading it through
File.ReadAllText(file)
and saving it in python.
It is transferred via sockets.
For some reason, it keeps corrupting the files. Even though the sending is perfectly fine(I checked it a thousand times), the files is read and written properly, so I'm looking for information about how to read a file through text(only text) and not having it corrupted. Any help is welcome, thanks.
Networking(Python):
def send(msg, sock):
msg = msg.__repr__()
size_of_package = sys.getsizeof(msg)
package = str(size_of_package)+":"+ msg
sock.send(package)
def recv(sock):
try:
header = sock.recv(2)
while ":" not in header:
header += sock.recv(2)
size_of_package, separator, message_fragment = header.partition(":")
message = ""
while len(message) < int(size_of_package) - len(message_fragment):
recvsize = int(size_of_package) - len(message_fragment) - len(message)
if recvsize > 2048:
recvsize = 2048
message+=sock.recv(recvsize)
full_message = message_fragment + message
return full_message
except OverflowError:
return "OverflowError."
except:
print "Unexpected error:", sys.exc_info()[0]
raise
Networking C#:
private void Send(string st)
{
int size = Encoding.ASCII.GetByteCount(st);
string pack = size + ":" + st;
buffer = Encoding.ASCII.GetBytes(pack);
connection.Send(buffer);
MessageBox.Show(buffer.Length.ToString());
}
private string Recv()
{
try
{
buffer = new byte[2];
connection.Receive(buffer, 2, SocketFlags.Partial);
string header = Encoding.ASCII.GetString(buffer, 0, 2);
while (!header.Contains(":"))
{
connection.Receive(buffer, 2, SocketFlags.Partial);
header += Encoding.ASCII.GetString(buffer, 0, 2);
}
int size = int.Parse(header.Split(':')[0]);
string mes0 = header.Split(':')[1];
buffer = new byte[size];
connection.Receive(buffer, size, SocketFlags.Partial);
string fullmes = mes0 + Encoding.ASCII.GetString(buffer);
return fullmes;
}
catch(Exception e)
{
MessageBox.Show(e.ToString());
}
return "";
}
File saving (Python)
class fileSaver:
def __init__(self , fileInf):
self.fileInf = fileInf
self.file = open(BASE_PATH+fileInf.getPath(), "wb")
self.file.write(fileInf.getContent())
self.file.close()
where fileInf is an object that contains all the info, including content, etc.
File loading(C#):
StreamReader sr = new StreamReader(file);
networking.Upload(file.Substring(file.LastIndexOf('/')+1), basepath, sr.ReadToEnd());
sr.Close();

C# Importing CSV using OleDb

I'm trying to set up code to import .CSV files into .NET.
I've tried both Microsoft.Jet.OLEDB.4.0 and Microsoft.ACE.OLEDB.12.0 providers, including modifying the Extended Properties and even modifying corresponding registry keys for each. I have yet to come up with a solution for what I am attempting to do:
I would like to import each field as text, but leave fields longer than 255 characters un-truncated.
What I've found so far is that I can have one or the other, but not both.
If I set the ImportMixedTypes registry value to Majority Type, it leaves 255+ character text fields un-truncated, but converts other fields to unwanted types.
If I set the ImportMixedTypes registry value to Text, it truncates 255+ character text fields, but leaves the other field types as text.
How do I accomplish this using OleDb?
Additional info:
I have a "notes" column, which can contain very lengthy text. I also have a "zip code" column, which contains mixed zip-code formats (5-digit and 9-digit with a dash). Typically, the 5-digit zip-code format is more popular, so the importer thinks that the column should be integer type, leaving the 9-digit zip-codes as null values after import.

Have you considered using something as versatile as the FileHelpers library (http://filehelpers.sourceforge.net/) instead?
Or alternatively if your requirements are no more than you state (read csv file, get string fields), use something really simple such as:
public static class SimpleCsvImport
{
public static IEnumerable<List<string>> Import(string csvFileName)
{
using (var reader = File.OpenText(csvFileName))
{
while (!reader.EndOfStream)
{
var fields = reader.ReadLine().Split(new[] { ',' }, StringSplitOptions.None).Select(f => f.Trim()).ToList();
if (fields.Count > 0)
yield return fields;
}
}
}
}

i have implemented this code to read memo field (Microsoft Access):
private string GetMemoField(string TableName, string FieldName, string IdentityFieldName, string IdentityFieldValue, OleDbConnection conn)
{
string ret = "";
OleDbCommand cmd1 = new OleDbCommand("SELECT " + FieldName + " FROM “ + TableName + “ WHERE " + IdentityFieldName + "=" + IdentityFieldValue, conn);
var reader = cmd1.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); // Create the DataReader that will get the memo field one buffer at a time
if (reader.Read())
{
long numberOfChars = reader.GetChars(/*Field pos*/ 0, 0, null, 0, 0); // Total number of memo field's chars
if (numberOfChars > 0)
{
int bufferSize = 1024;
char[] totalBuffer = new char[64*bufferSize]; // Array to hold memo field content
long dataIndex = 0;
do
{
char[] buffer = new char[bufferSize]; // Buffer to hold single read
long numberOfCharsReaded = reader.GetChars(0, dataIndex, buffer, 0, bufferSize);
if (numberOfCharsReaded == 0)
{
ret = new string(totalBuffer,0, (int)numberOfChars);
break;
}
Array.Copy(buffer, 0, totalBuffer, dataIndex, numberOfCharsReaded); // Add temporary buffer to main buffer
dataIndex += numberOfCharsReaded;
} while (true);
}
}
return ret;
}