We Keep Coding

c#.net regex to remove certain non ascii chars does not work

I'm newbie to .net, I use script task in SSIS. I am trying to load a file to Database that has some characters like below. This looks like a data copied from word where - has turned to –
Sample text:
Correction – Spring Promo 2016
Notepad++ shows:
I used the regex in .net script [^\x00-\x7F] but even though it falls in the range it gets replaced. I do not want these characters be altered. What am I missing here?
If I don't replace I get a truncation error as I believe these characters take more than a bit size.
Edit: I added sample rows. First two rows have problem and last two are okay.
123|NA|0|-.10000|Correction – Spring Promo 2016|.000000|gift|2013-06-29
345|NA|1|-.50000|Correction–Spring Promo 2011|.000000|makr|2012-06-29
117|ER|0|12.000000|EDR - (WR) US STATE|.000000|TEST MARGIN|2016-02-30
232|TV|0|.100000|UFT / MGT v8|.000000|test. second|2006-06-09
After good long weekend :) I am beginning to think that this is due to code page error. The exact error message when loading the flat file is as below.
Error: Data conversion failed. The data conversion for column "NAME" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.".
This is what I do in my ssis package.
Script task that validates the flat files.
The only validation that affect the contents of the file is to check the number of delimited columns in the file is same as what it should be for that file. I need to read each line (if there is an extra pipe delimiter (user entry), remove that line from the file and log that into custom table).
Using the StreamWriter class, I write all the valid lines to a temp file and rename/move the file at the end.
apologies but I have just noticed that this process changes all such lines above to something like this.
Notepad: Correction � Spring Promo 2016
How do I stop my script task doing this? (which should be the solution)
If that's not easy, option 2 being..
My connection managers are flat file source and OLEDB destination. The OLEDB uses the default code page which is 1252. If these characters are not a match in code page 1252, what should I be using? Are there any other workarounds without changing the code page?
Script task:
foreach (string file in files)... some other checks
{
var tFile = Path.GetTempFileName();
using (StreamReader rFile = new StreamReader(file))
using (var swriter = new StreamWriter(tFile))
{
string line;
while ((line = rFile.ReadLine()) != null)
{
NrDelimtrInLine = line.Count(x => x == '|') + 1;
if (columnCount == NrDelimtrInLine)
{
swriter.WriteLine(line);
}
}}}
Thank you so much.

It's not clear to me what you intend since "I do not want these characters to be altered" seems mutually exclusive with "they must be replaced to avoid truncation". I would need to see the code to give you further advice.
In general I recommend always testing your regex patterns outside of code first. I usually use http://regexr.com
If you want to match your special characters:
If you want to match anything except your special characters:

How to test for non ASCII characters in a file name

On a asp.net web site, a user tried to upload a file as an email attachment that contained an emdash in the file name. When sending this as an email attachment (exchange server) the file got converted to _utf8_B_****.dat
So, on a .aspx page, I need to be able to detect if an emdash is present in the filename of a file that is uploaded as part of the Request.Files collection.
string s = "a—b-";
byte[] arr = Encoding.ASCII.GetBytes(s);
foreach (byte element in arr)
{
Response.Write(element.ToString() + ",");
}
The string above has an emdash as the second character and a normal hyphen as the fourth character.
The code above prints 97,63,97,45 to the screen.
I assumed that as an emdash is not a valid ASCII character, either an error would be thrown or some indication shown that it was not a valid ASCII character. Yet it returns 63.
How can I detect an emdash in a file name so I can say to the user 'Your file name has an invalid character in it'? I have seen other questions on this issue, I can't get them to work.

This should probably do the trick:
foreach (char c in s) {
if (c >= 128) {
Response.Write("Non-ascii char detected: {0}", c);
}
}
I believe that Encoding.ASCII.GetBytes converts to ASCII first, so you should never see non-ASCII characters when you call that.

How can I detect an emdash in a file name so I can say to the user 'Your file name has an invalid character in it'?
That's the wrong way around, because tomorrow a user will upload a file with another unicode character your filesystem or its API doesn't support. Besides you don't need ASCII, because NTFS can handle a lot more than 7 bytes per character.
The right question is: "What characters can I use to save a file"? But then again you'll be tied to the filesystem implementation. You'd best just generate a random filename and write the file to that path, and store the filename in a database so you can view the original filename.
If you do want to save the file under the user-provided path, you'll have to remove Path.GetInvalidPathChars() and Path.GetInvalidFileNameChars() from your the input.
If the problem is not the filesystem but the mail system, please show relevant code and error message.

Why does my filename cut-off when I copy a file from one directory to another?

I have this line of code that copies a file from one directory to another:
File.copy(Path.Combine(sourceDir, fileName), Path.Combine(destinationDir, fileName), true);
When I step through the code, all variables have the correct values and the full filename, but when I go view the file in the destination directory (destinationDir) the file is there with only part of the original file name...
The original file name is New Employee Orientation - Benefits and when it is copied over to the new destination is it New Employee Orientation - Benefi
I have another file that is similarly named and it cuts off at the same amount of characters (including white spaces).

Is File.copy your own function or a typo? Should be File.Copy if using .NET's Copy function. If it's your own, need to see and walkthrough that code still.
Otherwise, character limit is 260 and a likely culprit. Move the Path.Combine(string,string) outside of File.copy and check the value and length.
See Naming Files, Paths and Namespaces for more information.

Avoid max char limit c#

To exceed the max path limit in c# you apparently need to concatenate your drive path with #"\\?\" at the beginning of it. If I do this then I get a drive path with the following at the front
\\\\?\\\\\\server\\share\\...
Now if I look for the file/folder it will fail because of illegal charachters in the path (I assume the ?) so how can I adopt the approach outlined on Microsoft's website (http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx) correctly?
foreach (string filePath in Directory.GetFiles(folder))
{
String s = #"\\?\" + filePath;
if (filePath.Length > 255)
{
Console.WriteLine(filePath);
}
if (File.Exists(filePath))
{
FileInfo finfo = new FileInfo(s);
folderSize += finfo.Length;
}
}
foreach (string dir in Directory.GetDirectories(folder))
folderSize += GetDirectorySize(dir);

Standard System.IO doesn't support path longer than 260 characters.
But it seems there's a library with extended-length path support:
http://alphafs.codeplex.com/
I personally haven't tried it so far.

I don't think you should add all those slashes before the path, I think you did not understand everything what was written in MSDN here:
Maximum Path Length Limitation
In the Windows API (with some exceptions discussed in the following
paragraphs), the maximum length for a path is MAX_PATH, which is
defined as 260 characters. A local path is structured in the following
order: drive letter, colon, backslash, name components separated by
backslashes, and a terminating null character. For example, the
maximum path on drive D is "D:\some 256-character path string"
where "" represents the invisible terminating null character for
the current system codepage. (The characters < > are used here for
visual clarity and cannot be part of a valid path string.)
Note: File I/O functions in the Windows API convert "/" to "\" as part
of converting the name to an NT-style name, except when using the
"\?\" prefix as detailed in the following sections.
The Windows API has many functions that also have Unicode versions to
permit an extended-length path for a maximum total path length of
32,767 characters. This type of path is composed of components
separated by backslashes, each up to the value returned in the
lpMaximumComponentLength parameter of the GetVolumeInformation
function (this value is commonly 255 characters). To specify an
extended-length path, use the "\?\" prefix. For example, "\?\D:\very
long path".
Note: The maximum path of 32,767 characters is approximate, because
the "\?\" prefix may be expanded to a longer string by the system at
run time, and this expansion applies to the total length.
as you can read there,:
The Windows API has many functions that also have Unicode versions to
permit an extended-length path for a maximum total path length of
32,767 characters.
this is the key for your issue, if you need to create or access to a path longer than ~260 chars you should use specific Windows APIs.

C# Help needed changing code that deletes folders with long paths

A while back I asked a question on stackoverflow about deleting folders that have long paths (>260 characters), the most popular solution was to move into each directory to reduce the length of the path. I've struggled with this and I'm no further on, could someone please suggest how I would intergrate the suggested code into my code?
A typical path is:
\\serverName\share\dave\Private\Careers\Careers Ed\Fun Careers Education\Chris's not used 2006 to07\old 4.Careers Area Activity Week 1 30.10.06 or 6.11.06 or 13.11.06 Introduction to job levels and careers resources\Occupational Areas & Job levels Tutor Help Sheet[1].doc
Many thanks
//Suggested code:
var curDir = Directory.GetCurrentDirectory();
Environment.CurrentDirectory = #"C:\Part\Of\The\Really\Long\Path";
Directory.Delete("Relative\Path\To\Directory");
Environment.CurrentDirectory = curDir;
//My code:
try
{
var dir = new DirectoryInfo(#FolderPath);
dir.Attributes = dir.Attributes & ~FileAttributes.ReadOnly;
dir.Delete();
}
catch (IOException ex)
{
MessageBox.Show(ex.Message,"Delete Error",MessageBoxButtons.OK,MessageBoxIcon.Error);
}

Before 'removing a directory' we have to be sure that it is empty. You could consider using the reverse 'directory walk' approach.
This would entail dealing with each directory seperately in deep-to-shallow order.
Some pseudo code;
While (fullPath.Length > 0)
{
DirectoryToDelete = GetLastPartOfPath( fullPath );
CurrentDirectory = fullPath - DirectoryToDelete;
ChangeDirectory(CurrentDirectory);
DeleteDirectory(DirectoryToDelete);
fullPath = fullPath - DirectoryToDelete;
}
Hope this helps,

Have you tried using the long path name syntax ?
From the CreateFile function in the platform SDK:
Maximum Path Length In the Windows API
(with some exceptions discussed
later), the maximum length for a path
is MAX_PATH, which is defined as 260
characters. A local path is structured
in the following order: drive letter,
colon, backslash, components separated
by backslashes, and a terminating null
character. For example, the maximum
path on drive D is "D:\<256
chars>NUL".
The Windows API has many functions
that also have Unicode versions to
permit a maximum path length of
approximately 32,000 characters
composed of components up to 255
characters each in length. To specify
that kind of extended length path, use
the "\?\" prefix. For example,
"\?\D:\".
Note The maximum path of 32,000
characters is approximate, because the
"\?\" prefix can be expanded to a
longer string, and the expansion
applies to the total length.
To specify such a path using UNC, use
the "\?\UNC\" prefix. For example,
"\?\UNC\\". These
prefixes are not used as part of the
path itself. They indicate that the
path should be passed to the system
with minimal modification, which means
that you cannot use forward slashes to
represent path separators, or a period
to represent the current directory.
Also, you cannot use the "\?\" prefix
with a relative path. Relative paths
are limited to MAX_PATH characters.
The last paragraph is of course the one that is relevant to your case.
It is not sure that .NET supports this kind of path. You could use P/Invoke to call RemoveDirectory from the Win32 SDK.

Use ZetaLongPaths. It handles long paths. Google ZetaLongPaths