We Keep Coding

Retrieving shell script exit code (return value)

Trying to validate a Logstash config file. When running the following line from a Windows command line:
C:> C:\Logstash\bin\logstash -t -f C:\Logstash\config\my.config
I can then check the result using
echo %errorlevel%
which returns 1 in case of a syntax error. Now I want to do this programatically in C#, so:
using System.Diagnostics;
var logstashProcess = Process.Start(#"C:\Logstash\bin\logstash", #"-t -f C:\Logstash\config\my.config");
logstashProcess.WaitForExit();
return logstashProcess.ExitCode == 0;
The problem is that it always returns true (exit code is zero) - even when the config file is totally messed up.
My guess: since C:\Logstash\bin\logstash is a shell script, the zero I get is the shell itself running successfully - not the Logstash process (which is executed from within that script using jruby). Any idea on how to get the real return value? Will a batch file work? (I prefer not to add an extra script to the party at this point)

Invalid character in bcp

I'm trying to call a BCP command from c#.I'm building the command within the c# code, and am getting an error. Here is the BCP command as I'm building it:
string.Format("bcp \"EXEC {0} {1}\" queryout \"{2}\" -c -t\"\0\" -S {3} -U {4} -P {5}", spName, filterList, path, server, user, password);
Here is the command after formatting:
"bcp \"EXEC StatisticsByType #Date='1/1/2013'\" queryout \"\\\\Server\\Folder\\MagicWords.txt\" -c -t\"\0\" -S server.com -U userName -P password"
The error I'm getting is:
{"Incorrect syntax near 'queryout'.\r\n'.' is an invalid name because it contains a NULL character or an invalid unicode character."}
I'm assuming it has something to do with how .Net is escaping the ". Would anyone know the proper way to pass those quotation marks around, or if thats even the issue?
EDIT: I ran the BCP command in question from the powershell ISE. It threw an "Unable to open BCP host data-file" error. So I changed the queryout server name to it's mapped network drive name, and It worked. However, doing the same thing in C# yielded no change.

I suspect the problem is the delimiter. The sequence -t\"\0\" translates to a NULL character between quotes. You might try replacing \0 with another delimeter like , or ;. If you do want the NULL character as a delimiter you should insert -t\\0 in your string.

Problem with credential chaining sequence in Windows and .NET

Problem Summary
When run in batch mode, my user identity is lost but (like a really good cliffhanger) not until the very last moment.
Problem Details
I have a PowerShell script running on WinXpSp3 that runs a script block (via Invoke-Command) on a remote machine as a particular test user (via -Session parameter) in the background (via -AsJob parameter). The session is created with this:
New-PSSession -computername $myServer -credential $myCredential
The script block performs a number of actions, culminating in running the NUnit test framework. The C# code under test records the "myTestUser" username (via Environment.UserName) so the credentials provided by PowerShell are properly received that far. This is further confirmed by Process Explorer: examining properties of nunit-console running the batch tests shows it is owned by myTestUser.
The test includes accessing a Sql Server 2008 R2 database; the connection string is set via a new SqlConnection(connectionString) call. The connection string is set up for Windows Authentication and uses this form:
Data Source=<my_db_server_name>;Initial Catalog=<my_db_name>;Integrated Security=True;Persist Security Info=True
Even though I have conclusively pushed the myTestUser credentials all the way to the C# code under test, the DB access attempt is not seeing these credentials, resulting in this error: Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'
Some supplemental info:
I have confirmed that the test user (myTestUser) has DB permissions and the NUnit test is capable of accessing the DB: When I run the NUnit test manually (via NUnit GUI) logged in as myTestUser, the test works properly and SqlProfiler clearly shows this activity with myTestUser appearing in the NTUserName column.
The same error occurs if I run locally rather than on a remote machine.
The same error occurs if I run as myself on my local machine (i.e. omitting the -credential parameter).
Question
How can I rescue myTestUser from the brink of doom and get him DB access?
2011.05.16 Update
Here is a simplified example exhibiting the same problem.
First, my test program DBFetchVersion that prints the name of the current user and the results of a simple query:
class Program
{
const string connString = ...your connection string here... ;
const string query = "SELECT getdate() [Date], substring(##version,1,charindex('-',##version)-1) +convert(varchar(100),SERVERPROPERTY('edition'))+ ' ' +convert(varchar(100),SERVERPROPERTY('productlevel')) [SQL Server Version], ##servicename [Service Name], ##servername [Server Host], db_name() [Database], user_name() [User], host_name() [Client]";
static void Main(string[] args)
{
DataView dataView;
using (var connection = new SqlConnection(connString))
{
Console.WriteLine("user = " + Environment.UserName);
using (var dataAdapter = new SqlDataAdapter(query, connection))
{
var dataSet = new DataSet();
try
{
connection.Open();
dataAdapter.SelectCommand.CommandType = CommandType.Text;
dataAdapter.Fill(dataSet, query);
}
finally { if (connection.State == ConnectionState.Open) connection.Close(); }
dataView = dataSet.Tables[0].DefaultView;
}
foreach (var item in dataView.Table.Rows[0].ItemArray)
Console.WriteLine(item);
}
}
}
And here is the Powershell script that calls the above program.
$scriptBlock = {
& "...path to my executable...\DBFetchVersion\bin\Debug\DBFetchVersion.exe"
}
$serverName = ... my server name ...
$username = "testuser"
$password = ... my user password ...
$adjPwd = $password | ConvertTo-SecureString -asPlainText -Force
$testCred = (New-Object System.Management.Automation.PSCredential($username,$adjPwd))
$mySession = New-PSSession -computername $serverName -credential $testCred
# Test Scenarios:
Invoke-Command $scriptBlock
#Invoke-Command $scriptBlock -computername $serverName
#Invoke-Command $scriptBlock -computername $serverName -credential $testCred
#Invoke-Command $scriptBlock -Session $mySession
In the list of four test scenarios at the end, the uncommented one works, printing my user name and the results of the query.
DBFetchVersion still reports I am the user with the second line, but the DB connection fails with the " Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON' " error.
The remaining two lines report the "testuser" user name, but both report the same login failure for the DB connection.
What this isolated example tells me is not that I think there is anything buggy about Powershell, .NET, or my code, but there is something with the authentication mechanism that I do not yet understand, since specifying another computer or a session both involve a path that should, in some sense, have stronger protection.
2011.08.03 Update - Eureka!
Well, Matt was correct in identifying the double-hop issue as the culprit and CredSSP authentication as the solution. Unfortunately, as I quickly found out, CredSSP requires Windows 7, so I went about setting up a couple VMs as a sandbox. CredSSP, however, was not one to easily relinquish its secrets (at least to me) as I detailed in this post on ServerFault: Cannot get CredSSP authentication to work in PowerShell
I finally was able to get CredSSP authentication to work so I could then come back to the problem I posed here in this thread. As a test, I used these 3 script blocks plugged into the PowerShell script I provided above:
$scriptBlockA = {
Write-Host ("hello, world: {0}, {1}" -f $env:USERNAME, (hostname))
}
# Simple DB test, but requires SqlServer installed!
$scriptBlockB = {
if (! (Get-PSSnapin | ? { $_.name -eq "SqlServerCmdletSnapin100" } ) )
{ Add-PSSnapin SqlServerCmdletSnapin100; }
Invoke-Sqlcmd -Query "SELECT getdate() as [Now]" -ServerInstance CinDevDB5
}
# Indirect DB test; requires .NET but not SqlServer,
# plus DBFetchVersion in home dir for targeted user.
$scriptBlockC = {
& ".\DBFetchVersion.exe"
}
Block A worked with or without CredSSP, since there is no double-hop. Blocks B and C would only work with CredSSP because they both attempt to access a remote database. QED.

Initially i read this and thought of the "double hop" issue, but the supplemental info maybe me question that though.
When you run it locally (as yourself or the testuser) what commands do you use? this:
& "...path to my executable...\DBFetchVersion\bin\Debug\DBFetchVersion.exe"
also does this work from your local machine (as either yourself or the user):
Add-PSSnapin SqlServerCmdletSnapin100;
Invoke-Sqlcmd -Query "SELECT getdate()" -ServerInstance Server
Also what OS are you using? If it is Windows 2008 and the issue is double hop you may be able to us CredSSP to avoid it.

Cannot access Sqlite Database (of Firefox) using System.Data.Sqlite

I wrote a small C#/.Net application that is able to read the cookies.sqlite file of Firefox. Since I upgraded to Firefox 4 my application is not able to open the database file:
Executing the line "connection.Open();" (in the code sample below) there will be an execption that says:
"File opened that is not a database file. file is encrypted or is not a database"
This is my program code:
class Program
{
static void Main()
{
const string PATH_TO_DATABASE = #"C:\Users\Boris\Desktop\TEMP\cookies.sqlite";
const string CONNECTION_STRING = #"Data Source=" + PATH_TO_DATABASE;
if (!File.Exists(PATH_TO_DATABASE)) return;
using (SQLiteConnection connection = new SQLiteConnection(CONNECTION_STRING))
{
connection.Open();
using (SQLiteCommand command = new SQLiteCommand("SELECT id, name, host, path FROM moz_cookies", connection))
{
using (SQLiteDataReader read = command.ExecuteReader())
{
while (read.Read())
{
string id = read[0].ToString();
string name = read[1].ToString();
string host = read[2].ToString();
string path = read[3].ToString();
Console.WriteLine("ID: " + id);
Console.WriteLine("Name: " + name);
Console.WriteLine("Host: " + host);
Console.WriteLine("Path: " + path);
}
}
}
}
}
}
I am using the .Net Wrapper DLL for Sqlite v. 3.6.23.1.
The target framework of the application is .Net 2.0.
I was able to open the sqlite database without any problems using an application called SqliteExpert.
It would be great if anybody has an idea!
Regards,
Boris

Firefox 4.0 uses SQLite version 3.7.4. (To see this, download SQLite Manager and run select sqlite_version(); in the "Execute SQL" tab.)
It appears that v3.7.4 creates databases that can't be read by v3.6.23.1. I can't find this explicitly stated in the release notes, but it's clear from comparing the cookies.sqlite file to the file format documentation that the format has changed. Specifically, according to the documentation, byte 19 (0x13) should be 1, but that byte in the cookies.sqlite file is 2. As per the documentation:
If a value greater than 1 is read by
SQLite, then the library will refuse
to open the database.
Like the "write version" described
above, this field exists to facilitate
some degree of forwards compatibility,
in case it is ever required. If a
version of SQLite created in the
future uses a file format that may not
be safely read by older SQLite
versions, then this field will be set
to a value greater than 1.
To read the database, you will need to use the latest version of SQLite; unfortunately System.Data.SQLite hasn't been updated in almost a year. The project has since been taken over by sqlite.org, but downloads are not currently available.
If you don't want to wait for sqlite.org to release a v3.7.x-based .NET wrapper, you could try downloading the source and manually upgrading SQLite.NET/SQLite.Interop/src/sqlite3.c to the latest amalgamation C source file (available here).
EDIT: As noted by sdwilsh, Firefox 4.0 uses Write-Ahead Logging; as the description of that new journal mode states:
Thus, if an older version of SQLite
attempts to connect to an SQLite
database that is operating in WAL
mode, it will report an error along
the lines of "file is encrypted or is
not a database".

I had the same problem. Previously I used a Python script (with sqlite integration) to fetch up data from the cookies.sqlite and perms.sqlite files. The latter still works fine. Indeed, all of the other .sqlite files are readable ... except for cookies.sqlite and places.sqlite. Those two produce the "encrypted or is not a database" error.
Rather than hunt down an upgrade for Python (or sqlite3), I created a patch for the .sqlite files. This C-code changes the bytes at offset 18 and 19 from 2 to 1. I run this on Windows under Cygwin, but it should compile and run on Unix/Linux.
Warning: Don't do this to the original Firefox cookies.sqlite file.
Instead, copy that to a temp file, then run the patch on the copy.
// ffpatch.c
// Edits the specified Firefox .sqlite file.
// Changes 0x0202 to 0x0101 at offset 18/19.
// BEFORE
// 0000000 S Q L i t e f o r m a t 3 \0
// 0000020 004 \0 002 002 \0 # \0 \0 005 034 \0 \0 \0 N
// AFTER
// 0000000 S Q L i t e f o r m a t 3 \0
// 0000020 004 \0 001 001 \0 # \0 \0 005 034 \0 \0 \0 N
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <fcntl.h>
#define PROGNAME "ffpatch"
#define MY_OFFSET 18
int main (int argc, char *argv[])
{
int fd;
char buf[2];
off_t offset;
ssize_t wsize;
if (argc != 2)
{
fprintf(stderr, "usage: %s sqlite-file\n", PROGNAME);
exit(1);
}
if ((fd = open(argv[1], O_RDWR)) == -1)
{
fprintf(stderr, "cannot open %s\n", argv[1]);
exit(1);
}
if ((offset = lseek(fd, MY_OFFSET, SEEK_SET)) != MY_OFFSET)
{
fprintf(stderr, "lseek() failed\n");
exit(1);
}
buf[0] = 1;
buf[1] = 1;
wsize = write(fd, buf, 2);
close(fd);
exit(0);
}

Hello and thanks alot for your answers!
I made my application work this way:
1.
I downloaded the sources of the current ADO.NET Connector for .NET 4.0 here: http://system.data.sqlite.org/index.html/timeline?r=trunk
Maybe you will have to logon on the website first using a anonymous user name and a capcha as passoword.
2.
I obtained the precompiled version of the sqlite3.dll v.3.7.4 by downloading an installing this ADO.NET connector: http://www.devart.com/dotconnect/sqlite/
(You may use the ADO.NET connector libs as an replacement for connector from system.data.sqlite.org as well.
I myself am only interested in the sqlite3.dll.)
3.
After compiling the sources from system.data.sqlite.org I copied the resulting System.Data.Sqlite.dll and the sqlite3.dll to my applications output directory. Please note that both DLLs are compiled either for x86 or x64 machines.
Regards

I think I got your problem. Your db connection is OK. First of all you are using which version of dotnetFramework? Accordingly you can download and use System.Data.SQLite.dll file to your references then your problem may be solved. I think you are using a System.Data.SQLite.dll file that is older one (doesn't match your version of dot net).

The ';' character, hexadecimal value 0x3B, cannot be included in a name

I have seen the error "The ';' character, hexadecimal value 0x3B, cannot be included in a name." in my log files for an ASP.NET Web App. The url that's logged looks something like this:
mypage.aspx?paramone=one+two&paramtwo=zero+1
So my first question is what type of system/browser is encoding the original query string? (This happens rarely)
I've tried to address this problem with the following snippet of code in the Page_Load() event:
string rawUrl = Request.RawUrl;
if (rawUrl.Contains(amp))
{
rawUrl = rawUrl.Replace("&", "&");
Server.Transfer(rawUrl, false);
return;
}
However when it transfers back to this page the & is back in the query string. So I'm guessing that the .Transfer() function encodes the first param.
Suggestions about solving this problem?

Your web server should be able to log the "user agent" field from the HTTP Request, which should enable you to identify the culprit.
Don't fix it - there's a very well defined set of legal syntaxes for URI parameters, and this ain't one of them.

When you try to export a Microsoft catalog to XML, the resulting file cannot be imported, and you receive the following error message
"The XML file path/filename contains an error at line. " "A Name contained an invalid character."
If you validate the XML catalog by using Microsoft Visual Studio .NET you receive the following error message:
"The '(' character, hexadecimal value 0x28, cannot begin a name. Line #, Position #"
This problem occurs because the Commerce Server export was not encoding the following special characters:
The range 0x0021 – 0x002F includes ! “ # $ % & ‘ ( ) * + , - . /
The range 0x03A – 0x0040 includes : ; < = > ? #
The range 0x007B – 0x007E includes { | } ~
The range 0x005B – 0x005E [ \ ] ^**