I have an SSIS data flow that contains a script Component as a source. I'd like to generate the source data by running a script on an SQL Server database. The connection string to be used to connect to the database is set to be sensitive. How can I read this sensitive parameter inside the script component using C#?
In a Script Task, usually it is read for example as the following:
string mySecretPassword = Dts.Variables["$Project::MySecretPassword"].GetSensitiveValue().ToString
The Variable class in a Script Task has a GetSensitiveValue method. However, the Script Component Variable implements the interface Microsoft.SqlServer.Dts.Runtime.Wrapper.Variable which has no GetSensitiveValue method defined.
Let's assume the connection string is a project parameter for now.
Despite the lack of a GetSensitiveValue method existing in the Script Component, I was able to access the value just fine.
What I did fumble with was my Package Protection level and how it interacts with Project Parameters that are marked as Sensitive.
I defined a Project Parameter named MySecretPassword and populated it with SO_71308161 and marked it as sensitive.
Script Source
I defined a single column output and my intention was to just push the password into the dataflow to confirm I was able to access it
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public string MySecret;
public override void PreExecute()
{
base.PreExecute();
MySecret = Variables.MySecretPassword;
}
public override void CreateNewOutputRows()
{
Output0Buffer.AddRow();
Output0Buffer.Column = this.MySecret;
}
}
I then routed the data to a derived column and dropped a data viewer between the two. My code did not throw an exception but it did show an empty string.
If I tried to access my sensitive parameter value in a script task using myVariable.Value it would throw an error as expected but I was getting my sensitive value back.
Package Protection Level
I love StackOverflow questions that teach me things. My default mode is to define projects that use a Project & Package protection level of DontSaveSensitive. Which is incompatible with using Sensitive project parameters. Not incompatible in that it will throw an exception, but when you run the package, accessing that value will be blanked out (for strings at least).
As soon as I change the Project and then Package's protection level to EncryptSensitiveWithUserKey (or anything that isn't DontSaveSensitive), both the Script Task's .GetSensitiveValue() and the Script Component's Variables.MySecretPassword worked.
I'm trying to build a simple custom SSIS component which looks at a single input column and validates it, creating an output column of type bool depending on the value of each row.
I've successfully built an even simpler component that takes a value and transforms it: that doesn't require fiddling with the output columns. In this instance I need to take in a string and output a boolean and the component needs to know that it outputs a boolean so I can feed the value into a conditional split.
I'm struggling to add the output columns. Based on code samples from Microsoft, I have done this:
public override DTSValidationStatus Validate()
{
IDTSOutput100 output = ComponentMetaData.OutputCollection[0];
IDTSOutputColumn100 outputcol = output.OutputColumnCollection.New();
outputcol.Name = "IsValid";
outputcol.SetDataTypeProperties(DataType.DT_BOOL, 0, 0, 0, 0);
return DTSValidationStatus.VS_ISVALID;
}
And then I attempt to populate it during the ProcessInput step:
public override void ProcessInput(int inputID, PipelineBuffer buffer)
{
while (buffer.NextRow())
{
string str = buffer.GetString(0);
buffer.SetBoolean(0, IsValid(str)); // validation code not relevant
}
}
When I try to use this component in the package, I get this error:
The component has detected potential metadata corruption during validation.
Error at Data Flow Task [Uppercase [24]]: System.MissingMethodException: Method not found: 'Void Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSOutputColumn100.SetDataTypeProperties(Microsoft.SqlServer.Dts.Runtime.Wrapper.DataType, Int32, Int32, Int32, Int32)'.
at EmailValidation.Uppercase.Validate()
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostValidate(IDTSManagedComponentWrapper100 wrapper)
Searching on this error message has yielded nothing of value.
In the original sample - and some other tutorials online - adding output columns is done by looping through the input column and adding an additional output for each. I have tried this and get the same error.
I have also tried moving the output column code from Validate to OnInputPathAttached which still yields the same error.
What am I doing wrong?
On investigation this appears to be a bug in SQL Server Data Tools for Visual Studio 2015. I have built, deployed and used a custom component with customised output columns in an Integration Services package in Visual Studio 2013. However, the same tool deployed in a package in 2015 causes the error described.
In case it's still relevant, I encountered a similar issue (With the ComponentMetaData property rather than SetDataTypeProperties), and the solution that worked for me was setting the Embed Interop Types property of the Microsoft.SqlServer.DTSPipelineWrap and Microsoft.SQLServer.DTSRuntimeWrap references to false.
I found this solution here. It's listed as a solution for getting an InvalidCastException, but it seems relevant for whenever you're referencing the DTSPipelineWrap or the DTSRuntimeWrap assemblies in custom components.
I am writing a program which takes an ethereum contract ABI string in C# and uses it to call a function which allows me to interact with the smart contract using nethereum. When I use a smaller ABI string I am able to make it work but for some reason it won't work with this longer string or any other ABI which exceeds the length of a small example ABI. When I compile the code below I get the error message (bare in mind that I have tried JSON deserializing but that still makes the same error):
An exception of type 'Newtonsoft.Json.JsonReaderException' occurred in Newtonsoft.Json.dll but was not handled in user code
Additional information: After parsing a value an unexpected character was encountered: t. Path '[0].outputs[0].name', line 3, position 20.
protected void init()
{
web3 = new Nethereum.Web3.Web3();
string abi = #"[{'constant':false,'inputs':[{'name':'username','type':'string'},
{'name':'location','type':'string'}],'name':'addUser','outputs':
[{'name':','type':'string'}],'type':'function'},{'constant':false,'inputs'
:[],'name':'burnCoins','outputs':[{'name':','type':'uint256'}],'type':'function'},
{'constant':false,'inputs':[{'name':'vendor','type':'address'},
{'name':'recipient','type':'address'}],'name':'trade','outputs':[],
'type':'function'},{'constant':false,'inputs':[{'name':'vendor','type':'address'},
{'name':'isPositive','type':'bool'},{'name':'message','type':'string'}],
'name':'giveReputation','outputs':[],'type':'function'},{'constant':false,'inputs':
[{'name':'user','type':'address'}],'name':'showBurnedCoins','outputs':[{'name':
','type':'uint256'}],'type':'function'},{'constant':false,'inputs':[{'name':'user',
'type':'address'}],'name':'viewReputation','outputs':[{'name':','type':'uint256'},
{'name':','type':'uint256'},{'name':','type':'uint256'}],'type':'function'},
{'anonymous':false,'inputs':[{'indexed':true,'name':'user','type':'address'},
{'indexed':true,'name':'amountBurned','type':'uint256'}],'name':'_coinsBurned',
'type':'event'},{'anonymous':false,'inputs':[{'indexed':true,'name':'user',
'type':'address'},{'indexed':true,'name':'message','type':'string'}],'
name':'_positiveReputation','type':'event'},{'anonymous':false,'inputs'
:[{'indexed':true,'name':'user','type':'address'},{'indexed':true,'name':'message'
,'type':'string'}],'name':'_negativeReputation','type':'event'},
{'anonymous':false,'inputs':[{'indexed':true,'name':'username','type':'string'},
{'indexed':true,'name':'location','type':'string'},{'indexed':true,'name':
'user','type':'address'}],'name':'_addUser','type':'event'},{'anonymous':false,
'inputs':[{'indexed':true,'name':'vendor','type':'address'},{'indexed':true,
'name':'buyer','type':'address'}],'name':'_newTrade','type':'event'},{'anonymous':
false,'inputs':[{'indexed':true,'name':'user','type':'address'},{'indexed':true,
'name':'positive','type':'uint256'},{'indexed':true,'name':'negative','type':'uint256'},
{'indexed':false,'name':'total','type':'uint256'}],'name':'_viewedReputation',
'type':'event'}]";
string contractAddress = "0xd53c3dc2f3fcf1779b68ea8e441d857b4af5a413";
Reputation = web3.Eth.GetContract(abi, contractAddress);
}
From my comment:
The error is because of the line 'name':'showBurnedCouns', 'outputs':[{'name':','type':'uint256'}]. A ' is missing inside the outputs block.
See diiN_ comment on my question as this answers the question. Silly me!
For my log4net solution, I have an API wrapper that uses the CallerInfo attributes, e.g.
public void Write(string message,
[CallerMemberName] string memberName = "",
[CallerFilePath] string filePath = "",
[CallerLineNumber] int lineNumber = 0)
However, I am also using Unity Interception so that I can perform trace logging of the before/after responses, e.g. using ICallHandler like below in the Invoke method.
public class TraceCallHandler : ICallHandler
{
...
public IMethodReturn Invoke(IMethodInvocation input,
GetNextHandlerDelegate getNext)
{
//---- Trace method inputs
this.LogInfoBeforeInvoke(input);
//---- invoking the target method
InvokeHandlerDelegate next = getNext();
IMethodReturn methodReturn = next(input, getNext);
//---- invoking the target method
this.LogInfoAfterInvoke(methodReturn.ReturnValue);
}
}
Note: The above code is in no way complete/correct... but just wanted to show you what I was doing for Unity Interception.
My question / challenge is this:
when I eventually call log.Write(...), I want the target's caller info, not my TraceCallHandler info.
e.g. for method name, I can do this:
string methodName = input.MethodBase.Name;
How do I get the Caller's File Path and Caller's Line Number? Is it even possible to do via reflection?
Thanks!
Yes, you can get these using reflection:
var sf = new System.Diagnostics.StackTrace(1).GetFrame(0);
Console.WriteLine(" File: {0}", sf.GetFileName());
Console.WriteLine(" Line Number: {0}", sf.GetFileLineNumber());
// Note that the column number defaults to zero
// when not initialized.
Console.WriteLine(" Column Number: {0}", sf.GetFileColumnNumber());
However as it says clearly in the documentation:
StackFrame information will be most informative with Debug build
configurations. By default, Debug builds include debug symbols, while
Release builds do not. The debug symbols contain most of the file,
method name, line number, and column information used in constructing
StackFrame objects.
So if all you want this for is debugging, then enable it in debug builds and log away. In Release builds though it will be at best unhelpful and at worst downright misleading as apart from the symbol considerations above the compiler will aggressively inline methods and reorder things and generally mess with your stuff.
I just ran across this issue and thought I would share what I learned. First, when you include [CallerFilePath] in a method argument a side effect is that the full path of the file, including any user identifiable data, will be included in your .exe. I created a simple program with one method. I created an exe. I then added a [CallerFilePath] attribute to the test function. When I compared the results of strings.exe (from sysinternals), the one with the attribute differed in that it included the full path of my source file.
c:\users\<my name>\documents\visual studio 2015\Projects\TestCallerAttribute\TestCallerAttribute\Program.cs
The answer above by stuartd is correct in that you will not be able to get the data you want from the stack trace in a release build.
There is a solution to getting strong data however: Event Tracing for Windows. From msdn: "Event Tracing for Windows (ETW) is an efficient kernel-level tracing facility that lets you log kernel or application-defined events to a log file. You can consume the events in real time or from a log file and use them to debug an application or to determine where performance issues are occurring in the application."
This is not a quick solution. There is work in setting up the events and the listeners to get the provenance you need. The long term payoff is strong.
It appears msbuild writes all output (including error output) to standard output.
Is there some way to have it write error output (what's normally output in red) to standard error instead?
I'm writing a .NET application with a WPF and console interface and calling msbuild using System.Diagnostics.Process. I'd like to be able to distinguish error output to the user somehow.
Is there a better of separating the output than looking for "error " in each line or using Microsoft.Build directly + a custom logger?
Take a look at the MSBUILD.EXE command line arguments page. Specifically the consoleloggerparameters switch.
You can use /clp:ErrorsOnly to display only errors in the console output.
If you need the rest of the output, include an errors only file log with
/fl4 /flp4:errorsOnly;logfile=MSBuild.Errors.log
then monitor the file for new lines.
I know you said you wanted to avoid a custom logger, but... I also wanted error output on stderr and found that writing a custom logger was not that painful - 1 class with 1 method with 1 statement:
using Microsoft.Build.Utilities;
using Microsoft.Build.Framework;
public class ErrorOnlyLogger : Logger
{
public override void Initialize(IEventSource eventSource)
{
eventSource.ErrorRaised += (s, e) => {
System.Console.Error.WriteLine(
"{0}({1},{2}): error {3}: {4} [{5}]",
e.File, e.LineNumber, e.ColumnNumber, e.Code, e.Message, e.ProjectFile);
};
}
}
This uses the same error formatting as msbuild. This works with command-line msbuild 14.0.25420.1. The code references Microsoft.Build.Utilities.Core.dll and Microsoft.Build.Framework.dll in C:\Program Files (x86)\MSBuild\14.0\Bin. I add /logger:ErrorOnlyLogger,path\ErrLogger.dll to the command line to invoke it.