We Keep Coding

Apache Ignite, C#: Distributed Computing - Topology projection error

I am trying to emulate the Distributed Computing examples in the Apache doc, specifically the Broadcasting example.
I start the cluster remotely, where 2 nodes make up the cluster with the following XML configuration file:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>[remoteHost]:47500..47509</value>
<value>[localHost1]:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Then, I start a node locally with an IP address of [localHost1]:
class PrintNodeIdAction : IComputeAction
{
public void Invoke()
{
Console.WriteLine("Hello node: " +
Ignition.GetIgnite().GetCluster().GetLocalNode().Id);
}
}
static void Main(string[] args)
{
var cfg = new IgniteConfiguration
{
DiscoverySpi = new TcpDiscoverySpi
{
IpFinder = new TcpDiscoveryStaticIpFinder
{
Endpoints = new[] { "[remoteHost]", "[remoteHost]:47500..47509" }
}
}
};
using (IIgnite ignite = Ignition.Start(cfg))
{
// Limit broadcast to remote nodes only.
var compute = ignite.GetCluster().ForRemotes().GetCompute();
// Print out hello message on remote nodes in the cluster group.
compute.Broadcast(new PrintNodeIdAction());
}
}
However, I run into the following two (2) errors after starting the local node:
**1 of 2**
ClusterGroupEmptyException: Topology projection is empty.
**2 of 2**
JavaException: class org.apache.ignite.internal.cluster.ClusterGroupEmptyCheckedException: Topology projection is empty.
at org.apache.ignite.internal.processors.task.GridTaskWorker.getTaskTopology(GridTaskWorker.java:688)
at org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:503)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:830)
at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:498)
at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:466)
at org.apache.ignite.internal.IgniteComputeImpl.executeAsync0(IgniteComputeImpl.java:564)
at org.apache.ignite.internal.processors.platform.compute.PlatformCompute.executeNative0(PlatformCompute.java:329)
at org.apache.ignite.internal.processors.platform.compute.PlatformCompute.processClosures(PlatformCompute.java:294)
at org.apache.ignite.internal.processors.platform.compute.PlatformCompute.processInStreamOutObject(PlatformCompute.java:140)
at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79)

log4net database logging with custom parameters

I have database logging in place using the AdoNetAppender. What I'd like to do is log the user identity on each log statement. However, I don't want to use the standard log4net %identity parameter for two reasons:
log4net warn that its extremely slow as it has to look up the context identity.
In some service components the standard identity is a service account but we have already captured the user identity in a variable and I'd like to use that.
I have seen code where some people use the log4net.ThreadContext to add additional properties but I understand that this is 'unsafe' due to thread interleaving (and it is also a performance drain).
My approach has been to extend the AdoNetAppenderParameter class thus:
public class UserAdoNetAppenderParameter : AdoNetAppenderParameter
{
public UserAdoNetAppenderParameter()
{
DbType = DbType.String;
PatternLayout layout = new PatternLayout();
Layout2RawLayoutAdapter converter = new Layout2RawLayoutAdapter(layout);
Layout = converter;
ParameterName = "#username";
Size = 255;
}
public override void Prepare(IDbCommand command)
{
command.Parameters.Add(this);
}
public override void FormatValue(IDbCommand command, LoggingEvent loggingEvent)
{
string[] data = loggingEvent.RenderedMessage.Split('~');
string username = data[0];
command.Parameters["#username"] = username;
}
}
and then programmatically add this to the current appender like so:
ILog myLog = LogManager.GetLogger("ConnectionService");
IAppender[] appenders = myLog.Logger.Repository.GetAppenders();
AdoNetAppender appender = (AdoNetAppender)appenders[0];
appender.AddParameter(new UserAdoNetAppenderParameter());
myLog.InfoFormat("{0}~{1}~{2}~{3}", userName, "ClassName", "Class Method", "Message");
The intention here is to use a standard format for messages and parse the first part of the string which should always be the username. The FormatValue() method of the custom appender parameter should then use only that part of the string so that it can be written to a separate field in the log database.
My problem is that no log statements are written to the database. Oddly, when debugging, a breakpoint in the FormatValue() method is only hit when I stop the service.
I've trawled through loads of stuff relating to this but haven't yet found any answers.
Has anyone managed to do this, or am I on the wrong trail.
P.S. I've also tried extending the AdoNetAppender but it doesnt give you access to set the parameter values.

I also needed to log structured data and liked to use logging interface like this:
log.Debug( new {
SomeProperty: "some value",
OtherProperty: 123
})
So I also wrote custom AdoNetAppenderParameter class to do the job:
public class CustomAdoNetAppenderParameter : AdoNetAppenderParameter
{
public override void FormatValue(IDbCommand command, LoggingEvent loggingEvent)
{
// Try to get property value
object propertyValue = null;
var propertyName = ParameterName.Replace("#", "");
var messageObject = loggingEvent.MessageObject;
if (messageObject != null)
{
var property = messageObject.GetType().GetProperty(propertyName);
if (property != null)
{
propertyValue = property.GetValue(messageObject, null);
}
}
// Insert property value (or db null) into parameter
var dataParameter = (IDbDataParameter)command.Parameters[ParameterName];
dataParameter.Value = propertyValue ?? DBNull.Value;
}
}
Now log4net configuration can be used to log any property of given object:
<?xml version="1.0" encoding="utf-8"?>
<log4net>
<appender name="MyAdoNetAppender" type="log4net.Appender.AdoNetAppender">
<connectionType value="System.Data.SqlClient.SqlConnection, System.Data, Version=1.0.3300.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<connectionString value="... your connection string ..." />
<commandText value="INSERT INTO mylog ([level],[someProperty]) VALUES (#log_level,#SomeProperty)" />
<parameter>
<parameterName value="#log_level" />
<dbType value="String" />
<size value="50" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%level" />
</layout>
</parameter>
<parameter type="yourNamespace.CustomAdoNetAppenderParameter, yourAssemblyName">
<parameterName value="#SomeProperty" />
<dbType value="String" />
<size value="255" />
</parameter>
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="MyAdoNetAppender" />
</root>
</log4net>

After some experimentation, I finally got this to work. Ensuring that log4net's internal logging helped identify the errors and downloading the log4net source code and reviewing the AdoNetAppenderParameter class showed how the FormatValue() method should be used. So, here's the amended custom appender parameter:
public class UserAdoNetAppenderParameter : AdoNetAppenderParameter
{
public override void FormatValue(IDbCommand command, LoggingEvent loggingEvent)
{
string[] data = loggingEvent.RenderedMessage.Split('~');
string username = string.Empty;
if (data != null && data.Length >= 1)
username = data[0];
// Lookup the parameter
IDbDataParameter param = (IDbDataParameter)command.Parameters[ParameterName];
// Format the value
object formattedValue = username;
// If the value is null then convert to a DBNull
if (formattedValue == null)
{
formattedValue = DBNull.Value;
}
param.Value = formattedValue;
}
}
And to use this, I add it in the log4net config file like this:
<parameter type="MyAssembly.Logging.UserAdoNetAppenderParameter, MyAssembly">
<parameterName value="#username" />
<dbType value="String" />
<size value="255" />
<layout type="log4net.Layout.PatternLayout" value="%message" />
</parameter>
And by convention, my log statements will be something like this:
if (log.IsDebugEnabled)
log.DebugFormat("{0}~{1}~{2}", username, someOtherParameter, message);
If you look at the class, it uses data[0] as the username, so it is dependant on following the convention. It does, however, get the username into its own parameter and into a separate field in the log database table, without resorting to stuffing it temporarily into the unsafe ThreadContext.

Yes, thread agility means that you might not get the correct data back. For log4net, you will want to stick it in your HttpContext's Items collection.
The trouble is you have to do some work to get it back out when it's time to write those values to the database because of this I have always used Marek's Adaptive Property Provider class to do the grunt work for me. It's super simple to use it as all you have to do is the following:
log4net.ThreadContext.Properties["UserName"] = AdaptivePropertyProvider.Create("UserName", Thread.CurrentPrincipal.Identity.Name);
The adaptive property will know the appropriate place to retrieve the value when log4net requests it.
Alternative Option
If you're not stuck with log4net, NLog makes logging for ASP.NET websites way more simple because they natively support ASP.NET applications. Usage and even configuration is almost identical to log4net!

How to add remotingappender programmatically in log4net

When I have following xml configuration to configure log4net for remotingappender, it all works.
<log4net>
<appender name="RemotingAppender" type="log4net.Appender.RemotingAppender" >
<sink value="tcp://localhost:8085/LoggingSink" />
<lossy value="false" />
<bufferSize value="1" />
<onlyFixPartialEventData value="true" />
</appender>
<root>
<level value="ALL" />
<appender-ref ref="RemotingAppender" />
</root>
</log4net>
I want to do the same thing in code. I searched a littlebit and find an example like the following. But i could not get it working.
ILog log = log4net.LogManager.GetLogger("logName");
Repository.Hierarchy.Logger l = (Repository.Hierarchy.Logger)log.Logger;
// set level
l.Level = l.Hierarchy.LevelMap["ALL"];
// create appander
Appender.RemotingAppender remotingAppender = new Appender.RemotingAppender();
remotingAppender.Name = "custom";
remotingAppender.Sink = "tcp://localhost:8085/LoggingSink";
remotingAppender.Lossy = false;
remotingAppender.BufferSize = 1;
//remotingAppender.Fix = log4net.Core.FixFlags.All;
// create pattern
log4net.Layout.PatternLayout layout = new log4net.Layout.PatternLayout();
layout.ConversionPattern = "%d [%thread] %-5p %c [%a] - %m [%line] [%M]%n";
layout.ActivateOptions();
remotingAppender.Layout = layout;
remotingAppender.ActivateOptions();
// add appender
l.AddAppender(remotingAppender);
// perform logging (doesnt work)
log.Warn("my warning");
log.Error("my error");
What is the missing side in my code?

try one of log4net.Config.BasicConfigurator.Configure(...) oveloads.

I finally get it working. i firstly load and xml configuration which is almost emptpy. Then i put the pattern layout which has same pattern string as remoting listener. My modified xml file is below:
<log4net>
<root>
<level value="ALL" />
</root>
</log4net>
And i load this configuration at the beginning of my application (program.c)
log4net.Config.XmlConfigurator.Configure(file)

Using log4net to Log Powershell Output to Dynamic Database Variables

I'm building a web UI to help automate our deployment process and am going to write a powershell script to do the deployment and would like it's Write-Debug (or any statement to log, just let me know which to use :) ) statements to be logged to the deployed package's database variable Log. I haven't really used log4net before so please don't laugh if I'm doing this completely wrong.
I figure since the location is dynamic, I'd have to code the log4net appenders, but would it be easier/better to do all of the log4net stuff inside of the powershell script? I read this and found I should use ps.Streams.Debug.DataAdded += new EventHandler<DataAddedEventArgs>(delegate(object sender, DataAddedEventArgs e) to get the write-debug information.
Here is what I have so far:
public static void Test(Package pkg)
{
//Do roll_out
//Creates a cmd prompt
PowerShell ps = PowerShell.Create();
string myCommand = #"C:\Users\evan.layman\Desktop\test.ps1";
ps.AddCommand(myCommand);
ps.Streams.Debug.DataAdded += new EventHandler<DataAddedEventArgs>(delegate(object sender, DataAddedEventArgs e)
{
PSDataCollection<DebugRecord> debugStream = (PSDataCollection<DebugRecord>)sender;
DebugRecord record = debugStream[e.Index];
Hierarchy hierarchy = (Hierarchy)LogManager.GetRepository();
hierarchy.Root.RemoveAllAppenders(); /*Remove any other appenders*/
AdoNetAppender appender = new AdoNetAppender();
appender.ConnectionString = ConfigurationManager.ConnectionStrings["DeploymentConnectionString"].ConnectionString;
appender.CommandText = "with cte as (SELECT * FROM Package PackageID =" + pkg.PackageID + ") UPDATE cte SET (Log) VALUES (?logText)";
AdoNetAppenderParameter param = new AdoNetAppenderParameter();
param.DbType = System.Data.DbType.String;
param.ParameterName = "logText";
param.Layout = new log4net.Layout.RawTimeStampLayout();
appender.AddParameter(param);
BasicConfigurator.Configure(appender);
ILog log = LogManager.GetLogger("PowerShell");
log.Debug(record.Message);
//log.DebugFormat("{0}:{1}", DateTime.UtcNow, record);
//log.Warn(record, new Exception("Log failed"));
});
Collection<PSObject> commandResults = ps.Invoke();
Hopefully I can get this working :)

I would keep as much log4net config out of your code as possible. In your code, the config is being recreated on each debug statement, which is inefficient.
It's possible to do what you want using event context properties in log4net. I've blogged about log4net event context a bit on my blog.
Here's a quick example that's close to your existing codebase....
This C# code shows how to use log4net global properties to store custom event context data; note the setting of the "PackageID" global property value before the pipeline is executed...
using System;
using System.Management.Automation;
using log4net;
// load log4net configuration from app.config
[assembly:log4net.Config.XmlConfigurator]
namespace ConsoleApplication1
{
class Program
{
private static PowerShell _ps;
private static ILog Log = log4net.LogManager.GetLogger(typeof (Program));
static void Main(string[] args)
{
string script = "write-debug 'this is a debug string' -debug";
for (int packageId = 1; packageId <= 5; ++packageId)
{
using (_ps = PowerShell.Create())
{
_ps.Commands.AddScript(script);
_ps.Streams.Debug.DataAdded += WriteDebugLog;
// set the PackageID global log4net property
log4net.GlobalContext.Properties["PackageID"] = packageId;
// sync invoke your pipeline
_ps.Invoke();
// clear the PackageID global log4net property
log4net.GlobalContext.Properties["PackageID"] = null;
}
}
}
private static void WriteDebugLog(object sender, DataAddedEventArgs e)
{
// get the debug record and log the message
var record = _ps.Streams.Debug[e.Index];
Log.Debug(record.Message);
}
}
}
And here is the app.config that drops the logs into the database; note the custom PackageID parameter in the SQL, and how the value is pulled from the log4net property stack:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>
</configSections>
<log4net>
<appender name="Ado" type="log4net.Appender.AdoNetAppender">
<connectionType value="System.Data.SqlClient.SqlConnection, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<connectionString value="data source=vbox-xp-sql;initial catalog=test1;integrated security=false;persist security info=True;User ID=test1;Password=password" />
<commandText value="INSERT INTO Log ([Message],[PackageID]) VALUES (#message, #packageid)" />
<parameter>
<parameterName value="#message" />
<dbType value="String" />
<size value="4000" />
<layout type="log4net.Layout.PatternLayout" value="%message" />
</parameter>
<parameter>
<parameterName value="#packageid" />
<dbType value="Int32" />
<size value="4" />
<!-- use the current value of the PackageID property -->
<layout type="log4net.Layout.PatternLayout" value="%property{PackageID}" />
</parameter>
</appender>
<root>
<level value="ALL" />
<appender-ref ref="Ado" />
</root>
</log4net>
</configuration>
Hope this helps.

How to save a large nhibernate collection without causing OutOfMemoryException

How do I save a large collection with NHibernate which has elements that surpass the amount of memory allowed for the process?
I am trying to save a Video object with nhibernate which has a large number of Screenshots (see below for code). Each Screenshot contains a byte[], so after nhibernate tries to save 10,000 or so records at once, an OutOfMemoryException is thrown. Normally I would try to break up the save and flush the session after every 500 or so records, but in this case, I need to save the collection because it automatically saves the SortOrder and VideoId for me (without the Screenshot having to know that it was a part of a Video). What is the best approach given my situation? Is there a way to break up this save without forcing the Screenshot to have knowledge of its parent Video?
For your reference, here is the code from the simple sample I created:
public class Video
{
public long Id { get; set; }
public string Name { get; set; }
public Video()
{
Screenshots = new ArrayList();
}
public IList Screenshots { get; set; }
}
public class Screenshot
{
public long Id { get; set; }
public byte[] Data { get; set; }
}
And mappings:
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
assembly="SavingScreenshotsTrial"
namespace="SavingScreenshotsTrial"
default-access="property">
<class name="Screenshot"
lazy="false">
<id name="Id"
type="Int64">
<generator class="hilo"/>
</id>
<property name="Data" column="Data" type="BinaryBlob" length="2147483647" not-null="true" />
</class>
</hibernate-mapping>
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
assembly="SavingScreenshotsTrial"
namespace="SavingScreenshotsTrial" >
<class name="Video"
lazy="false"
table="Video"
discriminator-value="0"
abstract="true">
<id name="Id"
type="Int64"
access="property">
<generator class="hilo"/>
</id>
<property name="Name" />
<list name="Screenshots"
cascade="all-delete-orphan"
lazy="false">
<key column="VideoId" />
<index column="SortOrder" />
<one-to-many class="Screenshot" />
</list>
</class>
</hibernate-mapping>
When I try to save a Video with 10000 screenshots, it throws an OutOfMemoryException. Here is the code I'm using:
using (var session = CreateSession())
{
Video video = new Video();
for (int i = 0; i < 10000; i++)
{
video.Screenshots.Add(new Screenshot() {Data = camera.TakeScreenshot(resolution)});
}
session.SaveOrUpdate(video);
}

For this reason, we have typically had the child entity reference the parent rather than vice versa.

Create a custom type by using AbstractType and IParameterizedType these are in NHibernate.Type namespace. Use stateless session and provide batch size.

The chapter 13 of nhibernate documentation handle this issue.
"A naive approach to inserting 100 000 rows in the database using NHibernate might look like this:
using (ISession session = sessionFactory.OpenSession())
using (ITransaction tx = session.BeginTransaction())
{
for (int i = 0; i < 100000; i++)
{
Customer customer = new Customer(.....);
session.Save(customer);
}
tx.Commit();
}
This would fall over with an OutOfMemoryException somewhere around the 50 000th row[...]"
To resume... the solution is work with batch size and no second level cache by setting this properties:
Batch size:
adonet.batch_size 20
Second level cache:
cache.use_second_level_cache false
It should be enougth to solve OutOfMemoryException.
More datails at documentation reference: http://nhibernate.info/previous-doc/v5.0/single/nhibernate_reference.pdf