We Keep Coding

Accessing data in hdfs in a remote machine

I am trying to access hadoop deployed in a remote machine uisng the WebHDFS API provided by Microsoft (Microsoft.Hadoop.WebHDFS). In order to display content of the HDFS file on a client machine I am using the following code:
using System.Threading.Tasks;
using System.Windows.Forms;
using Microsoft.Hadoop.WebHDFS;
using System.Windows.Forms;
public static List<string> listFilesName(string strHadoopUserName, string strDirectoryPath)
{
//uri to access the hadoop through the web
Uri uri = new Uri("http://192.168.0.165:50070/");
List<string> lstFilesName = new List<string>();
try
{
WebHDFSClient hdfsClient = new WebHDFSClient(uri, strHadoopUserName);
Microsoft.Hadoop.WebHDFS.DirectoryListing directroyStatus = hdfsClient.GetDirectoryStatus(strDirectoryPath).Result;
List<DirectoryEntry> lst = directroyStatus.Files.ToList();
foreach (DirectoryEntry var in lst)
{
lstFilesName.Add(var.PathSuffix);
MessageBox.Show(var.PathSuffix);
}
return lstFilesName;
}
catch(Exception exException)
{
MessageBox.Show(exException.Message);
return null;
}
This relevant configuration file content is displayed below:
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.0.165:9000</value>
</property>
</configuration>
hdfs-site .xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/hadoop/data/dfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/hadoop/data/dfs/datanode</value>
</property>
<property>
<name>dfs.permissions</name>
<value>true</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
Both machines are within the same network. When I run the above code I am getting the following error:
an aggregate exception has beeh caught/one or more error occured.
Any help will be appreciated.

NHibernate.Exceptions.GenericADOException : could not execute query

I have a legacy application (vfp 8) that I need to pull data from (no inserts). I am using the Accnum field as the primary key, it is defined in the table as character 11.
Factory configuration:
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-configuration xmlns="urn:nhibernate-configuration-2.2">
<reflection-optimizer use="false" />
<session-factory>
<property name="connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
<property name="dialect">NHibernate.Dialect.GenericDialect</property>
<property name="connection.driver_class">NHibernate.Driver.OleDbDriver</property>
<property name="connection.connection_string">Provider=VFPOLEDB.1;Data Source=C:\Analysis\Quantium\development\RD warehouse\_RDAUWH\Data;Collating Sequence=MACHINE</property>
<property name="show_sql">false</property>
</session-factory>
</hibernate-configuration>
This is my mapping file:
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
assembly="RDLabels"
namespace="RDLabels.Domain">
<class name="CustMast">
<id name="Accnum" column="Accnum" type="string">
<generator class="assigned"/>
</id>
<property name="Fullname" />
<property name="Add" />
<property name="State" />
</class>
</hibernate-mapping>
The class:
public class CustMast
{
private string _accnum;
public virtual string Accnum
{
get { return _accnum; }
set { _accnum = value; }
}
private string _fullname;
public virtual string Fullname
{
get { return _fullname; }
set { _fullname = value; }
}
private string _add;
public virtual string Add
{
get { return _add; }
set { _add = value; }
}
private string _state;
public virtual string State
{
get { return _state; }
set { _state = value; }
}
}
Here is the code that gets the record:
public CustMast GetByAccnum(String accnum)
{
using (ISession session = NHibernateHelper.OpenSession())
{
CustMast custMast = session
.CreateCriteria(typeof(CustMast))
.Add(Restrictions.Eq("Accnum", accnum))
.UniqueResult<CustMast>();
return custMast;
}
}
The full error is:
NHibernate.Exceptions.GenericADOException : could not execute query
[ SELECT this_.Accnum as Accnum0_0_, this_.Fullname as Fullname0_0_, this_.Add as Add0_0_, this_.State as State0_0_ FROM CustMast this_ WHERE this_.Accnum = ? ]
Name:cp0 - Value:00059337444
[SQL: SELECT this_.Accnum as Accnum0_0_, this_.Fullname as Fullname0_0_, this_.Add as Add0_0_, this_.State as State0_0_ FROM CustMast this_ WHERE this_.Accnum = ?]
----> System.IndexOutOfRangeException : Invalid index 0 for this OleDbParameterCollection with Count=0. - d:\CSharp\NH\NH\nhibernate\src\NHibernate\Loader\Loader.cs:1590
Running NHibernate Profiler it shows:
WARN:
reflection-optimizer property is ignored out of application configuration file.
WARN:
System.IndexOutOfRangeException: Invalid index 0 for this OleDbParameterCollection with Count=0.
at System.Data.OleDb.OleDbParameterCollection.RangeCheck(Int32 index)
at System.Data.OleDb.OleDbParameterCollection.GetParameter(Int32 index)
at System.Data.Common.DbParameterCollection.System.Collections.IList.get_Item(Int32 index)
at NHibernate.Driver.DriverBase.ExpandQueryParameters(IDbCommand cmd, SqlString sqlString) in d:\CSharp\NH\NH\nhibernate\src\NHibernate\Driver\DriverBase.cs:line 235
at NHibernate.AdoNet.AbstractBatcher.ExpandQueryParameters(IDbCommand cmd, SqlString sqlString) in d:\CSharp\NH\NH\nhibernate\src\NHibernate\AdoNet\AbstractBatcher.cs:line 232
at NHibernate.Loader.Loader.PrepareQueryCommand(QueryParameters queryParameters, Boolean scroll, ISessionImplementor session) in d:\CSharp\NH\NH\nhibernate\src\NHibernate\Loader\Loader.cs:line 1152
ERROR:
Invalid index 0 for this OleDbParameterCollection with Count=0.

I was struggling with my linq queries throwing the same error whenever i passed them a parameter. If i didn't pass any parameters and did a session.Query() they would work fine.
I struggled with this for days but i found this Nhibernate jira ticket here . It explains an apparent problem with SQLParameters and the Iseries Db2 provider.
I understand you're using a different provider but you might benefit from just downloading the latest Nhibernate core source code, building it, and referencing the latest version in your project. It fixed my issue.

First thing I would try is to run this in SQL and see what happens as it might be a data issue.
SELECT this_.Accnum as Accnum0_0_, this_.Fullname as Fullname0_0_,
this_.Add as Add0_0_, this_.State as State0_0_ FROM CustMast this_
WHERE this_.Accnum = '00059337444'
Is Accnum column defined as a string type (varchar, nvarchar etc) in your database?
Edit OK the next step is to actually confirm the SQL being sent to FoxPro. You will need to set up logging (or download a trial copy of NHProf) to find out if the SQL is correct. Your setup and code looks correct however I am not 100% sure of the choice of dialect as this may be causing you problems.
I take it you have seen this and this.
Edit2 The NHProf error seems to me that it thinks your Id should be a Int32 as it looks like it is calling at System.Data.OleDb.OleDbParameterCollection.GetParameter(Int32 index).
I think you need to add this to your mappings:-
<id name="Accnum" column="Accnum" type="string" >
Note the additional type="string"

If you use ODP.Net then you need an oracle client installed and configured, on your target machine, as well as your development machine.
The error message you post might be caused by having more than one Oracle client installed, or possibly trying to use something like a later version of odp.net with an earlier client version.

FetchExpression results appear to be cached, how do I prevent this?

I am using a FetchExpression against a RetrieveMultiple operation on a CrmOrganizationServiceContext within a windows service to fetch and process items out of a queue.
The first time this runs this fetches the items to be processed correctly. On subsequent calls using the same CrmOrganizationServiceContext instance it always retrieves zero entities with no errors thrown. I have added in new entities and reactivated existing ones that should be fetched using the FetchXml and they aren't retrieved.
As soon as I restart my service it creates a new instance of the CrmOrganizationServiceContext and fetches the new items.
What am I doing wrong here?
public CrmConnector(string connectionString)
{
Context = new CrmOrganizationServiceContext(CrmConnection.Parse(connectionString));
}
public void FetchStuff()
{
string fetchXml = "...";
FetchExpression fetchExpression = new FetchExpression(fetchXml);
EntityCollection entityCollection = Context.RetrieveMultiple(fetchExpression);
// entityCollection.Entities is always empty following first run
}
private CrmOrganizationServiceContext Context { get; set; }
Fetch Xml as requested, the only customisation is the count attribute which limits the number of items being returned (as this is a queue processor)
<fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false" count="10">
<entity name="xxx1">
<attribute name="xxx_name" />
<attribute name="createdon" />
<attribute name="xxx_1" />
<attribute name="xxx_2" />
<attribute name="xxx_3" />
<attribute name="xxx_4" />
<attribute name="statecode" />
<order attribute="createdon" descending="false" />
<filter type="and">
<condition attribute="xxx_exported" value="0" operator="eq"/>
</filter>
</entity>
</fetch>

It is the CrmOrganizationServiceContext that is doing the caching - I found the following worked a treat and the results of my RetrieveMultiple are no longer cached :)
Context = new CrmOrganizationServiceContext(CrmConnection.Parse(connectionString));
Context.TryAccessCache(cache => cache.Mode = OrganizationServiceCacheMode.Disabled);

How to save a large nhibernate collection without causing OutOfMemoryException

How do I save a large collection with NHibernate which has elements that surpass the amount of memory allowed for the process?
I am trying to save a Video object with nhibernate which has a large number of Screenshots (see below for code). Each Screenshot contains a byte[], so after nhibernate tries to save 10,000 or so records at once, an OutOfMemoryException is thrown. Normally I would try to break up the save and flush the session after every 500 or so records, but in this case, I need to save the collection because it automatically saves the SortOrder and VideoId for me (without the Screenshot having to know that it was a part of a Video). What is the best approach given my situation? Is there a way to break up this save without forcing the Screenshot to have knowledge of its parent Video?
For your reference, here is the code from the simple sample I created:
public class Video
{
public long Id { get; set; }
public string Name { get; set; }
public Video()
{
Screenshots = new ArrayList();
}
public IList Screenshots { get; set; }
}
public class Screenshot
{
public long Id { get; set; }
public byte[] Data { get; set; }
}
And mappings:
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
assembly="SavingScreenshotsTrial"
namespace="SavingScreenshotsTrial"
default-access="property">
<class name="Screenshot"
lazy="false">
<id name="Id"
type="Int64">
<generator class="hilo"/>
</id>
<property name="Data" column="Data" type="BinaryBlob" length="2147483647" not-null="true" />
</class>
</hibernate-mapping>
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
assembly="SavingScreenshotsTrial"
namespace="SavingScreenshotsTrial" >
<class name="Video"
lazy="false"
table="Video"
discriminator-value="0"
abstract="true">
<id name="Id"
type="Int64"
access="property">
<generator class="hilo"/>
</id>
<property name="Name" />
<list name="Screenshots"
cascade="all-delete-orphan"
lazy="false">
<key column="VideoId" />
<index column="SortOrder" />
<one-to-many class="Screenshot" />
</list>
</class>
</hibernate-mapping>
When I try to save a Video with 10000 screenshots, it throws an OutOfMemoryException. Here is the code I'm using:
using (var session = CreateSession())
{
Video video = new Video();
for (int i = 0; i < 10000; i++)
{
video.Screenshots.Add(new Screenshot() {Data = camera.TakeScreenshot(resolution)});
}
session.SaveOrUpdate(video);
}

For this reason, we have typically had the child entity reference the parent rather than vice versa.

Create a custom type by using AbstractType and IParameterizedType these are in NHibernate.Type namespace. Use stateless session and provide batch size.

The chapter 13 of nhibernate documentation handle this issue.
"A naive approach to inserting 100 000 rows in the database using NHibernate might look like this:
using (ISession session = sessionFactory.OpenSession())
using (ITransaction tx = session.BeginTransaction())
{
for (int i = 0; i < 100000; i++)
{
Customer customer = new Customer(.....);
session.Save(customer);
}
tx.Commit();
}
This would fall over with an OutOfMemoryException somewhere around the 50 000th row[...]"
To resume... the solution is work with batch size and no second level cache by setting this properties:
Batch size:
adonet.batch_size 20
Second level cache:
cache.use_second_level_cache false
It should be enougth to solve OutOfMemoryException.
More datails at documentation reference: http://nhibernate.info/previous-doc/v5.0/single/nhibernate_reference.pdf

How to create an XmlMappingSource during runtime?

(Follow-Up-Question to How to change LINQ O/R-M table name/source during runtime?)
I need to change the table source of a LINQ 2 SQL O/R-Mapper table during runtime. To achieve this, I need to create an XmlMappingSource. On command line, I could use SqlMetal to create this mapping file, but I would like to create the mapping file during runtime in memory.
The XmlMappingSource is a simple xml file, looking something like this:
<?xml version="1.0" encoding="utf-8"?>
<Database Name="MyDatabase" xmlns="http://schemas.microsoft.com/linqtosql/mapping/2007">
<Table Name="dbo.MyFirstTable" Member="MyFirstTable">
<Type Name="MyFirstTable">
<Column Name="ID" Member="ID" Storage="_ID" DbType="UniqueIdentifier NOT NULL" IsPrimaryKey="true" IsDbGenerated="true" AutoSync="OnInsert" />
<Association Name="WaStaArtArtikel_WaVerPreisanfragen" Member="WaStaArtArtikel" Storage="_WaStaArtArtikel" ThisKey="ArtikelID" OtherKey="ID" IsForeignKey="true" />
</Type>
</Table>
<Table Name="dbo.MySecondTable" Member="MySecondTable">
<Type Name="MySecondTable">
<Column Name="ID" Member="ID" Storage="_ID" DbType="UniqueIdentifier NOT NULL" IsPrimaryKey="true" IsDbGenerated="true" AutoSync="OnInsert" />
<Column Name="FirstTableID" Member="FirstTableID" Storage="_FirstTableID" DbType="UniqueIdentifier NOT NULL" />
<Association Name="MySecondTable_MyFirstTable" Member="MyFirstTable" Storage="_MyFirstTable" ThisKey="FirstTableID" OtherKey="ID" IsForeignKey="true" />
</Type>
</Table>
</Database>
This should be possible to create using reflection, for example I can get the database name from a data context like this:
using System.Data.Linq.Mapping;
using System.Xml.Linq;
XDocument mapWriter = new XDocument();
DatabaseAttribute[] catx = (DatabaseAttribute[])typeof(WcfInterface.WaDataClassesDataContext).GetCustomAttributes(typeof(DatabaseAttribute), false);
XElement xDatabase = new XElement("Database");
xDatabase.Add(new XAttribute("Name", catx[0].Name));
mapWriter.Add(xDatabase);
My problem: I can't find good documentation of the mapping, so extracting the necessary information is quite error-prone - maybe someone can point me to good docs of the mapping, or, even better, to a code example how to create the mapping file?

Have you considered using LINQ to Entities, the mapping formats for LINQ to Entities are documented.

Use Damien Guard's Open Source T4 templates. They do everything SQLMetal can do and more, and you'll have the full T4 engine behind you.

I just had same problem, also I had no option to change project as its too late to do this.
I needed to update database name in mapping file in my solution.
This solution works.
My database mapping
<?xml version="1.0" encoding="utf-8"?>
<Database Name="DatabaseName" xmlns="http://schemas.microsoft.com/linqtosql/mapping/2007">
<Table Name="dbo.tblDictionary" Member="TblDictionary">
<Type Name="TblDictionary">
<Column Name="lngRecordID" Member="LngRecordID" Storage="_LngRecordID" DbType="Int NOT NULL IDENTITY" IsPrimaryKey="true" IsDbGenerated="true" AutoSync="OnInsert" />
<Column Name="txtWord" Member="TxtWord" Storage="_TxtWord" DbType="VarChar(50) NOT NULL" CanBeNull="false" />
</Type>
</Table>
</Database>
and finally the code:
class Program
{
static void Main(string[] args)
{
// to get embeded file name you have to add namespace of the application
const string embeddedFilename = "ConsoleApplication3.FrostOrangeMappings.xml";
// load file into stream
var embeddedStream = GetEmbeddedFile(embeddedFilename);
// process stream
ProcessStreamToXmlMappingSource(embeddedStream);
Console.ReadKey();
}
private static void ProcessStreamToXmlMappingSource(Stream stream)
{
const string newDatabaseName = "pavsDatabaseName";
var mappingFile = new XmlDocument();
mappingFile.Load(stream);
stream.Close();
// populate collection of attribues
XmlAttributeCollection collection = mappingFile.DocumentElement.Attributes;
var attribute = collection["Name"];
if(attribute==null)
{
throw new Exception("Failed to find Name attribute in xml definition");
}
// set new database name definition
collection["Name"].Value = newDatabaseName;
//display xml to user
var stringWriter = new StringWriter();
using (var xmlTextWriter = XmlWriter.Create(stringWriter))
{
mappingFile.WriteTo(xmlTextWriter);
xmlTextWriter.Flush();
stringWriter.GetStringBuilder();
}
Console.WriteLine(stringWriter.ToString());
}
/// <summary>
/// Loads file into stream
/// </summary>
/// <param name="fileName"></param>
/// <returns></returns>
private static Stream GetEmbeddedFile(string fileName)
{
var assembly = Assembly.GetExecutingAssembly();
var stream = assembly.GetManifestResourceStream(fileName);
if (stream == null)
throw new Exception("Could not locate embedded resource '" + fileName + "' in assembly");
return stream;
}`