HBase AggregationClient in HDInsight - c#

I am trying out the C# REST client accessing HBase in HDInsight.
I would like to get a filtered row count without bringing results down to the client, and from what I've seen online, the AggregationClient coprocessor seems to be what I need. But I haven't been able to find any documentation as to whether it's available in HBase in HDInsight, and more specifically, available to the REST client.
Anyone have any experience with it, or suggestions?
Thanks.

Any processing on server that needs to be done, I typically end up mapping the HBase table to a hive table and leveraging the hive connection drivers or for it for C# or via the Hive Type Provider in F#...
Mapping HBase to Hive:
http://indiedevspot.com/2015/07/09/powering-azureml-with-hadoop-hbase/ (about 1/4 down)
Linq to Hive (C#):
http://hadoopsdk.codeplex.com/
Hive Type Provider F#:
http://fsprojects.github.io/FSharp.Data.HiveProvider/

Related

Accessing DataVerse data and running SQL statements against it from C# code on .NET 6 (Azure Function)

I'm working with Dynamics365 CE in the cloud. I'm trying to run some rather involved queries that I've built up as SQL scripts (using the wonderful "SQL-4-CDS" plugin for the XrmToolBox).
Now I know I can connect to the Dataverse data store through the TDS endpoint (if enabled - it is in my case), and from SSMS, it works just fine:
Server Name = myorg.crm4.dynamics.com,5558
Authentication = Azure Active Directory - Password
User Name = my company e-mail
I can connect to Dataverse, and run my queries - all is great.
Now I'd like to do the same from C# code (running on .NET 6) that I'm writing, that should end up being an Azure Function in the end - so it's a "server-to-server", behind-the-scenes, no interactive login context kind of scenario.
I can connect to Dataverse via the TDS endpoint using this connection string - as long as I'm running the app interactively - as me, in my user context:
Server=myorg.crm4.dynamics.com,5558;Authentication=Active Directory Password;Database=my_dbname;User Id=my_email;Password=my_pwd;
However - this won't work with a server-to-server "daemon"-style setup.
Since I'm using .NET 6 (for the Azure Function), and since I want to run some custom SQL statements, I cannot use the "CRM XRM Client" tooling (with the IOrganizationService classes) - I need to use straight ADO.NET - any idea would I could define an ADO.NET compatible connection string, that would use a Client ID and Client Secret (which I both have at my disposal)?
I've tried a great many values for the Authentication=...... setting - but none have worked so far. Any place I can find a complete list of the supported values for this connection string parameter?
Thanks for any help or pointers!

Informix connection string field descriptions and values to be used

I am writing a test application in .net using c# to connect to IBM's Informix database.
So far what i did is, i installed Informix client sdk v4.10 in my machine. After that i wrote a piece of code referring from here and here. In my code i have a reference to IBM.Data.Informix.dll which is referred from installed path of Informix client sdk's bin folder netf40.
When i run a test application, i am getting below error while trying to opening up an connection,
ERROR [HY000] [Informix .NET provider][Informix]System error occurred
in network function.
i assume this error is due to connection string field not been supplied properly, i referred https://www.connectionstrings.com/informix/ and tried using connection string like informix with ODBC driver and informix .net provider mentioned in above link but no use, i am also having difficulty in understanding from where to get values for each connection string fields like protocol, port, host-name , server-name and service name.
To find values of above fields, i tried looking for SQLHOSTS key in registry entries under HKEY_LOCAL_MACHINE\SOFTWARE\INFORMIX\ unfortunately it wasn't there! and also tried running setnet32.exe from client sdk's bin folder and i could see below screen with only protocol info!.
It would be really helpful if anyone can help me.
This is a very, very difficult question to answer blind. :-)
setnet32.exe will not know the information you are looking for, you need to provide this information to setnet32.exe.
The first question to ask is: is your database running on Unix or Linux? If it is, then by logging in to the database server as user "informix" and running the command
cat $INFORMIXDIR/etc/sqlhosts
If you're on Windows, then login to the Windows server and from a command prompt, run
TYPE %INFORMIXDIR%\etc\sqlhosts
This should give you a file with potentially a bunch of information, you're looking for lines that are not comments and have at least 4 columns. This is my sqlhosts file on a Docker I'm testing:
$ cat $INFORMIXDIR/etc/sqlhosts
############################################################
### DO NOT MODIFY THIS COMMENT SECTION
### HOST NAME = 7edf3045c382
############################################################
informix onsoctcp 7edf3045c382 9088
informix_dr drsoctcp 7edf3045c382 9089
The last two lines are the guts of the file.
Column 1 is the name of the INFORMIXSERVER or an alias (IBM Informix Server in setnet32.exe)
Column 2 is the protocol name (Protocolname in setnet32.exe)
Column 3 is the host name (HostName in setnet32.exe)
Column 4 is the port number or name (Service name in setnet32.exe)
If column 4 is a name and you're on Unix or Linux, then search for the port name in /etc/services on your Unix or Linux server. If you're on Windows, then it will be in %windir%\system32\drivers\etc\services (or similar).
Once you have that, you can then run the command
dbaccess
Choose the Database option, followed by the Select option. This should present you with a list of databases, roughly like:
SELECT DATABASE >>
Select a database with the Arrow Keys, or enter a name, then press Return.
------------------------------------------------ Press CTRL-W for Help --------
backbone#informix wallet#informix
cust#informix
retail#informix
sports#informix
sysadmin#informix
sysha#informix
sysmaster#informix
sysuser#informix
sysutils#informix
In general, databases called "sys" are reserved for Informix administration, and may not be actual databases, although you can query them with SELECTs, you probably won't be able to (and really shouldn't!!) INSERT, UPDATE or DELETE or use DDL.
In my database list above, all the sys* databases are Informix administration "databases". Database names are shown in my example in "databasename#informixservername" format.
You should now have all the information you need to access your database.

ODBC ISAM_EOF without any reason

I have a strange behavior with an ODBC Driver and the underlying COBOL database. The database driver is acuODBC of AcuCorp(now Microfocus), the database itself is a COBOL database.
The DSN is a system DSN and works just fine. Via Access/Excel u can read/write data with the DSN successfully.
In C# the ODBCConnection.ConnectionState is open. Retreiving the table headers works just fine. But when reading data, a strange error occurs. Here is a summerize of the trace log:
[Retreiving Column Headers]
ISAMRestrict - NO_ISAM_ERR
ISAMRewind - NO_ISAM_ERR
ISAMNextRecord – ISAM_EOF
This occurs when I add a WHERE clausel to the SELECT statement. When I do not add a WHERE clausel and just retreive the whole table, it takes incredible long (about 12 minutes for 40000 records), but at least I retreive data.
So my question would now be, has anyone else occured such a strange behavior with an ODBC driver? End of File where data should be?
Just as a side note, I have contacted Microfocus too, if they have a solution I will post it here.
It seems to be Windows UAC reliant. As our application run in compatibility mode, UAC visualization is active and causing may some problems. The reason for this is, that the COBOL databse is a file based database, and the client where are coding for uses these files in ODBC DSN config directly instead of running an ODBC server to handle the requests.
So the UAC leads to some strange behavior.

Advantage Database Error 5175

I'm trying to read data from an Advantage Database with Advantage .Net Data Provider.
When I execute my app, I have the following error :
"Error 5175 the index was created with a different collation sequence"
A way to fix it is to open the table with Advantage Data Architect and reindex or delete the index file (.adi) but I can't do this because the database is used by an accounting software (Sage BOB 50) and if I reindex or delete the index, the software is unable to access the data anymore.
Any ideas?
The collation your connection will use depends on the way you are connecting to the server. (See also Advantage Database Index Collation Sequence).
In order to avoid the 5175 error you should use the same connection method that the accounting software uses.
If it uses ALS you can copy the adslocal.cfg config file over to your application. If it uses ADS you should make sure that you connect to the same database server that the accounting software does.

Need help debugging: Having trouble getting data to Silverlight App through RIA Services, Entity Framework, MySQL

I'm trying to build a Silverlight App that accesses and presents data from a MySQL database. I'm trying to use Entity Framework to model the MySQL data and RIA Services to make the data via EF available to Silverlight.
My Silverlight App is showing the correct columns in the datagrid, but it does not show the data (alternate link to image) :
When I look at the DomainService file (used for RIA Services), I see this:
public IQueryable<saw_order> GetSaw_order(int intOrder)
{
return this.Context.saw_order
.Where(o => o.Wo == intOrder);
}
To test this step, I modified the LINQ to remove the where so that all I had was return this.Context.saw_order;. When I did this, I was able to check the MySQL server and verify that the query was in fact sent to the MySQL server and the MySQL server was "Writing to NET" and trying to send data back. The query sent from my test machine was valid.
From my test above, it seems that data is correctly being sent to the MySQL server but is lost somewhere on its return. My difficulty now is trying to figure out where in the chain (Entity Framework to RIA Services to Silverlight client) the data is getting lost and I'm not sure how to debug this at different points.
For example, what are other ways I might test Entity Framework to make sure EF is not the problem? How might I test RIA services? Should I test on the Silverlight Client?
I'm struggling with learning C# and am not sure what to do to test. How might I "catch" the return in the DomainService so I can do some basic debugging.
Any help is very much appreciated.
Change your code like this:
var qry = this.Context.saw_order.Where(o => o.Wo == intOrder);
return qry;
If you put a breakpoint in at the return, then you can try executing the query in the immediate window and see if it is executing correctly.
From my test above, it seems that data
is correctly being sent to the MySQL
server but is lost somewhere on its
return. My difficulty now is trying to
figure out where in the chain (Entity
Framework to RIA Services to
Silverlight client) the data is
getting lost and I'm not sure how to
debug this at different points.
I use tools like:
Linqpad: This is for testing my linq to sql statements. It is pretty straightforward and easy to use.
Fiddler: Fiddler will tell you what is going on between the server and the client.

Categories