We Keep Coding

Fetching more than 1000 rows from Domino LDAP server using .NET Core 5 and Novell.Directory.Ldap.NETStandard

I want to fetch all the users from a large location of our Domino LDAP, around ~2000 users altogether. Since .NET Core sadly doesn't have a platform independent LDAP library, I'm using Novell.Directory.Ldap.NETStandard with this POC:
var cn = new Novell.Directory.Ldap.LdapConnection();
cn.Connect("dc.internal", 389);
cn.Bind("user", "pw");
string filter = "location=MyLoc";
var result = cn.Search("", Novell.Directory.Ldap.LdapConnection.ScopeOne, filter, new string[] { Novell.Directory.Ldap.LdapConnection.AllUserAttrs }, typesOnly: false);
int count = 0;
while (result.HasMore()) {
var entry = result.Next();
count++;
Console.WriteLine(entry.Dn);
}
It prints me a lot of entries, but not all. When count = 1000 I got an Size Limit Exceeded exception. I guess this is because I need to use some kind of pagination, so not all entries woult be returned in a single request. There are different questions like this or this one. Both in Java, the .NET Core API seems somehow different.
Approach 1: Try to find out how LdapSearchRequest works in .NET Core
byte[] resumeCookie = null;
LdapMessageQueue queue = null;
var searchReq = new LdapSearchRequest("", LdapConnection.ScopeOne, filter, new string[] { LdapConnection.AllUserAttrs },
LdapSearchConstraints.DerefNever, maxResults: 3000, serverTimeLimit: 0, typesOnly: false, new LdapControl[] { new SimplePagedResultsControl(size: 100, resumeCookie) });
var searchRequest = cn.SendRequest(searchReq, queue);
I'm trying to figure out how the Java examples can be used in .NET Core. This looks good, however I can't figure out how to fetch the LDAP entries. I only get an message id. By looking into the source it seems that I'm on the right way, but they're using MessageAgent which cannot be used outside since it's internal sealed. This is propably the reason why searching for LdapRearchRequest in the source code doesn't give many results.
Approach 2: Using SimplePagedResultsControlHandler
var opts = new SearchOptions("", LdapConnection.ScopeOne, filter, new string[] { LdapConnection.AllUserAttrs });
// For testing purpose: https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard/issues/163
cn.SearchConstraints.ReferralFollowing = false;
var pageControlHandler = new SimplePagedResultsControlHandler(cn);
var rows = pageControlHandler.SearchWithSimplePaging(opts, pageSize: 100);
This throws a Unavaliable Cricital Extension exception. First I thought that this is an issue of the .NET port, which may doesn't support all the features of the original Java library yet. It seems complete and according to further researches, it looks like to be an LDAP error code. So this must be something which has to be supported by the server, but is not supported by Domino.

I couldn't make at least one of those approachs work, but found another way: Cross platform support for the System.DirectoryServices.Protocols namespace was was added in .NET 5. This was missing for a long time in .NET Core and I guess this is the main reason why libraries like Novell.Directory.Ldap.NETStandard were ported to .NET Core - in times of .NET Core 1.x this was the only way I found to authenticate against LDAP wich works on Linux too.
After having a deeper look into System.DirectoryServices.Protocols, it works well out of the box, even for ~2k users. My basic POC class looks like this:
public class DominoLdapManager {
LdapConnection cn = null;
public DominoLdapManager(string ldapHost, int ldapPort, string ldapBindUser, string ldapBindPassword) {
var server = new LdapDirectoryIdentifier(ldapHost, ldapPort);
var credentials = new NetworkCredential(ldapBindUser, ldapBindPassword);
cn = new LdapConnection(server);
cn.AuthType = AuthType.Basic;
cn.Bind(credentials);
}
public IEnumerable<DominoUser> Search(string filter, string searchBase = "") {
string[] attributes = { "cn", "mail", "companyname", "location" };
var req = new SearchRequest(searchBase, filter, SearchScope.Subtree, attributes);
var resp = (SearchResponse)cn.SendRequest(req);
foreach (SearchResultEntry entry in resp.Entries) {
var user = new DominoUser() {
Name = GetStringAttribute(entry, "cn"),
Mail = GetStringAttribute(entry, "mail"),
Company = GetStringAttribute(entry, "companyname"),
Location = GetStringAttribute(entry, "location")
};
yield return user;
}
yield break;
}
string GetStringAttribute(SearchResultEntry entry, string key) {
if (!entry.Attributes.Contains(key)) {
return string.Empty;
}
string[] rawVal = (string[])entry.Attributes[key].GetValues(typeof(string));
return rawVal[0];
}
}
Example usage:
var ldapManager = new DominoLdapManager("ldap.host", 389, "binduser", "pw");
var users = ldapManager.Search("objectClass=person");
But it's not solved with Novell.Directory.Ldap.NETStandard as the title said
This doesn't solve my problem with the Novell.Directory.Ldap.NETStandard library as the title suggested, yes. But since System.DirectoryServices.Protocols is a official .NET package maintained by Microsoft and the .NET foundation, this seems the better aproach for me. The foundation will take care to keep it maintained and compatible with further .NET releases. When I wrote the question, I was not aware of the fact that Linux support is added now.
Don't get me wrong, I don't want to say that third packages are bad by design - that would be completely wrong. However, when I have the choice between a official package and a third party one, I think it makes sense to prefer the official one. Except there would be a good reason against that - which is not the case here: The official package (which doesn't exist in the past) works better to solve this issue than the third party one.

How to create new AutoML DataSet for simple classification (C#)

As part of ML automation process I want to dynamically create new AutoML model. I'm using C# (.net framework) and Google.Cloud.AutoML.V1.
After trying to run CreateDataSet code:
var autoMlClient = AutoMlClient.Create();
var parent = LocationName.FromProjectLocation(_projectId, _locationId);
var dataset = new Google.Cloud.AutoML.V1.Dataset();
dataset.DisplayName = "NewDataSet";
var response = autoMlClient.CreateDataset(parent, dataset);
I get the following error:
Field: dataset.dataset_metadata; Message: Required field not set
According to this user manual I should set Dataset Metadata Type, but the list contains only specific types of classifications (Translation/ImageClassifications etc.), I can't find a simple classification type.
How do I create a simple classification data set with the API ? in the AutoML UI its just with a simple button click ("NEW DATASET") - and have to provide only name & region - no classification type.
I also tried to set:
dataset.TextClassificationDatasetMetadata =
new TextClassificationDatasetMetadata() { ClassificationType = ClassificationType.Multiclass };
But I was unable to import data to it (got too many errors of invalid inputs from the input CSV file), I guess its related to the reason that the input format is not suitable for Text Classification.
UPDATE
I've just notice that the Nuget works with AutoML v1 but v1 beta does contains TablesDatasetMetadata Dataset Metadata Type for normal classifications. I'm speechless.

I also experienced this scenario today while creating a dataset using the NodeJS client. Since the Google AutoML table service is in the beta level you need to use the beta version of the AutoML client. In the Google cloud documentation they have used the beta client to create a dataset.
In NodeJS importing the beta version require('#google-cloud/automl').v1beta1.AutoMlClient instead of importing the normal version (v1) require('#google-cloud/automl').v1 worked for me to successfully execute the create dataset functionality.
In C# you can achieve the same through a POST request. Hope this helps :)

After #RajithaWarusavitarana comment, and my last question update , below is the code that did the trick. The token is being generated by GoogleClientAPI nuget and AutoML is handled by REST.
string GcpGlobalEndPointUrl = "https://automl.googleapis.com";
string GcpGlobalLocation = "us-central1"; // api "parent" parameter
public string GetToken(string jsonFilePath)
{
var serviceAccountCredentialFileContents = System.IO.File.ReadAllText(jsonFilePath);
var credentialParameters = NewtonsoftJsonSerializer.Instance.Deserialize<JsonCredentialParameters>(serviceAccountCredentialFileContents);
var initializer = new ServiceAccountCredential.Initializer(credentialParameters.ClientEmail)
{
Scopes = new List<string> { "https://www.googleapis.com/auth/cloud-platform" }
};
var cred = new ServiceAccountCredential(initializer.FromPrivateKey(credentialParameters.PrivateKey));
string accessToken = cred.GetAccessTokenForRequestAsync("https://oauth2.googleapis.com/token").Result;
return accessToken;
}
public void GetDataSetList(string projectId, string token)
{
var restClient = new RestClient(GcpGlobalEndPointUrl);
var createDataSetReqUrl = $"v1beta1/projects/{projectId}/locations/{GcpGlobalLocation}/datasets";
var createDataSetReq = new RestRequest(createDataSetReqUrl, Method.GET);
createDataSetReq.AddHeader("Authorization", $"Bearer {token}");
var createDatasetResponse = restClient.Execute(createDataSetReq);
createDatasetResponse.Dump();
}
I took the token generation code from google-api-dotnet-client Test File

Retrieving contents from a branch using octokit

I have recently started to use the octokit library to retrieve data from a github repository.
I can form a search request like
var request = new SearchCodeRequest("ValueSets", "xyzconnect", "projectA")
{
// we can restrict search to the file, path or search both
In = new[] { CodeInQualifier.Path },
};
var result = await client.Search.SearchCode(request);
or retrieve directly using
var xsx1 = await client.Repository.Content.GetAllContents(repository.Id, "ValueSets");
This works fine when I am using the default branch (usually master) but how do I perform the same functions against other branches?
EDIT
So to get content from branches you use the API as such
var xsx2 = await client.Repository.Content.GetAllContentsByRef(repository.Id, "ValueSets","develop");

How to call google.apis.dialogflow.v2 in C#

I am new to Google APIs. I want to know how to call Google Dialogflow API in C# to get intent form the input text. But I can't find any example to call Dialogflow using C#.
Please provide some example to call Dialogflow from C#.

If I understand your question correctly you want to call the DialogFlow API from within a C# application (rather than writing fulfillment endpoint(s) that are called from DialogFlow. If that's the case here's a sample for making that call:
using Google.Cloud.Dialogflow.V2;
...
...
var query = new QueryInput
{
Text = new TextInput
{
Text = "Something you want to ask a DF agent",
LanguageCode = "en-us"
}
};
var sessionId = "SomeUniqueId";
var agent = "MyAgentName";
var creds = GoogleCredential.FromJson("{ json google credentials file)");
var channel = new Grpc.Core.Channel(SessionsClient.DefaultEndpoint.Host,
creds.ToChannelCredentials());
var client = SessionsClient.Create(channel);
var dialogFlow = client.DetectIntent(
new SessionName(agent, sessionId),
query
);
channel.ShutdownAsync();
In an earlier version of the DialogFlowAPI I was running into file locking issues when trying to re-deploy a web api project which the channel.ShutDownAsync() seemed to solve. I think this has been fixed in a recent release.
This is the simplest version of a DF request I've used. There is a more complicated version that passes in an input context in this post:
Making DialogFlow v2 DetectIntent Calls w/ C# (including input context)

(Nitpicking: I assume you know DialogFlow will call your code as specified/registered in the action at DialogFlow? So your code can only respond to DialogFlow, and not call it.)
Short answer/redirect:
Don't use Google.Apis.Dialogflow.v2 (with GoogleCloudDialogflowV2WebhookRequest and GoogleCloudDialogflowV2WebhookResponse) but use Google.Cloud.Dialogflow.v2 (with WebhookRequest and WebhookResponse) - see this eTag-error. I will also mention some other alternatives underneath.
Google.Cloud.Dialogflow.v2
Using Google.Cloud.Dialogflow.v2 NuGet (Edit: FWIW: this code was written for the beta-preview):
[HttpPost]
public dynamic PostWithCloudResponse([FromBody] WebhookRequest dialogflowRequest)
{
var intentName = dialogflowRequest.QueryResult.Intent.DisplayName;
var actualQuestion = dialogflowRequest.QueryResult.QueryText;
var testAnswer = $"Dialogflow Request for intent '{intentName}' and question '{actualQuestion}'";
var dialogflowResponse = new WebhookResponse
{
FulfillmentText = testAnswer,
FulfillmentMessages =
{ new Intent.Types.Message
{ SimpleResponses = new Intent.Types.Message.Types.SimpleResponses
{ SimpleResponses_ =
{ new Intent.Types.Message.Types.SimpleResponse
{
DisplayText = testAnswer,
TextToSpeech = testAnswer,
//Ssml = $"<speak>{testAnswer}</speak>"
}
}
}
}
}
};
var jsonResponse = dialogflowResponse.ToString();
return new ContentResult { Content = jsonResponse, ContentType = "application/json" }; ;
}
Edit: It turns out that the model binding may not bind all properties from the 'ProtoBuf-json' correctly (e.g. WebhookRequest.outputContexts[N].parameters),
so one should probably use the Google.Protobuf.JsonParser (e.g. see this documentation).
This parser may trip over unknown fields, so one probably also wants to ignore that. So now I use this code (I may one day make the generic method more generic and thus useful, by making HttpContext.Request.InputStream a parameter):
public ActionResult PostWithCloudResponse()
{
var dialogflowRequest = ParseProtobufRequest<WebhookRequest>();
...
var jsonResponse = dialogflowResponse.ToString();
return new ContentResult { Content = jsonResponse, ContentType = "application/json" }; ;
}
private T ParseProtobufRequest<T>() where T : Google.Protobuf.IMessage, new()
{
// parse ProtoBuf (not 'normal' json) with unknown fields, else it may not bind ProtoBuf correctly
// https://github.com/googleapis/google-cloud-dotnet/issues/2425 "ask the Protobuf code to parse the result"
string requestBody;
using (var reader = new StreamReader(HttpContext.Request.InputStream))
{
requestBody = reader.ReadToEnd();
}
var parser = new Google.Protobuf.JsonParser(JsonParser.Settings.Default.WithIgnoreUnknownFields(true));
var typedRequest = parser.Parse<T>(requestBody);
return typedRequest;
}
BTW: This 'ProtoBuf-json' is also the reason to use WebhookResponse.ToString() which in turn uses Google.Protobuf.JsonFormatter.ToDiagnosticString.
Microsoft's BotBuilder
Microsoft's BotBuilder packages and Visual Studio template.
I havent't used it yet, but expect approximately the same code?
Hand written proprietary code
A simple example of incoming request code (called an NLU-Response by Google) is provided by Madoka Chiyoda (Chomado) at Github. The incoming call is simply parsed to her DialogFlowResponseModel:
public static async Task<HttpResponseMessage> Run([...]HttpRequestMessage req, [...]CloudBlockBlob mp3Out, TraceWriter log)
...
var data = await req.Content.ReadAsAsync<Models.DialogFlowResponseModel>();
Gactions
If you plan to work without DialogFlow later on, please note that the interface for Gactions differs significantly from the interface with DialogFlow.
The json-parameters and return-values have some overlap, but nothing gaining you any programming time (probably loosing some time by starting 'over').
However, starting with DialogFlow may gain you some quick dialog-experience (e.g. question & answer design/prototyping).
And the DialogFlow-API does have a NuGet package, where the Gactions-interface does not have a NuGet-package just yet.

How to create indexes in MongoDB via .NET

I've programmatically created a new document collection using the MongoDB C# driver.
At this point I want to create and build indexes programmatically. How can I do that?

Starting from v2.0 of the driver there's a new async-only API. The old API should no longer be used as it's a blocking facade over the new API and is deprecated.
The currently recommended way to create an index is by calling and awaiting CreateOneAsync with an IndexKeysDefinition you get by using Builders.IndexKeys:
static async Task CreateIndexAsync()
{
var client = new MongoClient();
var database = client.GetDatabase("HamsterSchool");
var collection = database.GetCollection<Hamster>("Hamsters");
var indexKeysDefinition = Builders<Hamster>.IndexKeys.Ascending(hamster => hamster.Name);
await collection.Indexes.CreateOneAsync(new CreateIndexModel<Hamster>(indexKeysDefinition));
}

you should use CreateIndex as EnsureIndex is marked obsolete for future compatibility with the next versions of MongoDB:
var client = new MongoClient("mongodb://localhost");
var db = client.GetServer().GetDatabase("db");
var collection = db.GetCollection<Hamster>("Hamsters");
collection.CreateIndex(IndexKeys<Hamster>.Ascending(_ => _.Name));

The overload of CreateOneAsync in the currently accepted answer is now marked as obsolete with the message "Use CreateOneAsync with a CreateIndexModel instead." Here's how you do it:
static async Task CreateIndex(string connectionString)
{
var client = new MongoClient(connectionString);
var database = client.GetDatabase("HamsterSchool");
var collection = database.GetCollection<Hamster>("Hamsters");
var indexOptions = new CreateIndexOptions();
var indexKeys = Builders<Hamster>.IndexKeys.Ascending(hamster => hamster.Name);
var indexModel = new CreateIndexModel<Hamster>(indexKeys, indexOptions);
await collection.Indexes.CreateOneAsync(indexModel);
}

Something like this should do:
var server = MongoServer.Create("mongodb://localhost");
var db = server.GetDatabase("myapp");
var users = db.GetCollection<User>("users");
users.EnsureIndex(new IndexKeysBuilder().Ascending("EmailAddress"));
Please see the following bits in the documentation:
http://api.mongodb.org/csharp/current/html/06bcd201-8844-3df5-d170-15f2b423675c.htm

There is an entire area on Indexing under the Definitions and Builders documentation page:
http://mongodb.github.io/mongo-csharp-driver/2.4/reference/driver/definitions/#index-keys
Example:
IndexKeysDefinition<MyModel> keys = "{ Reference: 1 }";
var indexModel = new CreateIndexModel<MyModel>(keys);
await _context.Indexes.CreateOneAsync(indexModel);

the easiest way to create indexes in c# is by using the driver wrapper library MongoDB.Entities. here's an example of creating a text index:
DB.Index<Author>()
.Key(a => a.Name, Type.Text)
.Key(a => a.Surname, Type.Text)
.Create();
and to do a full-text search, you simply do:
DB.SearchText<Author>("search term");
haven't seen anything else that makes it simpler than that.