We Keep Coding

Fetching more than 1000 rows from Domino LDAP server using .NET Core 5 and Novell.Directory.Ldap.NETStandard

I want to fetch all the users from a large location of our Domino LDAP, around ~2000 users altogether. Since .NET Core sadly doesn't have a platform independent LDAP library, I'm using Novell.Directory.Ldap.NETStandard with this POC:
var cn = new Novell.Directory.Ldap.LdapConnection();
cn.Connect("dc.internal", 389);
cn.Bind("user", "pw");
string filter = "location=MyLoc";
var result = cn.Search("", Novell.Directory.Ldap.LdapConnection.ScopeOne, filter, new string[] { Novell.Directory.Ldap.LdapConnection.AllUserAttrs }, typesOnly: false);
int count = 0;
while (result.HasMore()) {
var entry = result.Next();
count++;
Console.WriteLine(entry.Dn);
}
It prints me a lot of entries, but not all. When count = 1000 I got an Size Limit Exceeded exception. I guess this is because I need to use some kind of pagination, so not all entries woult be returned in a single request. There are different questions like this or this one. Both in Java, the .NET Core API seems somehow different.
Approach 1: Try to find out how LdapSearchRequest works in .NET Core
byte[] resumeCookie = null;
LdapMessageQueue queue = null;
var searchReq = new LdapSearchRequest("", LdapConnection.ScopeOne, filter, new string[] { LdapConnection.AllUserAttrs },
LdapSearchConstraints.DerefNever, maxResults: 3000, serverTimeLimit: 0, typesOnly: false, new LdapControl[] { new SimplePagedResultsControl(size: 100, resumeCookie) });
var searchRequest = cn.SendRequest(searchReq, queue);
I'm trying to figure out how the Java examples can be used in .NET Core. This looks good, however I can't figure out how to fetch the LDAP entries. I only get an message id. By looking into the source it seems that I'm on the right way, but they're using MessageAgent which cannot be used outside since it's internal sealed. This is propably the reason why searching for LdapRearchRequest in the source code doesn't give many results.
Approach 2: Using SimplePagedResultsControlHandler
var opts = new SearchOptions("", LdapConnection.ScopeOne, filter, new string[] { LdapConnection.AllUserAttrs });
// For testing purpose: https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard/issues/163
cn.SearchConstraints.ReferralFollowing = false;
var pageControlHandler = new SimplePagedResultsControlHandler(cn);
var rows = pageControlHandler.SearchWithSimplePaging(opts, pageSize: 100);
This throws a Unavaliable Cricital Extension exception. First I thought that this is an issue of the .NET port, which may doesn't support all the features of the original Java library yet. It seems complete and according to further researches, it looks like to be an LDAP error code. So this must be something which has to be supported by the server, but is not supported by Domino.

I couldn't make at least one of those approachs work, but found another way: Cross platform support for the System.DirectoryServices.Protocols namespace was was added in .NET 5. This was missing for a long time in .NET Core and I guess this is the main reason why libraries like Novell.Directory.Ldap.NETStandard were ported to .NET Core - in times of .NET Core 1.x this was the only way I found to authenticate against LDAP wich works on Linux too.
After having a deeper look into System.DirectoryServices.Protocols, it works well out of the box, even for ~2k users. My basic POC class looks like this:
public class DominoLdapManager {
LdapConnection cn = null;
public DominoLdapManager(string ldapHost, int ldapPort, string ldapBindUser, string ldapBindPassword) {
var server = new LdapDirectoryIdentifier(ldapHost, ldapPort);
var credentials = new NetworkCredential(ldapBindUser, ldapBindPassword);
cn = new LdapConnection(server);
cn.AuthType = AuthType.Basic;
cn.Bind(credentials);
}
public IEnumerable<DominoUser> Search(string filter, string searchBase = "") {
string[] attributes = { "cn", "mail", "companyname", "location" };
var req = new SearchRequest(searchBase, filter, SearchScope.Subtree, attributes);
var resp = (SearchResponse)cn.SendRequest(req);
foreach (SearchResultEntry entry in resp.Entries) {
var user = new DominoUser() {
Name = GetStringAttribute(entry, "cn"),
Mail = GetStringAttribute(entry, "mail"),
Company = GetStringAttribute(entry, "companyname"),
Location = GetStringAttribute(entry, "location")
};
yield return user;
}
yield break;
}
string GetStringAttribute(SearchResultEntry entry, string key) {
if (!entry.Attributes.Contains(key)) {
return string.Empty;
}
string[] rawVal = (string[])entry.Attributes[key].GetValues(typeof(string));
return rawVal[0];
}
}
Example usage:
var ldapManager = new DominoLdapManager("ldap.host", 389, "binduser", "pw");
var users = ldapManager.Search("objectClass=person");
But it's not solved with Novell.Directory.Ldap.NETStandard as the title said
This doesn't solve my problem with the Novell.Directory.Ldap.NETStandard library as the title suggested, yes. But since System.DirectoryServices.Protocols is a official .NET package maintained by Microsoft and the .NET foundation, this seems the better aproach for me. The foundation will take care to keep it maintained and compatible with further .NET releases. When I wrote the question, I was not aware of the fact that Linux support is added now.
Don't get me wrong, I don't want to say that third packages are bad by design - that would be completely wrong. However, when I have the choice between a official package and a third party one, I think it makes sense to prefer the official one. Except there would be a good reason against that - which is not the case here: The official package (which doesn't exist in the past) works better to solve this issue than the third party one.

Retrieve all contents of Zoho module via REST API c#

I am trying to get the full contents of my modules From Zoho to our local Server. The deluge code does work as it returns to me the data which is being sent via the API. However, once it reaches the API, it is null. Any idea?
Below is the deluge code:
// Create a map that holds the values of the new contact that needs to be created
evaluation_info = Map();
evaluation_info.put("BulkData",zoho.crm.getRecords("Publishers"));
data = Map();
data.put(evaluation_info);
response = invokeurl
[
url :"https://zohoapi.xxxxx.com/publisher/publish"
type :POST
parameters:data
connection:"zohowebapi"
];
info data; (data returns all the data from publishers)
Here is my ASP.NET core restful API. It does ping it and create the file but the content of the file is null.
Route("[controller]")]
[ApiController]
public class PublisherController : ControllerBase
{
[HttpGet("[action]"), HttpPost("[action]")]
public void Publish(string data)
{
(it's already null when it comes here. why?)
string JSONresult = JsonConvert.SerializeObject(data);
string path = #"C:\storage\journalytics_evaluationsv2.json";
using (var file = new StreamWriter(path, true))
{
file.WriteLine(JSONresult.ToString());
file.Close();
}
}
}
}
What am I missing? Thank you

After contacting Zoho support, the solution he offered was to loop through the data in order to get all the contents from a module (if they are more than 200 records. With the solution provided, one doesn't really need the deluge code anymore as long as you have the ZOHO api set to your account in code. This was my final solution. This solution is not scalable at all. It's best to work with the BULK CSV.
// Our own ZohoAPI which lets us connect and authenticate etc. Yours may look slightly different
ZohoApi zohoApi = new ZohoApi();
zohoApi.Initialize();
ZCRMRestClient restClient = ZCRMRestClient.GetInstance();
var allMedicalJournals = new List<ZCRMRecord>();
for (int i = 1; i <= 30; i++)
{
List<ZCRMRecord> accountAccessRecords2 =
restClient.GetModuleInstance("Journals").SearchByCriteria("Tag:equals:MedicalSet", i, 200).BulkData.ToList();
foreach (var newData in accountAccessRecords2)
allMedicalJournals.Add(newData);
}

AWS Machine Learning RealTimePredictor returns UnknownoperationException in C#

Using Visual Studio, and AWS .NET V 3.0.
I'm trying to perform a real-time Predict operation, and to verify the basic setup works, I first perform a GetMLModel() which works and returns the endpoint (Somewhere in the documentation is was mentioned to use that result as the service endpoint, but it's the same that is listed in the console). Is has status "READY", so far so good.
The exception occurs below on the line below "Prediction P = RTP.Predict(Data)". Data contains a Dictionary with all the prediction values.
Error: Error making request with Error Code UnknownOperationException and Http Status Code BadRequest. No further error information was returned by the service.
public static APIResult GetRealTimePrediction(Dictionary<string, string> Data, string PayloadJSON = null) {
AmazonMachineLearningConfig MLConfig = new AmazonMachineLearningConfig();
MLConfig.RegionEndpoint = Amazon.RegionEndpoint.USEast1;
MLConfig.Validate();
AmazonMachineLearningClient MLClient = new AmazonMachineLearningClient("xxx", "xxx", MLConfig);
GetMLModelResponse MLMOdelResp = MLClient.GetMLModel("xxx"); // <-- WORKS
MLConfig.ServiceURL = MLMOdelResp.EndpointInfo.EndpointUrl;
Console.WriteLine(MLConfig.ServiceURL);
MLConfig.Validate();
Amazon.MachineLearning.Util.RealtimePredictor RTP = new Amazon.MachineLearning.Util.RealtimePredictor(MLClient, "xxx");
Prediction P = RTP.Predict(Data); // <----------------EXCEPTION HERE
}
(Obviously replace xxx with relevant values) :)

It turns out that this line:
MLConfig.ServiceURL = MLMOdelResp.EndpointInfo.EndpointUrl;
cases the MLConfig.RegionEndpoint to be reset. Even though the documentation indicates the RegionEndpoint can be determined from the ServiceURL (I'm pretty sure I read that), the RegionEndpoint needs to be set again before the RTP.Predict(Data) call.
Once I figured that out, I was able to reduce the code to just this, in case anyone else needs help. I guess adding too much information to the Configuration is NOT a good thing, as the AWS. NET library seems to figure all this out on its own.
public static APIResult GetRealTimePrediction(Dictionary<string, string> Data, string PayloadJSON = null) {
AmazonMachineLearningConfig MLConfig = new AmazonMachineLearningConfig();
MLConfig.RegionEndpoint = Amazon.RegionEndpoint.USEast1;
MLConfig.Validate(); // Just in case, not really needed
AmazonMachineLearningClient MLClient = new AmazonMachineLearningClient("xxx", "xxx", MLConfig);
Amazon.MachineLearning.Util.RealtimePredictor RTP = new Amazon.MachineLearning.Util.RealtimePredictor(MLClient, "xxx");
Prediction P = RTP.Predict(Data);
}

How do I retrieve some of videos details (duration, date, views) from a playlist with YouTube API V3 in one pass

Hello SO folks and more specifically Google folks monitoring this tag per your support page. I am working from .NET and PlaylistItems.List("snippet,contentDetails") does not do a whole lot compared to the old RSS Feed search. In fact adding part contentDetails adds little value in that only the VideoID is now returned but it is already part of Snippet.ResourceId.VideoId
"kind": "youtube#playlistItem",
bla,
bla,
"contentDetails": {
"videoId": "DLME0PsJRnk"
}
Why add a "part" which is only going to return one bit of information?
How about supporting something like "snippet,contentDetails(duration,PublishedAt,Views)"
I feel this is kind of basic metadata (snippet) most apps would want to list to the users.
While you are at it please please remove this non-sense of Java casing of parameters. Why would you leak-out your language of choice into an API, that's really sad. Yes it is frustrating to keep checking whether I case-spelled them correctly.
Well, it looks like you are forcing "us" to build a list of VideoIds than turn around and make more API calls when I was doing it previously with fewer.
It also means, I will have to manage the 50 items max paging twice, once for the playlist if it is over 50 videos and then manage manually my list of VideosIds paging when I turn around to make Videos.List calls.
Let me know if I missed an All-In-One call type of API, thank you.

Here is what I have now working, let me know if there is a better way
// 20150802
public async Task<List<YouTubeInfo>> PlaylistVideosInfo(String PlaylistID)
{
var YoutubeService = YouTubeService();
//
List<YouTubeInfo> VideoInfos = new List<YouTubeInfo>();
//
var NextPageToken = "";
while (NextPageToken != null)
{
//
var SearchListRequest = YoutubeService.PlaylistItems.List("snippet");
SearchListRequest.PlaylistId = PlaylistID;
SearchListRequest.MaxResults = 50;
SearchListRequest.PageToken = NextPageToken;
// Call the search.list method to retrieve results matching the specified query term.
var SearchListResponse = await SearchListRequest.ExecuteAsync();
// Collect Video IDs from this page
var VideoIDsBatch = new List<string>(); // batch Video detail search by 50 max
foreach (var searchResult in SearchListResponse.Items)
{
VideoIDsBatch.Add(searchResult.Snippet.ResourceId.VideoId);
}
// Make API call for this batch - expect a single page :(
var VideoListRequest = YoutubeService.Videos.List("snippet,contentDetails");
VideoListRequest.Id = String.Join(",", VideoIDsBatch);
VideoListRequest.MaxResults = 50;
var VideoListResponse = await VideoListRequest.ExecuteAsync();
// Collect each Video details
foreach (var VideoResult in VideoListResponse.Items)
{
YouTubeInfoAdd(VideoInfos, VideoResult);
}
// request next page
NextPageToken = SearchListResponse.NextPageToken;
}
// Return All Videos' detail
return VideoInfos;
}

ServiceNow - Getting all records

In ServiceNow, I am able to get only a maximum of 250 records in a SOAP request. How to get all the records?
Web Reference Url = https://*****.service-now.com/rm_story.do?WSDL
Code:
var url = "https://*****.service-now.com/rm_story.do?SOAP";
var userName = *****;
var password = *****;
var proxy = new ServiceNow_rm_story
{
Url = url,
Credentials = new NetworkCredential(userName, password)
};
try
{
var objRecord = new Namespace.WebReference.getRecords
{
// filters..
};
var recordResults = proxy.getRecords(objRecord);
}
catch (Exception ex)
{
}
In recordResults, I am getting only 250 records. How to get all the records ?

Also see this stack overflow answer which provides info.
Get ServiceNow Records Powershell - More than 250
Note that returning a large number of records can affect performance of the response and it may be more efficient to process your query in batches using offsets (i.e., get 1-100, then 101-200, ...). This can be achieved by using a sort order and offset. The ServiceNow REST Table API actually returns link headers from Get requests providing you links for the first, next and last set of records making it easy to know the url to query the next batch of records.
See: http://wiki.servicenow.com/index.php?title=Table_API#Methods
and look under 'Response Header'.

Have u tried to pass/override __limit parameter?
Google / wiki / Users manual / Release notes are always helpful

In your code snippet in line where it says //filter you should define __limit (and potentially __first_row and __last_row as explained in the example bellow)
int Skip = 0;
int Take = 250;
while (true)
{
using (var soapClient = new ServiceNowLocations.ServiceNow_cmn_location())
{
var cred = new System.Net.NetworkCredential(_user, _pass);
soapClient.Credentials = cred;
soapClient.Url = _apiUrl + "cmn_location.do?SOAP";
var getParams = new ServiceNowLocations.getRecords()
{
__first_row = Skip.ToString(),
__last_row = (Skip + Take).ToString(),
__limit = Take.ToString()
};
var records = soapClient.getRecords(getParams);
if (records != null)
{
if (records.Count() == 0)
{
break;
}
Skip += records.Count();
if (records.Count() != Take)
{
// last batch or everything in first batch
break;
}
}
else
{
// service now web service endpoint not configured correctly
break;
}
}
}

I made an library that handles interacting with ServiceNow Rest API much easier
https://emersonbottero.github.io/ServiceNow.Core/