Memory exceptions when scraping Active Directory - c#

I am trying to scrape all the groups a user is a member of in Active Directory (including groups that they are indirect members of), but I am running into memory exception errors.
Latest error message passed was:
Insufficient memory to continue the execution of the program
The code I run is started in a backgroundworker on a schedule.
I have looked through ALL the SO active directory group scraping posts, and tried as many of the iterations of this process as I could get to work. This one will (usually) work if i don't run it on a schedule. Sometimes it will crash the program though.
Our Active Directory database is quite considerably large.
Any help you could offer with this would be great. I don't mind it taking a bit longer, speed is not an overly great concern, as long as I can get it to perform well, and correctly.
Here's what I have:
DataTable resultsTable = new DataTable();
resultsTable.Columns.Add("EmailID");
resultsTable.Columns.Add("UserID");
resultsTable.Columns.Add("memberOf");
resultsTable.Columns.Add("groupType");
resultsTable.Columns.Add("record_date");
try
{
string RecordDate = DateTime.Now.ToString(format: "dd/MM/yyyy", provider: CultureInfo.CreateSpecificCulture("en-GB"));
string ou = "OU=Groups,OU=nonya,DC=my,DC=domain,DC=net";
using (PrincipalContext context = new PrincipalContext(ContextType.Domain, "my.domain.net", ou))
{
using (GroupPrincipal GroupPrin = new GroupPrincipal(context))
{
using (var searcher = new PrincipalSearcher(GroupPrin))
{
using (var results = searcher.FindAll())
{
foreach (var result in results)
{
DirectoryEntry de = result.GetUnderlyingObject() as DirectoryEntry;
string GUID = de.Guid.ToString();
string testname1 = de.Name;
string testParentName1 = de.Parent.Name;
string MasterGroupname = testname1.Substring(3, (testname1.Length - 3));
string type = testParentName1.Substring(3, (testParentName1.Length - 3));
using (var group = GroupPrincipal.FindByIdentity(context, IdentityType.Guid, GUID))
{
using (var users = group.GetMembers(true))
{
foreach (Principal user in users)
{
DataRow dr1 = resultsTable.NewRow();
dr1["EmailID"] = user.UserPrincipalName;
dr1["UserID"] = user.SamAccountName;
dr1["memberOf"] = MasterGroupname;
dr1["groupType"] = type;
dr1["record_date"] = RecordDate;
resultsTable.Rows.Add(dr1);
user.Dispose();
}
}
}
}
}
}
}
}
//Table is uploaded to SQL database here, unless it crashes before reaching this point
}
catch (Exception E)
{
logger.Error($"ADGroupScrape error. System says: {E.Message}");
return false;
}
finally
{
resultsTable.Dispose();
}

Disposing each object at the end of the loop will probably solve your problem (result.Dispose()). I've had memory problems with long loops when working with AD. Assuming you're working with thousands of results, and you're allocating memory for each, and there's no break between processing each result, the garbage collector doesn't have an opportunity to clean up for you.
But you're also unnecessarily going back out to AD to search for the group you already found (GroupPrincipal.FindByIdentity). That would also increase memory consumption, but also slow down the whole operation. Instead, just cast result to a GroupPrincipal so you can call .GetMembers() on it.
Unrelated, but helpful: When you have multiple using blocks nested like that, you can combine them all into one block. It saves you indenting so much.
This is what your code would look like with all those suggestions:
using (PrincipalContext context = new PrincipalContext(ContextType.Domain, "my.domain.net", ou))
using (GroupPrincipal GroupPrin = new GroupPrincipal(context))
using (var searcher = new PrincipalSearcher(GroupPrin))
using (var results = searcher.FindAll())
{
foreach (GroupPrincipal result in results)
{
DirectoryEntry de = result.GetUnderlyingObject() as DirectoryEntry;
string GUID = de.Guid.ToString();
string testname1 = de.Name;
string testParentName1 = de.Parent.Name;
string MasterGroupname = testname1.Substring(3, (testname1.Length - 3));
string type = testParentName1.Substring(3, (testParentName1.Length - 3));
using (var users = result.GetMembers(true))
{
foreach (Principal user in users)
{
DataRow dr1 = resultsTable.NewRow();
dr1["EmailID"] = user.UserPrincipalName;
dr1["UserID"] = user.SamAccountName;
dr1["memberOf"] = MasterGroupname;
dr1["groupType"] = type;
dr1["record_date"] = RecordDate;
user.Dispose();
}
}
result.Dispose();
}
}
This will probably work, but could probably still be much faster. Using Principal objects (and the whole AccountManagement namespace) is a wrapper around DirectoryEntry/DirectorySearcher which makes things somewhat easier for you, but at the cost of performance. Using DirectoryEntry/DirectorySearcher directly is always faster.
If you want to experiment with that, I wrote a couple articles that will help:
Active Directory: Better performance
Active Directory: Find all the members of a group

So the issue didn't appear to be with my code, but thank you all for the suggestions in cleaning it up.
I changed the Target Platform in Configuration Manager (Visual Studio) to X64, as it was originally set to "Any CPU", and my code now runs successfully every time.

Couple of months down the line and i started getting out of memory exceptions again.
added pagination (which i didnt actually know you could do) and it seems to have resolved the issue. The line i added was ((DirectorySearcher)searcher.GetUnderlyingSearcher()).PageSize = 500; in between the searcher declaration and the searcher.FindAll().
using (var searcher = new PrincipalSearcher(GroupPrin))
{
((DirectorySearcher)searcher.GetUnderlyingSearcher()).PageSize = 500;
using (var results = searcher.FindAll())
{

Related

Invert filter in AD search

I want to search in AD all users which names are NOT starts with prefix.
How shoud I do this?
This does not work
using (var context = new PrincipalContext(ContextType.Domain, "my_do_main"))
{
UserPrincipal template = new UserPrincipal(context);
template.UserPrincipalName = "!my_prefix*"; //invertion NOT works
using (var searcher = new PrincipalSearcher(template))
{
foreach (var result in searcher.FindAll())
{
var de = result.GetUnderlyingObject() as DirectoryEntry;
Console.WriteLine(de.Properties["userPrincipalName"].Value);
}
}
}
You can't do this using PrincipalSearcher, but you can do it using DirectorySearcher, which is what PrincipalSearcher uses behind the scenes anyway. Here is a quick example:
var search = new DirectorySearcher(new DirectoryEntry("LDAP://my_do_main")) {
PageSize = 1000,
Filter = "(&(objectClass=user)(!userPrincipalName=my_prefix*))"
};
search.PropertiesToLoad.Add("userPrincipalName");
using (var results = search.FindAll()) {
foreach (SearchResult result in results) {
Console.WriteLine((string) result.Properties["userPrincipalName"][0]);
}
}
You'll find this will perform much faster anyway. In my experience, using DirectorySearcher and DirectoryEntry directly is always much faster than using PrincipalSearcher (or anything in the AccountManagement namespace). A little while ago I wrote an article about that subject: Active Directory: Better Performance

Query user data from a specific AD DC

I'd like to query some user attribute in our AD, but on a specific DC, in C#.
We've a few dozens of DCs and I suspect that there is some replication issue between them. I'd like to make a cleanup of unused accounts, to which I'd like to use the last logon time attribute and I'd like to query this on all DCs one bye one (I know this is a bit like brute forcing, however I don't intended to do such a thing too often) so I can see if the most recent value is up-to date or not.
I had the code to query all the DCs:
Domain TestDomain = Domain.GetCurrentDomain();
Console.WriteLine("Number of found DCs in the domain {0}", TestDomain.DomainControllers.Count);
foreach (DomainController dc in TestDomain.DomainControllers)
{
Console.WriteLine("Name: " + dc.Name);
///DO STUFF
}
And I also found help to construct the code that can query a user from AD:
PrincipalContext context = new PrincipalContext(ContextType.Domain, "test.domain.com");
string userName = "testusername";
UserPrincipal user = UserPrincipal.FindByIdentity(context, userName);
Console.WriteLine(user.LastLogon.Value.ToString());
Console.ReadKey();
And here I stuck. Now I'd like to get the user's last logon timestamp from all DCs.
In the past I already deleted accidentaly account that seemed to be unused for a long time (check on only one DC), than it turned out the user use it every day so the info came from the DC was not synced.
I'm aware that the most reasonable action would be to review what cause this incorrect sync phenomena, however in my current status that would take ages and probably ended without any finding...
Thanks in advance for any constructive response/comment!
While the accepted answer was a good kick in the right direction, I found it to ultimately not produce the expected result: As hinted in the answer, "lastLogon" is not replicated between controllers, whereas the attribute "lastLogonTimeStamp" is replicated (but it is, on the other hand, not guaranteed to be more than 14 days wrong, cf. this answer).
Now, rather confusingly, UserPrincipal.LastLogon refers not to the unreplicated but precise "lastLogon" but to the replicated but imprecise "lastLogonTimeStamp", and I found that by running the code in the accepted answer, all produced DateTimes were equal and wrong.
Instead, inspired by this answer, I found that in order to find the most recent logon date for a user with a given sAMAccountName (which you can easily extend to search for all users), I would have to do something like the following:
public DateTime? FindLatestLogonDate(string username)
{
var logons = new List<DateTime>();
DomainControllerCollection domains = Domain.GetCurrentDomain().DomainControllers;
foreach (DomainController controller in domains)
{
using (var directoryEntry = new DirectoryEntry($"LDAP://{controller.Name}"))
{
using (var searcher = new DirectorySearcher(directoryEntry))
{
searcher.PageSize = 1000;
searcher.Filter = $"((sAMAccountName={username}))";
searcher.PropertiesToLoad.AddRange(new[] { "lastLogon" });
foreach (SearchResult searchResult in searcher.FindAll())
{
if (!searchResult.Properties.Contains("lastLogon")) continue;
var lastLogOn = DateTime.FromFileTime((long)searchResult.Properties["lastLogon"][0]);
logons.Add(lastLogOn);
}
}
}
}
return logons.Any() ? logons.Max() : (DateTime?)null;
}
EDIT: After re-reading your question I realised what the problem actually is. You believe you have a replication issue because a user's Last Logon attribute doesnt match on all domain controllers? This is by design! That attribute is domain controller specific and IS NOT REPLICATED. To check the true last logon time of a user you must always query every domain controller to find the latest time!
You are almost there, try this:
public static List<UserPrincipal> GetInactiveUsers(TimeSpan inactivityTime)
{
List<UserPrincipal> users = new List<UserPrincipal>();
using (Domain domain = Domain.GetCurrentDomain())
{
foreach (DomainController domainController in domain.DomainControllers)
{
using (PrincipalContext context = new PrincipalContext(ContextType.Domain, domainController.Name))
using (UserPrincipal userPrincipal = new UserPrincipal(context))
using (PrincipalSearcher searcher = new PrincipalSearcher(userPrincipal))
using (PrincipalSearchResult<Principal> results = searcher.FindAll())
{
users.AddRange(results.OfType<UserPrincipal>().Where(u => u.LastLogon.HasValue));
}
}
}
return users.Where(u1 => !users.Any(u2 => u2.UserPrincipalName == u1.UserPrincipalName && u2.LastLogon > u1.LastLogon))
.Where(u => (DateTime.Now - u.LastLogon) >= inactivityTime).ToList();
}
It won't show people who've never logged in though. If you need that you can probably figure it out.

c# Directory Services Synchronization does not return changed relationships

I'm using C# to work with AD (Win 2012R2).
We are syncing AD users,groups and their relationship to SQL database.
Full sync works well.
But when using synchronization cookie, the relationship changes does not detected.
What may be the reason?
Thanks.
Here is my code:
public void DirSyncChanges(DirectoryEntry de, byte[] cookie)
{
DirectorySynchronization syncData = new DirectorySynchronization(cookie);
srch = new DirectorySearcher(de)
{
Filter = "(&(objectClass=user)(objectCategory=person))",
SizeLimit = Int32.MaxValue,
Tombstone = true
};
srch.DirectorySynchronization = syncData;
syncData.Option = DirectorySynchronizationOptions.None;
using(SearchResultCollection results = srch.FindAll())
foreach (SearchResult res in results)
{
//results is empty. no loop
}
}
Please specify the DirectorySearcher.PropertiesToLoad. Only if any of the attributes in PropertiesToLoad are updated, you will get them in delta sync.
As i remember the search root of DirSync must be naming context root object.
Better use paged search. No matter how large the value you set to SizeLimit. It will only return at most 1000 or 1500 (forgot exact number) results.
My answer is based on .NET 3.5.

foreach loop in DirectorySearcher.FindAll() method

I need to run a foreach loop in DirectorySearcher.FindAll() and get the displayname property. It seems like there are memory issues with that (referred link: Memory Leak when using DirectorySearcher.FindAll()).
My code is as follows:
List<string> usersList = new List<string>();
string displayName = string.Empty;
try
{
using (DirectoryEntry directoryEntry = new DirectoryEntry(ldap, userName, password))
{
DirectorySearcher directorySearcher = new DirectorySearcher(directoryEntry);
directorySearcher.PageSize = 500; // ADD THIS LINE HERE !
string strFilter = "(&(objectCategory=User))";
directorySearcher.PropertiesToLoad.Add("displayname");//first name
directorySearcher.Filter = strFilter;
directorySearcher.CacheResults = false;
SearchResult result;
var resultOne = directorySearcher.FindOne();
using (var resultCol = directorySearcher.FindAll())
{
for (int counter = 0; counter < resultCol.Count; counter++)
{
result = resultCol[counter];
if (result.Properties.Contains("displayname"))
{
displayName = (String)result.Properties["displayname"][0];
usersList.Add(displayName);
}
}
}
}
}
Is there any possible way to looping. I have also tried calling Dispose() method but it doesn't work. Any help is really appreciated.
Here is a solution that I came up with for a project I'm working on that I think would work for you. My solution loops through the search results and returns the display name in a list of strings up to 20.
using (DirectorySearcher ds = new DirectorySearcher())
{
//My original filter
//ds.Filter = string.Format("(|(&(objectClass=group)(|(samaccountname=*{0}*)(displayname=*{0}*)))(&(objectCategory=person)(objectClass=user)(|(samaccountname=*{0}*)(displayname=*{0}*))))", name);
//Your Modified filter
ds.filter = "(objectCategory=User)"
ds.PropertiesToLoad.Add("displayname");
ds.SizeLimit = 20;
SearchResultCollection result = ds.FindAll();
List<string> names = new List<string>();
foreach (SearchResult r in result)
{
var n = r.Properties["displayname"][0].ToString();
if (!names.Contains(n))
names.Add(n);
}
return Json(names, JsonRequestBehavior.AllowGet);
}
If it's not valid solution in managed code and your code depend hard on not-managed things such Active Directory i think it's good idea to move logic of directory searching to isolate such logic from your main application domain. It's way to not run against wind.
For me - better is to use outer console process, send parameters to argument string and catch StandardOutput.
It will be slower, but memory will be not leaked because it will be freed by system after console process die.
Additionally in this case u can not to make self-made ad_searcher.ex, but use standard dsquery and so on. For example:
[https://serverfault.com/questions/49405/command-line-to-list-users-in-a-windows-active-directory-group]
Another approach - usage of managed sub-domains, BUT i think it's harder and i'm not sure that all unmanaged resources will be freed.

List all computers in active directory

Im wondering how to get a list of all computers / machines / pc from active directory?
(Trying to make this page a search engine bait, will reply myself. If someone has a better reply il accept that )
If you have a very big domain, or your domain has limits configured on how how many items can be returned per search, you might have to use paging.
using System.DirectoryServices; //add to references
public static List<string> GetComputers()
{
List<string> ComputerNames = new List<string>();
DirectoryEntry entry = new DirectoryEntry("LDAP://YourActiveDirectoryDomain.no");
DirectorySearcher mySearcher = new DirectorySearcher(entry);
mySearcher.Filter = ("(objectClass=computer)");
mySearcher.SizeLimit = int.MaxValue;
mySearcher.PageSize = int.MaxValue;
foreach(SearchResult resEnt in mySearcher.FindAll())
{
//"CN=SGSVG007DC"
string ComputerName = resEnt.GetDirectoryEntry().Name;
if (ComputerName.StartsWith("CN="))
ComputerName = ComputerName.Remove(0,"CN=".Length);
ComputerNames.Add(ComputerName);
}
mySearcher.Dispose();
entry.Dispose();
return ComputerNames;
}
What EKS suggested is correct, but is performing a little bit slow.
The reason for that is the call to GetDirectoryEntry() on each result. This creates a DirectoryEntry object, which is only needed if you need to modify the active directory (AD) object. It's OK if your query would return a single object, but when listing all object in AD, this greatly degrades performance.
If you only need to query AD, its better to just use the Properties collection of the result object. This will improve performance of the code several times.
This is explained in documentation for SearchResult class:
Instances of the SearchResult class are very similar to instances of
DirectoryEntry class. The crucial difference is that the
DirectoryEntry class retrieves its information from the Active
Directory Domain Services hierarchy each time a new object is
accessed, whereas the data for SearchResult is already available in
the SearchResultCollection, where it gets returned from a query that
is performed with the DirectorySearcher class.
Here is an example on how to use the Properties collection:
public static List<string> GetComputers()
{
List<string> computerNames = new List<string>();
using (DirectoryEntry entry = new DirectoryEntry("LDAP://YourActiveDirectoryDomain.no")) {
using (DirectorySearcher mySearcher = new DirectorySearcher(entry)) {
mySearcher.Filter = ("(objectClass=computer)");
// No size limit, reads all objects
mySearcher.SizeLimit = 0;
// Read data in pages of 250 objects. Make sure this value is below the limit configured in your AD domain (if there is a limit)
mySearcher.PageSize = 250;
// Let searcher know which properties are going to be used, and only load those
mySearcher.PropertiesToLoad.Add("name");
foreach(SearchResult resEnt in mySearcher.FindAll())
{
// Note: Properties can contain multiple values.
if (resEnt.Properties["name"].Count > 0)
{
string computerName = (string)resEnt.Properties["name"][0];
computerNames.Add(computerName);
}
}
}
}
return computerNames;
}
Documentation for SearchResult.Properties
Note that properties can have multiple values, that is why we use Properties["name"].Count to check the number of values.
To improve things even further, use the PropertiesToLoad collection to let the searcher know what properties you are going to use in advance. This allows the searcher to only read the data that is actually going to be used.
Note that the DirectoryEntry and DirectorySearcher objects should
be properly disposed in order to release all resources used. Its best
done with a using clause.
An LDAP query like: (objectCategory=computer) should do the trick.
if you only want to get the enabled computers:
(&(objectclass=computer)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))

Categories