Connect from local machine to aws MSK - c#

I followed "public access" to set up the configuration. I have two goals, Firstly, I want to create topic from local terminal by using this command line "/bin/kafka-topics.sh --create --bootstrap-server ZookeeperConnectString --replication-factor 3 --partitions 1 --topic ExampleTopicName", but it always return "the broker is not available". Secondly, I want to connect MKS from local .Net Application. However, it seams cannot connect to the MKS successfully.
This is my some configuration that attach on my MKS
Create public subnet 172.31.0.0/20 and 172.31.16.0/20 and attach an Internet Gateway
Close unauthenticated access control off and turn on SASL/SCRAM access-control methods. Besides, I attached an secret for this authentication and add allow.everyone.if.no.acl.found to false to cluster's configuration.
Turn on public access
Cluster configuration
Cluster configuration
Producer Configuration
Producer Configuration
Security Group
Security Group
Does anyone can give me some advice or hints? I do some research that not sure I have to add listeners in my cluster configuration? Thanks for your time and consideration.

I was struggling with MSK, too. I finally got it working and maybe give some hints here:
according to the docs at AWS, only SCRAM-SHA-512 is supported, not -256
in the SG, I did add a rule for inbound traffic to accept from anywhere (0.0.0.0)
Hope that helps,
donbachi

Related

Azure - Function App - VNet Integration - Adding Virtual Network to Web App - UpdateOnResourceNotAllowedWithApiVersion

I have followed this guide to Enable virtual network integration in my Azure Function.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-networking-options?tabs=azure-cli#enable-virtual-network-integration
At first it looks good and my Virtual network with a Virtual network gateway configured with Point to Site VPN shows up as expected:
However on add it fails with the following message:
If I then look at Activity log it does say Succeeded.
Looking at VNet Integration for the Azure Function it has been set up and I get GATEWAY STATUS Online but CERTIFICATE STATUS Certificates not in sync.
Looking at the Networking tab it also says VNet integration Off.
Now looking at the subscription that has the virtual network I did receive the following error the first time I tried to set up the VLAN:
Operation name Creates or updates a VirtualNetworkGateway
Error code UpdateOnResourceNotAllowedWithApiVersion
Message Resource /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-resource-group/providers/Microsoft.Network/virtualNetworkGateways/my-virtual-network-gateway
cannot be updated using API version 2016-09-01 since it uses the
property VpnClientConfiguration AAD authentication parameters which
has been set using a higher API version 2019-04-01. Please use api
version greater than or equal to 2019-04-01 to update the resource.
Does this mean that Azure GUI uses API version 2016-09-01? Can I set the GUI to use API version 2019-04-01 or how can I manually add the configuration needed to make this work?
I have tried to use Sync Network from App Service Plan Network GUI but it fails as well.
There I get the following errors:
Operation name Creates or updates a VirtualNetworkGateway
Error code UpdateOnResourceNotAllowedWithApiVersion
Message Resource /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-resource-group/providers/Microsoft.Network/virtualNetworkGateways/my-virtual-network-gateway
cannot be updated using API version 2016-09-01 since it uses the
property VpnClientConfiguration AAD authentication parameters which
has been set using a higher API version 2019-04-01. Please use api
version greater than or equal to 2019-04-01 to update the resource.
And:
Operation name Generate VpnClient package for virtualNetworkGateway
Error code VpnClientCMakGenerationNotSupportedForVpnClientProtocol
Message Legacy Cmak generation is not supported for gateway id
/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-resource-group/providers/Microsoft.Network/virtualNetworkGateways/my-virtual-network-gateway
when vpn client protocol OpenVPN is configured. Please use vpn profile
package option instead.
Created a new Function App located in the same subscription and region as the Virtual network that I wanted to use.
From here I could use Add VNet Integration and select which subnet I wanted to use. When doing this everything worked.

Authenticating RPC Server with NTLM

I am currently trying to improve an RPC Server I'm responsible for, both server and client run on the same machine locally, however I would like to restrict the server so that it only allows administrator (including built in /LocalSystem account) to connect to the rpc server through a named pipe.
First of all I am using the following library as a wrapper for the RPCserverApi/RPCClientApi:
https://github.com/csharptest/CSharpTest.Net.RpcLibrary
I create the Server like so:
server = new RpcServerApi(IId, MaxCalls, ushort.MaxValue, true);
server.AddProtocol(RpcProtseq.ncacn_np, Id, MaxCalls);
// Set authentication
server.AddAuthentication(RpcAuthentication.RPC_C_AUTHN_WINNT);
However when I check the named pipes security it still shows like it's not restricted at all, and my client can still connect even though I have yet to change that to specify authentication.
In addition I can check the access to that named pipe and I get:
\\.\pipe\myNamedPipe
RW Everyone
RW NT AUTHORITY\ANONYMOUS LOGON
RW BUILTIN\Administrators
Okay, So for anyone else that ran into this problem There's a few things I needed to do which was not exposed in the library I was using. So instead I created my own wrapper.
When Registering the Rpc Interface with RpcServerRegisterIf2() I had to pass through the flag:
RPC_IF_ALLOW_SECURE_ONLY
Then In addition when setting up the protocols for the RpcServer: RpcServerUseProtseqEp() I also had to pass through an SDDL, to describe the restrictions on the end point. You can find a description of SDDL's here:
https://learn.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-definition-language
To do this I created an Ace String, then used ConvertStringSecurityDescriptorToSecurityDescriptor() to create the correct object. This then locked down the end point like:
\\.\pipe\myNamedPipe
RW BUILTIN\Administrators
But also it enforced on the server that only authenticated accounts could reach it
My issue originally reported was full of misunderstandings about RPC Servers and Named pipes, I thoroughly recommend reading and understanding the following articles, as they were very helpful to me.
https://csandker.io/2021/01/10/Offensive-Windows-IPC-1-NamedPipes.html
https://csandker.io/2021/02/21/Offensive-Windows-IPC-2-RPC.html

Socket exception on AWS Secrets Manager GetSecretValueAsync on .NET Core 2.1

I have been struggling to implement secrets manager on .NET Core. It always returning me with Socket Exception and HTTPRequestException on GetSecretValueAsync method call, while it is working on my colleagues PC.
SocketException: A socket operation was attempted to an unreachable network
HttpRequestException: A socket operation was attempted to an unreachable network
I have already setup the credentials file needed in AWS default profile located at C:\Users\USERNAME.aws\credentials on Windows and also on each IIS App Pool .aws folder. All credentials stored in UTF-8 format previously ANSI.
Here is sample of the credentials that i got with MFA authentication:
Here is my aws credentials configuration and folder location:
Mostly my setting with my colleague are the same. Now I'm almost at a dead end. If anybody have experience or AWS Web Services Team willing to help me with the solution for this, please let me know. Thank you. Cheers.
var data = secretManager.GetSecretValueAsync(
new GetSecretValueRequest
{
SecretId = configName
}
).Result;
UPDATE:
I found out the real issue here but still got no solution yet. Because I can't really see where does the secrets manager get the credentials from. And for your information, this method works on .NET Framework in my PC.
Need more info to help .
1. the path for aws credentials is usually
C:\Users\USERNAME.aws\ not
C:\Users\USERNAME.aws\
2. in this folder should be a file named config where the region is defined. Put the region where the AWS Secrets Manager is configured. Here an example
[default]
region = us-east-1
If this is not helping describe the configuration please
I have finally found out the real issue here. It is actually because I need to allow IIS_IUSRS group user to access IIS system profile folder. Thank you everyone!

get hostname of failover cluster role in .Net

I have a Windows service which is supposed to run in a (Windows Server 2012 R2) failover cluster as a generic service in a dedicated role, that is, there is a hostname and IP address configured for this service in the failover cluster manager. (I think 'role' used to be called 'group' in earlier Windows server releases).
One requirement is that the service has to know/provide the hostname of the role it is running in. System.Net.Dns.GetHostName() returns the name of the physical server on which the service is currently active, but what is needed is the configured hostname of the role.
I've searched both in the dns APi direction and the MS documentation for the System.ServiceProcesses namespace, but was not able to figure this out from these resources.
Is there a .Net API which is able to retrieve this, or is that the wrong approach altogether? (I.e. should this information be written into a configuration database during installation and retrieved from there).
There is a .NET API for Failover Clustering. Please refer to it here -
https://msdn.microsoft.com/en-us/library/aa372876(v=vs.85).aspx
As for your qeustion, I believe every Role has an OwnerNode property and this WMI Class should help you.
MSCluster_Node class
[Dynamic, Provider ("MS_CLUSTER_PROVIDER"), UUID ("{C306EBED-0654-4360-AA70-DE912C5FC364}")]class MSCluster_Node : CIM_UnitaryComputerSystem
{
string Roles[];
}
https://msdn.microsoft.com/en-us/library/aa371446(v=vs.85).aspx
If you drill down to the methods there is also a -
ExecuteNodeControl method which even has a CLUSCTL_NODE_GET_ID
https://msdn.microsoft.com/en-us/library/cc512216(v=vs.85).aspx
If the above doesn't help you, you can also try the reference below.
The MSCluster_ResourceToPossibleOwner class is a dynamic association WMI class that represents a list of the resources and their possible owner nodes.
https://msdn.microsoft.com/en-us/library/aa371478(v=vs.85).aspx
Hope this helps, I'm pretty new to doing stuff with Failover Clustering and C#. I hope I can learn from this post as well.

Access Remote MSMQ Count

My machine is in Domain D1 and there are public MSMQs in a remote server in domain D2. I am connected through vpn to D2, i.e I can RDP the machine in D2 and access the MSMQ.
What I want is to access (Know the message count) of the MSMQ without RDPing the system. So I build an application for this. I used Impersonation to impersonate the user of D2(i.e used credentials of D2)but the problem is I am not able to access the "Public" MSMQ ( used Messagequeue.GetPublicQueue() ) and exceptions are thrown with message "A workgroup installation computer does not support the operation." but when I used MessageQueue.GetPrivateQueue() it returned a collection of private queue.
I tried using MSMQManager for messageCount
Path = #"Direct:OS:machine\publicqueue";
FormatName=null;
new MSMQManager.inIt(machineName, path , FormatName);
This also throws an exception either the queue is not present or not open. but I can check that queue is working fine.
Are you comfortable doing a tiny bit of programming? If not, are you comfortable using PowerShell?
Either way - I would check out this post as it seems to contain the answers you are looking for.
Good luck, hope this helps
Your problem might be that you are working remotely.
The method GetPublicQueuesByMachine() is indeed not available over remote access.
You can see this in a feature matrix in the MSDN documentation: MessageQueue.GetPublicQueuesByMachine:
The following table shows whether this method is available in various Workgroup modes.
Workgroup mode Available
-------------- ---------
Local computer No
Local computer and direct format name No
Remote computer No
Remote computer and direct format name No
Also check the access privileges of your queues.
If I am wrong in the previous suggestion, it might be as simple as experimenting with the access rights for specific users in the network.
MSDN article Public and private queues states:
Default security access for public queues gives everyone permission to
send messages to a public queue. Specific permissions must be
granted for read access.
As for the actual message counting, John Opincar wrote a nice article about counting messages here: Counting Messages in an MSMQ MessageQueue from C#

Categories