We Keep Coding

Connect from local machine to aws MSK

I followed "public access" to set up the configuration. I have two goals, Firstly, I want to create topic from local terminal by using this command line "/bin/kafka-topics.sh --create --bootstrap-server ZookeeperConnectString --replication-factor 3 --partitions 1 --topic ExampleTopicName", but it always return "the broker is not available". Secondly, I want to connect MKS from local .Net Application. However, it seams cannot connect to the MKS successfully.
This is my some configuration that attach on my MKS
Create public subnet 172.31.0.0/20 and 172.31.16.0/20 and attach an Internet Gateway
Close unauthenticated access control off and turn on SASL/SCRAM access-control methods. Besides, I attached an secret for this authentication and add allow.everyone.if.no.acl.found to false to cluster's configuration.
Turn on public access
Cluster configuration
Cluster configuration
Producer Configuration
Producer Configuration
Security Group
Security Group
Does anyone can give me some advice or hints? I do some research that not sure I have to add listeners in my cluster configuration? Thanks for your time and consideration.

I was struggling with MSK, too. I finally got it working and maybe give some hints here:
according to the docs at AWS, only SCRAM-SHA-512 is supported, not -256
in the SG, I did add a rule for inbound traffic to accept from anywhere (0.0.0.0)
Hope that helps,
donbachi

How to use service name in DataStax and not IP?

Using the Datastax C# Driver I'm trying to connect to Cassandra which was deployed to Azure Kubernetes Services using a Bitnami helm chart.
var cluster = Cluster.Builder()
.AddContactPoint("127.0.0.0") // example IP
.WithCredentials("cassie", "some-pass")
.Build();
When trying this locally I use kubectl port-forward, but when I'm uploading my service to Kubernetes I want to use service name. Many applications shown to me by colleagues use just that. When I add the link that helm chart gives to me after I install it
Cassandra can be accessed through the following URLs from within the cluster:
CQL: service-name.some-namespace.svc.cluster.local:9042
I'm unable to connect, I'm getting a Cassandra.NoHostAvailableException and I have to use an IP.
How to solve this problem. The IP changes every time I redeploy.
How can I use the name instead of the IP?

Apparently, providing the DNS with format "DNS:PORT" as a parameter for AddContactPoint was not working. Using the method WithPort did the trick.
var cluster = Cluster.Builder()
.WithPort(PORT)
.AddContactPoint("dns")
.WithCredentials("cassie", "some-pass")
.Build();

Google Cloud PubSub V1 using GCloud Emulator

I'm fighting with Google Docs for setting up Cloud PubSub with .NET using a PubSub emulator.
https://cloud.google.com/dotnet/docs/getting-started/using-pub-sub
https://cloud.google.com/pubsub/docs/publisher
https://cloud.google.com/pubsub/docs/emulator
Coming from a Rails background, I'm tasked to implement Cloud PubSub for a .NET product, running our google cloud on .NET Core, to enable it to publish.
Google::Cloud::Pubsub.new(project: project_id, emulator_host: emulator_host)
From the documentation using .NET, I keep coming back to the following:
PublisherServiceApiClient publisherClient = PublisherServiceApiClient.Create();
PublisherClient publisher = PublisherClient.Create(...)
However, the library used from the docs Google.Cloud.PubSub.V1 -Pre
does not contain the definition.
'PublisherClient' does not contain a definition for 'Create'.
Instead, I get CreateAsync that takes in TopicName, PublisherClient.ClientCreationSettings and PublisherClient.Settings.
https://googleapis.github.io/google-cloud-dotnet/docs/Google.Cloud.PubSub.V1/api/Google.Cloud.PubSub.V1.PublisherClient.html
I noticed that PublisherServiceApiClient can take in a Channel, but I'm confused on how to get this going.
To conclude with an actual question, how does one currently implement Cloud PubSub with .NET for in cloud and then locally with emulator? Adding to that, am I using the wrong library or the wrong docs?
Any suggestions, pointers or piece of advice would be truly appreciated.

I managed a solution that I am happy with.
Instead of using the PublisherClient, I went with using the PublisherServiceApiClient alone.
emulatorAddr = Environment.GetEnvironmentVariable("PUBSUB_EMULATOR_HOST");
if (emulatorAddr != null)
{
channel = new Channel(emulatorAddr, ChannelCredentials.Insecure);
pub = PublisherServiceApiClient.Create(channel);
}
else
{
pub = PublisherServiceApiClient.Create();
}
Which meant that publishing was slightly more involved then sending string to the PublisherClient, but overall not so bad.
PubsubMessage msg = new PubsubMessage
{
Data = ByteString.CopyFromUtf8(JsonConvert.SerializeObject(payload))
};
pub.PublishAsync(topic, new[]{ msg });
If the project is running in a Google Compute Engine, it will have default credentials. Otherwise, wether you're running an emulator locally or in docker you can define PUBSUB_EMULATOR_HOST.
What really helped was this https://googleapis.github.io/google-cloud-dotnet/docs/Google.Cloud.PubSub.V1/index.html

To make the PublisherClient connect to a local emulator, you need to pass custom ServiceEndpoint and ChannelCredentials to CreateAsync:
var serviceEndpoint = new ServiceEndpoint(theEmulatorHost, theEmulatorPort);
var publisherClient = await PublisherClient.CreateAsync(
topicName,
new PublisherClient.ClientCreationSettings(credentials: ChannelCredentials.Insecure, serviceEndpoint: serviceEndpoint));
To switch to the real PubSub, just leave away the ClientCreationSettings.

You can use the EmulatorDetection property on the ClientCreationSettings using extension method .WithEmulatorDetection(EmulatorDetection.EmulatorOrProduction). Like this:
PublisherClient publisher = await PublisherClient.CreateAsync(
topicName,
new PublisherClient.ClientCreationSettings()
.WithEmulatorDetection(EmulatorDetection.EmulatorOrProduction));
This will work if you have the following environment variable for the local emulator endpoint: PUBSUB_EMULATOR_HOST=localhost:8085
(If you use Visual Studio you might have to restart VS for the environment variable to be detected)
In windows I had problems using the set PUBSUB_EMULATOR_HOST=localhost:8085 command, so I ended up adding it manually.
Details here: https://cloud.google.com/pubsub/docs/emulator
Extra tip: you can add topics directly to API using curl: curl -X PUT http://localhost:8085/v1/projects/my-project-name/topics/my-topic

What is the proper endpoint for connecting to azure container service swarm cluster from c# https://github.com/Microsoft/Docker.DotNet

After deploying a azuze container service and using swarm, how do one connect using the example given:
var credentials = new CertificateCredentials (new X509Certificate2 ("CertFile", "Password"));
var config = new DockerClientConfiguration("http://ubuntu-docker.cloudapp.net:4243", credentials);
DockerClient client = config.CreateClient();
I have made the certificate and just cant figure out wht the proper endpoint to use is?
the url from azure portl: <name>-mgmt.<region>.cloudapp.azure.com

ACS does not use certs by default. We use SSH tunneling as documented at https://learn.microsoft.com/en-us/azure/container-service/container-service-connect
If you have connected to the masters and manually configured it to use certs as well as correctly installing those certs o the masters then there is nothing magical about the endpoints and connection details. It's just Docker, so follow the appropriate Docker documentation. The correct URL is, as you note in your question <name>-mgmt.<region>.cloudapp.azure.com.
However, you should be aware that since we do not use certs by default we do not open the necessary ports on the Master LB. You will also need to open those on your master LB. For an example (which is against the agent LB but the processes is the same) see https://learn.microsoft.com/en-us/azure/container-service/container-service-enable-public-access

Can a WebServiceHost be changed to avoid the use of HttpListener?

I am looking for a way to use a WCF WebServiceHost without having to rely on the HttpListener class and it's associated permission problems (see this question for details).
I'm working on a application which communicates locally with another (third-party) application via their REST API.
At the moment we are using WCF as an embedded HTTP server. We create a WebServiceHost as follows:
String hostPath = "http://localhost:" + portNo;
WebServiceHost host = new WebServiceHost(typeof(IntegrationService), new Uri(hostPath));
// create a webhttpbinding for rest/pox and enable cookie support for session management
WebHttpBinding webHttpBinding = new WebHttpBinding();
webHttpBinding.AllowCookies = true;
ServiceEndpoint ep = host.AddServiceEndpoint(typeof(IIntegrationService), webHttpBinding, "");
host.Open()
ChannelFactory<IIntegrationService> cf = new ChannelFactory<IIntegrationService>(webHttpBinding, hostPath);
IIntegrationService channel = cf.CreateChannel();
Everything works nicely as long as our application is run as administrator. If we run our application on a machine without administrative privileges the host.Open() will throw an HttpListenerException with ErrorCode == 5 (ERROR_ACCESS_DENIED).
We can get around the problem by running httpcfg.exe from the command line but this is a one-click desktop application and that's not really as long term solution for us.
We could ditch WCF and write our own HTTP server but I'd like to avoid that if possible.
What's the easiest way to replace HttpListener with a standard TCP socket while still using all of the remaining HTTP scaffolding that WCF provides?

Your problem is not related to HttpListener.
Your problem is:
* You have a oneClick application with limited permissions that
* Tries to open a Server port.
This is a contradiction. An untrusted limited permission application should NOT OPEN A SERVER PORT. This is why this is not allowed per definition.
Have you tried opening a normal socket port? It should not work either.
In general, limited trust end user applications should not host a web service ;)
That said, I ahve been in a similar situation trying to use WCF in a driver communication scenario - thank heaven my application runs with full permission.

You can easily compose your own stack via CustomBinding, using the higher level protocol stuff "as is", and rolling your own version of HttpTransport that isn't backed by HttpListener or IIS. Do-able, sure, but it's a lot of work. Take the existing HttpTransport bits apart with Reflector- there are a LOT of moving parts in there. You could probably hack up a simple PoC over Socket in a day or two if you don't need anything fancy like HTTPS or chunking, but making it robust will be tricky. Here's a good wrapup of a bunch of resources (may be a bit dated now).

You could also look at ripping apart enough of Cassini to make it hostable in your app, and loading the WCF pipeline in there (via .svc files and the service activation handler)- it'd require writing very little new code that way, but still give you a fairly robust and tested webserver.