We Keep Coding

AWS Kinesis .NET Consumer

I am experimenting with producer and consumer using AWS Kinesis and the issue is the consumer keeps receiving the first message (or record) that we produced though we have changed the data object sent multiple times . Additionally we have tried multiple ShardIteratorType and none have worked. Latest produces no results, all others produce the same original record.
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Amazon;
using Amazon.Internal;
using Amazon.Kinesis;
using Amazon.Kinesis.Model;
using BenchmarkRuleSetModel.Models;
using MongoDB.Driver;
using Newtonsoft.Json;
namespace ConsoleApp7
{
internal class Program
{
private static AmazonKinesisClient _client;
private static string _streamName;
static async Task ReadFromStream()
{
var kinesisStreamName = _streamName;
var describeRequest = new DescribeStreamRequest
{
StreamName = kinesisStreamName,
};
var describeResponse = await _client.DescribeStreamAsync(describeRequest);
var shards = describeResponse.StreamDescription.Shards;
foreach (var shard in shards)
{
var iteratorRequest = new GetShardIteratorRequest
{
StreamName = kinesisStreamName,
ShardId = shard.ShardId,
ShardIteratorType = ShardIteratorType.AT_TIMESTAMP,
Timestamp = DateTime.MinValue
};
var iteratorResponse = await _client.GetShardIteratorAsync(iteratorRequest);
var iteratorId = iteratorResponse.ShardIterator;
while (!string.IsNullOrEmpty(iteratorId))
{
var getRequest = new GetRecordsRequest
{
ShardIterator = iteratorId, Limit = 10000
};
var getResponse = await _client.GetRecordsAsync(getRequest);
var nextIterator = getResponse.NextShardIterator;
var records = getResponse.Records;
if (records.Count > 0)
{
Console.WriteLine("Received {0} records. ", records.Count);
foreach (var record in records)
{
var json = Encoding.UTF8.GetString(record.Data.ToArray());
Console.WriteLine("Json string: " + json);
}
}
iteratorId = nextIterator;
}
}
}
private static async Task<string> Produce()
{
var data = new
{
Message = "Hello world!",
Author = "Amir"
};
//convert to byte array in prep for adding to stream
var oByte = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(data));
using (var ms = new MemoryStream(oByte))
{
//create put request
var requestRecord = new PutRecordRequest
{
StreamName = _streamName,
PartitionKey = Guid.NewGuid().ToString(),
Data = ms
};
//list name of Kinesis stream
//give partition key that is used to place record in particular shard
//add record as memorystream
//PUT the record to Kinesis
var response = await _client.PutRecordAsync(requestRecord);
return response.SequenceNumber;
}
}
static void Main(string[] args)
{
_client = new AmazonKinesisClient("ExampleKey", "ExampleSecret", RegionEndpoint.EUWest2);
_streamName = "SomeStream";
Produce().Wait();
ReadFromStream().Wait();
}
}
}

First of all, as I have debugged your code, I noticed that it loops infinitely in the inner loop (while (!string.IsNullOrEmpty(iteratorId))) and never loops over all the shards in your stream (assuming you have >1). The reason is explained in https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html#getrecords-returns-empty - because the producer never called MergeShards or SplitShards, they remain open, thus NextShardIterator will never be NULL.
This is why you only ever see records put on the first shard (or at least I did when running your code) - you must read from shards in parallel.
As far as your usage pattern goes, you're using:
ShardIteratorType = ShardIteratorType.AT_TIMESTAMP,
Timestamp = DateTime.MinValue
By this, you're essentially telling Kinesis "give me all the records in the stream from the beginning of time" (or at least as far as the retention period reaches). That's why you keep seeing the same old records in addition to new ones (again, that's what I saw when I ran your code).
A GetRecords[Async] call does not actually remove records from the stream (see https://stackoverflow.com/a/25741304/4940707). The correct way of using Kinesis is to move checkpoint-to-checkpoint. If the consumer was to persist the SequenceNumber from the last record read and then restart as such:
ShardIteratorType = ShardIteratorType.AT_SEQUENCE_NUMBER,
StartingSequenceNumber = lastSeenSequenceNumber
Then you'd see only newer records.

Create task from template via API in Microsoft Team Foundation Server 2018

I'm creating an integration from our testing framework (Selenium) to Team Foundation Server (TFS) 2018 using C# (our tests are already written in it) - the integration will generate work items in TFS based on test results. I'd like to create the work item from a template, and I can't seem to find any documentation on doing so. I'm using the TFS client library from Microsoft found here.
I can pull a template from TFS:
var client = new HttpClient()
// Client auth stuff removed
var method = new HttpMethod("GET");
var httpRequestMessage = new HttpRequestMessage(method, "http://server:port/tfs/collection/team/project/_api/wit/templates/12345abc");
var httpResponseMessage = client.SendAsync(httpRequestMessage).Result;
WorkItemTemplate tfs_template = httpResponseMessage.Content.ReadAsAsync<WorkItemTemplate>().Result;
The API for creating new work items here looks fairly straightforward, but I can't find any way to connect the two actions, or a way to apply a template using this call. Is it possible to create a work item via API with a template, and if so, is there any documentation for it?

You can not create work item from work item template. You can use work item templates to see which fields contains some default values. This example for new work item from template with rest api:
using Microsoft.TeamFoundation.Core.WebApi;
using Microsoft.TeamFoundation.Core.WebApi.Types;
using Microsoft.TeamFoundation.WorkItemTracking.WebApi;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.WebApi;
using Microsoft.VisualStudio.Services.WebApi.Patch;
using Microsoft.VisualStudio.Services.WebApi.Patch.Json;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApp1
{
class Program
{
static string ServiceURL = "https://your_name.visualstudio.com";
static string PAT = "your pat";
static void Main(string[] args)
{
string projectName = "your project";
string templateName = "Critical bug";
/*connect to service. I use VSTS. In your case:
VssConnection connection = new VssConnection(new Uri(ServiceURL), new VssCredential());
https://learn.microsoft.com/ru-ru/azure/devops/integrate/get-started/client-libraries/samples?view=vsts
*/
VssConnection connection = new VssConnection(new Uri(ServiceURL), new VssBasicCredential(string.Empty, PAT));
//get clients
var WitClient = connection.GetClient<WorkItemTrackingHttpClient>();
var ProjectClient = connection.GetClient<ProjectHttpClient>();
var project = ProjectClient.GetProject(projectName).Result;
//get context for default project team
TeamContext tmcntx = new TeamContext(project.Id, project.DefaultTeam.Id);
//get all template for team
var templates = WitClient.GetTemplatesAsync(tmcntx).Result;
//get tempate through its name
var id = (from tm in templates where tm.Name == templateName select tm.Id).FirstOrDefault();
if (id != null)
{
var template = WitClient.GetTemplateAsync(tmcntx, id).Result;
JsonPatchDocument patchDocument = new JsonPatchDocument();
foreach (var key in template.Fields.Keys) //set default fields from template
patchDocument.Add(new JsonPatchOperation()
{
Operation = Operation.Add,
Path = "/fields/" + key,
Value = template.Fields[key]
});
//add any additional fields
patchDocument.Add(new JsonPatchOperation()
{
Operation = Operation.Add,
Path = "/fields/System.Title",
Value = "My critical bug"
});
var newWorkItem = WitClient.CreateWorkItemAsync(patchDocument, projectName, template.WorkItemTypeName).Result;
}
}
}
}
My template:

Authenticating Website Members as Users in CKFinder v3

Before beginning this question, I should point out that my knowledge of ASP.NET & C# is pretty much nil.
I'm in the process of trying to integrate the ASP.NET version of CKFinder v3 into a site built in a different language and all is going well so far; I have everything setup as I want it and it's working when I grant unrestricted access to CKF but I'm stuck at the point now of trying to restrict access to it by authenticating only certain members of my site to use it. All the pages that CKFinder appears on on my site are only accessible by those certain members but I need an extra level of security if, for example, anyone figures out the direct path to my "ckfinder.html" file.
In the ASP version of CKFinder, I simply added this line in the function that checks my member's privileges, where isEditor was a boolean whose value was assigned per member based on information from my database:
session("accessckf")=isEditor
and then edited the CheckAuthentication() function in CKFinder's "config.asp" file to read:
function CheckAuthentication()
CheckAuthentication=session("accessckf")
end function
Reading through this "Howto", authentication seems to be much more complex in v3 but, after a lot of trial and error and some help from Lesiman, I created this C# file, which is located in my CKF directory:
<%#page codepage="65001" debug="true" language="c#" lcid="6153"%>
<%#import namespace="CKSource.CKFinder.Connector.Core"%>
<%#import namespace="CKSource.CKFinder.Connector.Core.Authentication"%>
<%#import namespace="CKSource.CKFinder.Connector.Core.Builders"%>
<%#import namespace="CKSource.CKFinder.Connector.Host.Owin"%>
<%#import namespace="Owin"%>
<%#import namespace="System.Data.Odbc"%>
<%#import namespace="System.Threading"%>
<%#import namespace="System.Threading.Tasks"%>
<script runat="server">
public void Configuration(IAppBuilder appBuilder){
var connectorBuilder=ConfigureConnector();
var connector=connectorBuilder.Build(new OwinConnectorFactory());
appBuilder.Map("/path/to/connector",builder=>builder.UseConnector(connector));
}
public ConnectorBuilder ConfigureConnector(){
var connectorBuilder=new ConnectorBuilder();
connectorBuilder.SetAuthenticator(new MyAuthenticator());
return connectorBuilder;
}
public class MyAuthenticator:IAuthenticator{
public Task<IUser> AuthenticateAsync(ICommandRequest commandRequest,CancellationToken cancellationToken){
var domain=HttpContext.Current.Request.Url.Host;
var cookie=HttpContext.Current.Request.Cookies[urlDomain];
var password="";
var username="";
var user=new User(false,null);
if (cookie!=null){
if (cookie["username"]!=null)
username=cookie["username"];
if (cookie["password"]!=null)
password=cookie["password"];
if(username!=""&&password!=""){
var connection=new OdbcConnection("database=[database];driver=MySQL;pwd=[pwd];server=[server];uid=[uid];");
connection.Open();
OdbcDataReader records=new OdbcCommand("SELECT ISEDITOR FROM MEMBERS WHERE USERNAME='"+username+"' AND PASSWORD='"+password+"'",connection).ExecuteReader();
if(records.HasRows){
records.Read();
bool isEditor=records.GetString(0)=="1";
var roles="member";
if(isEditor)
roles="editor,member";
user=new User(isEditor,roles.Split(','));
}
records.Close();
connection.Close();
}
}
return Task.FromResult((IUser)user);
}
}
</script>
Loading that page produces no errors (which doesn't necessarily mean it's working as trying to write anything to screen from within the public class doesn't work, for some reason) so now I'm at the stage of somehow checking that file for authentication.
Originally, I had tried loading it via XMLHttp from within my function that checks membership privileges for the site but, as I suspected and as Lesmian confirmed, that wouldn't work. After more trial & error, I added code to check website member privileges to the C# file, which leads me to where I am now: stuck!
What do I need to edit within CKFinder in order to have it use this custom file to check whether or not a user is authenticated?

First you'll need a connector between the ASP's Session and CKFinder's .Net authenticator. Here's an example that serializes ASP Session contents into JSON.
Put the connector.asp into a publicly accessible location. http://myaspwebsite.com/connector.asp for example.
connector.asp
<%#Language=VBScript CodePage=65001%>
<% Option Explicit %>
<!--#include file="JSON.asp"-->
<%
' obtain JSON.asp from https://github.com/tugrul/aspjson/archive/master.zip
' just for testing, must be removed in the production environment
Session("isEditor") = True
Session("isMember") = True
' only local requests allowed
' instead of local and remote ip comparison, a secret key can be used
If Request.ServerVariables("LOCAL_ADDR") <> Request.ServerVariables("REMOTE_ADDR") Then
Response.Status = "403 Forbidden"
Response.End
End If
Response.ContentType = "application/json"
Response.Charset = "utf-8"
Dim JSONObject, Key
Set JSONObject = jsObject()
For Each Key In Session.Contents
If Not IsObject(Session.Contents(Key)) Then 'skip the objects cannot be serialized
JSONObject(Key) = Session.Contents(Key)
End If
Next
JSONObject.Flush
%>
CKFinder 3.3.0 comes with a default connector which can be found in /ckfinder/bin/CKSource.CKFinder.Connector.WebApp.dll, remove it.
Examine the following program and remember to replace builder.Map("/connector", SetupConnector); and new Uri("http://myaspwebsite.com/connector.asp"); with your own values.
It simply authenticates the users by checking ASP Session varaibles isEditor and isMember via connector.asp and finally claims the roles editor , member or none.
I assume that you have configured the roles editor and member in the web.config.
Then put the Shaggy.cs into /ckfinder/App_Code. Create App_Code directory if not exist. .Net files in this folder will be compiled on the fly.
For more information have a look at Shared Code Folders in ASP.NET Web Projects
Shaggy.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Owin.Security;
using Microsoft.Owin.Security.Cookies;
using Newtonsoft.Json.Linq;
using Owin;
[assembly: Microsoft.Owin.OwinStartup(typeof(CKSource.CKFinder.Connector.Shaggy.Startup))]
namespace CKSource.CKFinder.Connector.Shaggy
{
using FileSystem.Local;
using FileSystem.Dropbox;
using Core;
using Core.Authentication;
using Config;
using Core.Builders;
using Core.Logs;
using Host.Owin;
using Logs.NLog;
using KeyValue.EntityFramework;
public class Startup
{
public void Configuration(IAppBuilder builder)
{
LoggerManager.LoggerAdapterFactory = new NLogLoggerAdapterFactory();
RegisterFileSystems();
builder.UseCookieAuthentication(new CookieAuthenticationOptions
{
AuthenticationType = "ApplicationCookie",
AuthenticationMode = AuthenticationMode.Active
});
//replace connector path with yours
builder.Map("/connector", SetupConnector);
}
private static void RegisterFileSystems()
{
FileSystemFactory.RegisterFileSystem<LocalStorage>();
FileSystemFactory.RegisterFileSystem<DropboxStorage>();
}
private static void SetupConnector(IAppBuilder builder)
{
var keyValueStoreProvider = new EntityFrameworkKeyValueStoreProvider("CacheConnectionString");
var authenticator = new ShaggysAuthenticator();
var connectorFactory = new OwinConnectorFactory();
var connectorBuilder = new ConnectorBuilder();
var connector = connectorBuilder
.LoadConfig()
.SetAuthenticator(authenticator)
.SetRequestConfiguration(
(request, config) =>
{
config.LoadConfig();
config.SetKeyValueStoreProvider(keyValueStoreProvider);
})
.Build(connectorFactory);
builder.UseConnector(connector);
}
}
public class ShaggysAuthenticator : IAuthenticator
{
// this method makes an http request on the background to gather ASP's all session contents and returns a JSON object
// if the request contains ASP's session cookie(s)
private static JObject GetAspSessionState(ICommandRequest requestContext)
{
// building Cookie header with ASP's session cookies
var aspSessionCookies = string.Join(";",
requestContext.Cookies.Where(cookie => cookie.Key.StartsWith("ASPSESSIONID"))
.Select(cookie => string.Join("=", cookie.Key, cookie.Value)));
if (aspSessionCookies.Length == 0)
{
// logs can be found in /ckfinder/App_Data/logs
LoggerManager.GetLoggerForCurrentClass().Info("No ASP session cookie found");
// don't make an extra request to the connector.asp, there's no session initiated
return new JObject();
}
//replace this URL with your connector.asp's
var publicAspSessionConnectorUrl = new Uri("http://myaspwebsite.com/connector.asp");
var localSafeAspSessionConnectorUrl = new UriBuilder(publicAspSessionConnectorUrl) { Host = requestContext.LocalIpAddress };
using (var wCli = new WebClient())
try
{
wCli.Headers.Add(HttpRequestHeader.Cookie, aspSessionCookies);
wCli.Headers.Add(HttpRequestHeader.Host, publicAspSessionConnectorUrl.Host);
return JObject.Parse(wCli.DownloadString(localSafeAspSessionConnectorUrl.Uri));
}
catch (Exception ex) // returning an empty JObject object in any fault
{
// logs can be found in /ckfinder/App_Data/logs
LoggerManager.GetLoggerForCurrentClass().Error(ex);
return new JObject();
}
}
public Task<IUser> AuthenticateAsync(ICommandRequest commandRequest, CancellationToken cancellationToken)
{
var aspSessionState = GetAspSessionState(commandRequest);
var roles = new List<string>();
var isEditor = aspSessionState.GetNullSafeValue("isEditor", false);
var isMember = aspSessionState.GetNullSafeValue("isMember", false);
if (isEditor) roles.Add("editor");
if (isMember) roles.Add("member");
var isAuthenticated = isEditor || isMember;
var user = new User(isAuthenticated, roles);
return Task.FromResult((IUser)user);
}
}
public static class JObjectExtensions
{
// an extension method to help case insensitive lookups with a default value to get avoid NullReferenceException
public static T GetNullSafeValue<T>(this JObject jobj, string key, T defaultValue = default(T))
{
dynamic val = jobj.GetValue(key, StringComparison.OrdinalIgnoreCase);
if (val == null) return defaultValue;
return (T)val;
}
}
}
Now you should have a working CKFinder connector. Change the logic in the method AuthenticateAsync if you need and see how CKFinder handles your Classic ASP membership management.

Did you setup your custom authentication provider with ConnectorBuilder?
public ConnectorBuilder ConfigureConnector()
{
var connectorBuilder = new ConnectorBuilder();
connectorBuilder.SetAuthenticator(new MyAuthenticator());
return connectorBuilder;
}
You can find full example here: http://docs.cksource.com/ckfinder3-net/configuration_by_code.html.
UPDATE
Additionally you should register ConnectorBuilder inside Startup class to add it to request pipeline:
public void Configuration(IAppBuilder appBuilder)
{
var connectorBuilder = ConfigureConnector();
var connector = connectorBuilder.Build(new OwinConnectorFactory());
appBuilder.Map("/CKFinder/connector", builder => builder.UseConnector(connector));
}
All this is from a documentation link I've provided before.

run a primitive mycouch query

I'm new to couchdb and mycouch. I'm trying to implement a very simple query, I just want to get the results of a view and save it into my DTO class.
My couchdb query works, when I query it manually via HTTP:
http://localhost:5984/mydb/_design/tshirts/_view/getAllTshirts
However, when I try running it from my app using mycouch, I can't get to run it. My current query:
using MyCouch.Requests;
using MyCouch.Responses;
// (...)
using (var client = new Client("http://localhost:5984/samples")) {
var query = new QueryViewRequest("getAllTshirts");
ViewQueryResponse<TShirt[]> result = await client.Views.QueryAsync<TShirt[]>(query);
Console.WriteLine (result);
}
For some reason, it won't find the Client class. I found an example where Client is used on github, as you can see, I'm using all the MyCouch related namespaces as in the example.
I also tried using MyCouchStore instead:
using (var store = new MyCouchStore("http://localhost:5984/", "samples")) {
var query = new QueryViewRequest("getAllTshirts");
ViewQueryResponse<TShirt[]> result = await store.Views.QueryAsync<TShirt[]>(query);
Console.WriteLine (result);
}
However, the store doesn't contain any property named Views.
Any ideas how to query my view using MyCouch?

This is what I do, with the MyCouchStore
using (var store = new MyCouchStore("http://user:password#localhost:5984", "samples")) {
var query = new Query("tshirts", "getAllTshirts");
var rows = store.QueryAsync<TShirt>(query).Result;
}

Apparantely, the documentation was not up to date. The constructor requires now 2 arguments, the second being an optional bootstrapper. This worked for me:
var client = new Client("http://localhost:5984/samples", null)

Windows Service connecting to TFS. IGroupSecurityService instance creation fails with no exception

I have a running service installed on a Windows 2012 R2 server that is trying to connect to a TFS instance.
After turning on remote debugging and looking at the code I noticed the section below which had a breakpoint set. As soon as I clicked step into at the first line, the debugger behaves as though I clicked continue, I exit my step through and the debugger claims the code is continuing. There is no exception thrown.
This code is being called every five minute by a service timer, I think that the code called by the timer never completes, after the first line it experiences some kind of crash.
Five minutes later, the code tries to run again, so I know it has not crashed the entire service.
I have looked at the event view, there is not information there.
IGroupSecurityService gss = (IGroupSecurityService)TfsConfiguration._tfsProjectCollection.GetService(typeof(IGroupSecurityService));
Identity SIDS = gss.ReadIdentity(SearchFactor.AccountName, "Project Collection Valid Users", QueryMembership.Expanded);
List<Identity> FoundIds = gss.ReadIdentities(SearchFactor.Sid, SIDS.Members, QueryMembership.None).ToList();

Check this case:
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.Server;
using Microsoft.TeamFoundation.VersionControl.Client;
using Microsoft.TeamFoundation.Framework.Client;
namespace API
{
class Program
{
static void Main(string[] args)
{
string project = "http://xxx.xxx.xxx.xxx:8080/tfs";
TfsTeamProjectCollection tpc = new TfsTeamProjectCollection(new Uri(project));
var tps = tpc.GetService<VersionControlServer>();
var ttt = tps.GetTeamProject("ProjectName");
ISecurityService securityService = tpc.GetService<ISecurityService>();
System.Collections.ObjectModel.ReadOnlyCollection<SecurityNamespace> securityNamespaces = securityService.GetSecurityNamespaces();
IGroupSecurityService gss = tpc.GetService<IGroupSecurityService>();
Identity SIDS = gss.ReadIdentity(SearchFactor.AccountName, "GroupName", QueryMembership.Expanded);//GourName format: [ProjectName]\\GourpName
IdentityDescriptor id = new IdentityDescriptor("Microsoft.TeamFoundation.Identity", SIDS.Sid);
List<SecurityNamespace> securityList = securityNamespaces.ToList<SecurityNamespace>();
string securityToken;
foreach (SecurityNamespace sn in securityList)
{
if (sn.Description.DisplayName == "Project")
{
securityToken = "$PROJECT:" + ttt.ArtifactUri.AbsoluteUri;
sn.SetPermissions(securityToken, id, 115, 0, true);
}
}
}
}
}

My assumptions ended up being way off base. This is not the source of the problems.... I'll post in another question the real case. This has been an two week issue now, apologies for the waste of a question here.