We Keep Coding

How to manage a series of PATCH calls to Azure DevOps REST API to avoid HTTP Response Conflict error 409

In a custom C# Winforms app, I'm using the Azure DevOps REST API Update Comments call to update work item comments using async/await
My call to UpdateComment_Async is in a tight loop designed to submit all update comment requests for a single work item and then process each comment update as it completes.
Following is a mix of p-code and C# for (almost) working code. In the case where there are 2 (or more) comments to update, the first update returns HttpResponseMessage.Status 200 (success), but the second update submitted returns HttpResponseMessage.Status 409 "Conflict". I assume that ADS has the work item locked for the first update, and so the 2nd update fails with the 409. I think I proved this by sleeping the thread for 5 seconds after the call to UpdateComment_Async. With the sleep in place, both updates work.
Is there a way to manage the series of calls to UpdateComment_Async so that subsequent calls aren't done until the previous one is complete?
CODE
// get comment(s) for one work item .......
foreach (comment in workItem)
{
newCommentText = "new comment text blah-blah-blah";
Task<Tuple<string, string, HttpResponseMessage>> updateCommentTask = UpdateComment_Async(projectUrl, workItem.Id.ToString(), comment.Id.ToString(), newCommentText);
// I put a thread.sleep(5000) right here
updateCommentTaskList.Add(updateCommentTask);
}
// Process update comments tasks as they complete
while (updateCommentTaskList.Count > 0)
{
Task<Tuple<string, string, HttpResponseMessage>> finishedUpdateCommentTask = await Task.WhenAny(updateCommentTaskList);
// Get Results
Tuple<string, string, HttpResponseMessage> updateCommentTaskResult = finishedUpdateCommentTask.Result;
// process updateCommentTaskResult
// etc
// etc
// etc
updateCommentTaskList.Remove(finishedUpdateCommentTask);
}
//*******************************
public async Task<Tuple<string, string, HttpResponseMessage>> UpdateComment_Async(string projectUrl, string workItemId, string commentId, string commentNumber, string newCommentText)
{
HttpResponseMessage responseResult = null;
#region MAKE JSON REQUEST BODY
IList<ClsUpdateComment> updateFieldJsonList = new List<ClsUpdateComment>();
updateFieldJsonList.Clear();
// Note: This code works but is not in compliance with MS docs on 2 counts.
//
// 1) The MS docs on comment update say that the body should look like fig 1.
// The only I could do this was to create a new ClsUpdateComment and then add it to
// List<ClsUpdateComment>. Adding the ClsUpdateComment object to a list causes the [ ]
// to be created when the list is serialized. To make this work, I had to serialize
// just the ClsUpdateComment object so that when serialzed, it ends up looking like Fig 2 (no brackets)
//
// 2) application/json-patch+json causes error 415 - unsupported media type to occur. application/json
// works.
/*
Fig 1
[
{
"text": "Moving to the right area path - Fabrikam-Git"
}
]
Fig 2
{
"text": "Moving to the right area path - Fabrikam-Git"
}
*/
ClsUpdateComment updateFieldJson = new ClsUpdateComment
{
Text = $"{newCommentText}"
};
updateFieldJsonList.Add(updateFieldJson);
#endregion MAKE JSON REQUEST BODY
#region SUBMIT UPDATE REQUEST
string request = $"{projectUrl}/_apis/wit/workitems/{workItemId}/comments/{commentId}?api-version=5.1-preview.3";
JsonSerializerSettings jsonSerializerSettings = new JsonSerializerSettings
{
StringEscapeHandling = StringEscapeHandling.EscapeHtml,
};
string updateFieldJsonSerialized = JsonConvert.SerializeObject(updateFieldJson, Formatting.None, jsonSerializerSettings);
using (HttpClient client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true, ClientCertificateOptions = ClientCertificateOption.Manual }))
{
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
HttpRequestMessage httpRequestMessage = new HttpRequestMessage(new HttpMethod("PATCH"), request)
{
Content = new StringContent(updateFieldJsonSerialized, Encoding.UTF8, "application/json")
};
using (responseResult = await client.SendAsync(httpRequestMessage))
{
// don't need Content, just the HttpResponseMessage
// //string content = await responseResult.Content.ReadAsStringAsync();
}
}
#endregion SUBMIT UPDATE REQUEST
return Tuple.Create(workItemId, commentNumber, responseResult);
}

It's not an answer, but after trying to do mass updates of comments using the REST API in various ways, I've come to the conclusion that it simply doesn't support mass updates of comments

ASP.NET Web-api core: Handling client connections problems and find conflicts

Background
I have a web api server (asp.net core v2.1) that serve some basic operation, like managing entities on the server. This is the interface:
[HttpPost]
[Route("create")]
public async Task<ActionResult<NewEntityResponse>> Create(CreateEntityModel model)
{
// 1) Validate the request.
// 2) Create a new row on the database
// 3) Return the new entity in response.
}
The user running this REST method in this way:
POST https://example.com/create
Content-Type: application/json
{
"firstName": "Michael",
"lastName": "Jorden"
}
And getting response like this:
Status 200
{
"id": "123456" // The newly created entity id
}
The Problem
When sending thousands of requests like this, at some point it will fail because of network connections. When connection fails, it can leads us into two different situations:
The network call was ended on the way to the server - In this case, the server don't know about this request. Therefore, the entity wasn't created. The user just have to send the same message again.
The network call was sent from the server to back to the client but never rich the destination - In this case the request was fulfill completely, but the client don't aware for this. The expected solution is to send the same request again. In this case, it will create the same entity twice - and this is the problem.
The Requested Solution
I want to create an generic solution for web-api that "remmeber" which commands it already done. if he got same request twice, it's return HTTP status code Conflict.
Where I got so far
I thought to add the client an option to add a unique id to the request, in this way:
POST https://example.com/create?call-id=XXX
Add to my server a new filter that check if the key XXX is already fulfill. If yes, return Conflict. Otherwise - continue.
Add another server filter that checks the response of the method and marking it as "completed" for further checks.
The problem with this solution on concurrency calls. If my method takes 5 seconds to be returned and the client sent the same message again after 1 second - it will create two entities with same data.
The Questions:
Do you think that this is good approach to solve this problem?
Do you familiar with ready to use solutions that doing this?
How to solve my "concurrency" problem?
Any other tips will be great!
thanks.

Isnt the easiest solution to make the REST action idempotent?
I mean by that: the call should check if the resource already exists and either create a new resource if it doesnt or return the existing if it does?

OK, I just figure it up how to make it right. So, I implemented it by myself and share it with you.
In order to sync all requests between different servers, I used Redis as cache service. If you have only one server, you can use Dictionary<string, string> instead.
This filter do:
Before processing the request - add a new empty value key to Redis.
After the server processed the request - store the server response in Redis. This data will be used when the user will ask again for same request.
public class ConflictsFilter : ActionFilterAttribute
{
const string CONFLICT_KEY_NAME = "conflict-checker";
static readonly TimeSpan EXPIRE_AFTER = TimeSpan.FromMinutes(30);
private static bool ShouldCheck(ActionDescriptor actionDescription, IQueryCollection queries)
{
return queries.ContainsKey(CONFLICT_KEY_NAME);
}
private string BuildKey(string uid, string requestId)
{
return $"{uid}_{requestId}";
}
public override void OnActionExecuting(ActionExecutingContext context)
{
if (ShouldCheck(context.ActionDescriptor, context.HttpContext.Request.Query))
{
using (var client = RedisConnectionPool.ConnectionPool.GetClient())
{
string key = BuildKey(context.HttpContext.User.GetId(), context.HttpContext.Request.Query[CONFLICT_KEY_NAME]);
string existing = client.Get<string>(key);
if (existing != null)
{
var conflict = new ContentResult();
conflict.Content = existing;
conflict.ContentType = "application/json";
conflict.StatusCode = 409;
context.Result = conflict;
return;
}
else
{
client.Set(key, string.Empty, EXPIRE_AFTER);
}
}
}
base.OnActionExecuting(context);
}
public override void OnResultExecuted(ResultExecutedContext context)
{
base.OnResultExecuted(context);
if (ShouldCheck(context.ActionDescriptor, context.HttpContext.Request.Query) && context.HttpContext.Response.StatusCode == 200)
{
string key = BuildKey(context.HttpContext.User.GetId(), context.HttpContext.Request.Query[CONFLICT_KEY_NAME]);
using (var client = RedisConnectionPool.ConnectionPool.GetClient())
{
var responseBody = string.Empty;
if (context.Result is ObjectResult)
{
ObjectResult result = context.Result as ObjectResult;
responseBody = JsonConvert.SerializeObject(result.Value);
}
if (responseBody != string.Empty)
client.Set(key, responseBody, EXPIRE_AFTER);
}
}
}
}
The code is executed only if the query ?conflict-checker=XXX is exists.
This code is provide you under MIT license.
Enjoy the ride :)

Search amazon example with new amazon service

I can not find a working example of the new amazon service (or at least, within the last couple of years). The closest working example just comes back with a null item no matter what I put in the title. The code is:
// Amazon ProductAdvertisingAPI client
AWSECommerceServicePortTypeClient amazonClient = new AWSECommerceServicePortTypeClient();
// prepare an ItemSearch request
ItemSearchRequest request = new ItemSearchRequest();
request.SearchIndex = "Books";
request.Title = "C#";
request.Condition = Condition.All;
//request.ResponseGroup = new string[] { "Small" };
ItemSearch itemSearch = new ItemSearch();
itemSearch.Request = new ItemSearchRequest[] { request };
itemSearch.AWSAccessKeyId = ConfigurationManager.AppSettings["accessKeyId"];
// send the ItemSearch request
ItemSearchResponse response = amazonClient.ItemSearch(itemSearch);
// write out the results from the ItemSearch request
foreach (var itemLst in response.Items)
{
if (itemLst.Item != null)
{
foreach (var item in response.Items[0].Item)
{
Console.WriteLine(item.ItemAttributes.Title);
}
}
else
Console.WriteLine("No item info was found for this response list item.");
}
Console.WriteLine("<Done...press enter to continue>");
Console.ReadLine();
What am I doing wrong?

I'm assuming that you've downloaded the code from here. If this is correct then you need to replace this line:
AWSECommerceServicePortTypeClient amazonClient = new AWSECommerceServicePortTypeClient();
With these lines:
BasicHttpBinding binding = new BasicHttpBinding(BasicHttpSecurityMode.Transport);
binding.MaxReceivedMessageSize = int.MaxValue;
AWSECommerceServicePortTypeClient amazonClient = new AWSECommerceServicePortTypeClient(
binding,
new EndpointAddress("https://webservices.amazon.com/onca/soap?Service=AWSECommerceService"));
// add authentication to the ECS client
amazonClient.ChannelFactory.Endpoint.Behaviors.Add(new AmazonSigningEndpointBehavior(accessKeyId, secretKey));
The problem is two fold:
You are not binding the amazonClient to an HttpBinding
You are not signing the request
If my assumption is incorrect then you should download the code from the above link as it is a working example of how to call the Amazon Product API.

I believe your problem may be lack of an Associate Tag. As of November, 2011, this is required for all requests and I noticed early on in my testing that I got null responses back (with an error code) when I didn't include it. I'm not sure if that's still the behavior but I'd definitely assume that if you aren't adding it (which I don't see in your code) that's a likely suspect.
Look at top change note here
If you don't have an Associate ID you will need to apply for one.

how to get around amazon product advertising API request maximum

i am using the amazon API product description in order to get price lists for specific items in their database. however after getting a few thousand items i am getting:
System.ServiceModel.ServerTooBusyException was unhandled
Message="The HTTP service located at https://webservices.amazon.com/onca/soap?Service=AWSECommerceService is too busy. "
it looks like they do not want users to bother them too much with too many requests.
i will need to do probably around 1,000,000 requests per day.
i am wondering if there is any way to get over this limit?
here is how i am request my data:
// prepare the first ItemSearchRequest
// prepare a second ItemSearchRequest
ItemSearchRequest request1 = new ItemSearchRequest();
request1.SearchIndex = "All";
request1.Keywords = table.Rows[i].ItemArray[0].ToString();
request1.ItemPage = "1";
request1.ResponseGroup = new string[] { "OfferSummary" };
// batch the two requests together
ItemSearch itemSearch = new ItemSearch();
itemSearch.Request = new ItemSearchRequest[] { request1 };
itemSearch.AWSAccessKeyId = accessKeyId;
// issue the ItemSearch request
ItemSearchResponse response = client.ItemSearch(itemSearch);

You would contact Amazon and ask them. You don't try to "get around" it.
I suppose you could get a bunch of servers on different IPs. But just ask Amazon, it's the right thing to do.

Update Twitter Status in C#

I'm trying to update a user's Twitter status from my C# application.
I searched the web and found several possibilities, but I'm a bit confused by the recent (?) change in Twitter's authentication process. I also found what seems to be a relevant StackOverflow post, but it simply does not answer my question because it's ultra-specific regading a code snippet that does not work.
I'm attempting to reach the REST API and not the Search API, which means I should live up to the stricter OAuth authentication.
I looked at two solutions. The Twitterizer Framework worked fine, but it's an external DLL and I would rather use source code. Just as an example, the code using it is very clear and looks like so:
Twitter twitter = new Twitter("username", "password");
twitter.Status.Update("Hello World!");
I also examined Yedda's Twitter library, but this one failed on what I believe to be the authentication process, when trying basically the same code as above (Yedda expects the username and password in the status update itself but everything else is supposed to be the same).
Since I could not find a clear cut answer on the web, I'm bringing it to StackOverflow.
What's the simplest way to get a Twitter status update working in a C# application, without external DLL dependency?
Thanks

If you like the Twitterizer Framework but just don't like not having the source, why not download the source? (Or browse it if you just want to see what it's doing...)

I'm not a fan of re-inventing the wheel, especially when it comes to products that already exist that provide 100% of the sought functionality. I actually have the source code for Twitterizer running side by side my ASP.NET MVC application just so that I could make any necessary changes...
If you really don't want the DLL reference to exist, here is an example on how to code the updates in C#. Check this out from dreamincode.
/*
* A function to post an update to Twitter programmatically
* Author: Danny Battison
* Contact: gabehabe#hotmail.com
*/
/// <summary>
/// Post an update to a Twitter acount
/// </summary>
/// <param name="username">The username of the account</param>
/// <param name="password">The password of the account</param>
/// <param name="tweet">The status to post</param>
public static void PostTweet(string username, string password, string tweet)
{
try {
// encode the username/password
string user = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
// determine what we want to upload as a status
byte[] bytes = System.Text.Encoding.ASCII.GetBytes("status=" + tweet);
// connect with the update page
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://twitter.com/statuses/update.xml");
// set the method to POST
request.Method="POST";
request.ServicePoint.Expect100Continue = false; // thanks to argodev for this recent change!
// set the authorisation levels
request.Headers.Add("Authorization", "Basic " + user);
request.ContentType="application/x-www-form-urlencoded";
// set the length of the content
request.ContentLength = bytes.Length;
// set up the stream
Stream reqStream = request.GetRequestStream();
// write to the stream
reqStream.Write(bytes, 0, bytes.Length);
// close the stream
reqStream.Close();
} catch (Exception ex) {/* DO NOTHING */}
}

Another Twitter library I have used sucessfully is TweetSharp, which provides a fluent API.
The source code is available at Google code. Why don't you want to use a dll? That is by far the easiest way to include a library in a project.

The simplest way to post stuff to twitter is to use basic authentication , which isn't very strong.
static void PostTweet(string username, string password, string tweet)
{
// Create a webclient with the twitter account credentials, which will be used to set the HTTP header for basic authentication
WebClient client = new WebClient { Credentials = new NetworkCredential { UserName = username, Password = password } };
// Don't wait to receive a 100 Continue HTTP response from the server before sending out the message body
ServicePointManager.Expect100Continue = false;
// Construct the message body
byte[] messageBody = Encoding.ASCII.GetBytes("status=" + tweet);
// Send the HTTP headers and message body (a.k.a. Post the data)
client.UploadData("http://twitter.com/statuses/update.xml", messageBody);
}

Try LINQ To Twitter. Find LINQ To Twitter update status with media complete code example that works with Twitter REST API V1.1. Solution is also available for download.
LINQ To Twitter Code Sample
var twitterCtx = new TwitterContext(auth);
string status = "Testing TweetWithMedia #Linq2Twitter " +
DateTime.Now.ToString(CultureInfo.InvariantCulture);
const bool PossiblySensitive = false;
const decimal Latitude = StatusExtensions.NoCoordinate;
const decimal Longitude = StatusExtensions.NoCoordinate;
const bool DisplayCoordinates = false;
string ReplaceThisWithYourImageLocation = Server.MapPath("~/test.jpg");
var mediaItems =
new List<media>
{
new Media
{
Data = Utilities.GetFileBytes(ReplaceThisWithYourImageLocation),
FileName = "test.jpg",
ContentType = MediaContentType.Jpeg
}
};
Status tweet = twitterCtx.TweetWithMedia(
status, PossiblySensitive, Latitude, Longitude,
null, DisplayCoordinates, mediaItems, null);

Try TweetSharp . Find TweetSharp update status with media complete code example works with Twitter REST API V1.1. Solution is also available for download.
TweetSharp Code Sample
//if you want status update only uncomment the below line of code instead
//var result = tService.SendTweet(new SendTweetOptions { Status = Guid.NewGuid().ToString() });
Bitmap img = new Bitmap(Server.MapPath("~/test.jpg"));
if (img != null)
{
MemoryStream ms = new MemoryStream();
img.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
ms.Seek(0, SeekOrigin.Begin);
Dictionary<string, Stream> images = new Dictionary<string, Stream>{{"mypicture", ms}};
//Twitter compares status contents and rejects dublicated status messages.
//Therefore in order to create a unique message dynamically, a generic guid has been used
var result = tService.SendTweetWithMedia(new SendTweetWithMediaOptions { Status = Guid.NewGuid().ToString(), Images = images });
if (result != null && result.Id > 0)
{
Response.Redirect("https://twitter.com");
}
else
{
Response.Write("fails to update status");
}
}

Here's another solution with minimal code using the excellent AsyncOAuth Nuget package and Microsoft's HttpClient. This solution also assumes you're posting on your own behalf so you have your access token key/secret already, however even if you don't the flow is pretty easy (see AsyncOauth docs).
using System.Threading.Tasks;
using AsyncOAuth;
using System.Net.Http;
using System.Security.Cryptography;
public class TwitterClient
{
private readonly HttpClient _httpClient;
public TwitterClient()
{
// See AsyncOAuth docs (differs for WinRT)
OAuthUtility.ComputeHash = (key, buffer) =>
{
using (var hmac = new HMACSHA1(key))
{
return hmac.ComputeHash(buffer);
}
};
// Best to store secrets outside app (Azure Portal/etc.)
_httpClient = OAuthUtility.CreateOAuthClient(
AppSettings.TwitterAppId, AppSettings.TwitterAppSecret,
new AccessToken(AppSettings.TwitterAccessTokenKey, AppSettings.TwitterAccessTokenSecret));
}
public async Task UpdateStatus(string status)
{
try
{
var content = new FormUrlEncodedContent(new Dictionary<string, string>()
{
{"status", status}
});
var response = await _httpClient.PostAsync("https://api.twitter.com/1.1/statuses/update.json", content);
if (response.IsSuccessStatusCode)
{
// OK
}
else
{
// Not OK
}
}
catch (Exception ex)
{
// Log ex
}
}
}
This works on all platforms due to HttpClient's nature. I use this method myself on Windows Phone 7/8 for a completely different service.