We Keep Coding

ASP NET Core Twitter OAuth Request Token Issues

Background
I have a back end application that has a Twitter app setup and I can query and pull user tweet/post data. This is great, however, right now on the front end I don't have full Twitter integration setup. What I mean by this is that on the front end the user can enter any Twitter username and I want to know for sure that the Twitter username entered actually belongs to the user. With a Twitter application key you can pull public Twitter data for any twitter account which works well for large scale data ingestion and in my case proof of concept kind of work. At the point I am now, I need to have the assumption enforced in the back end that the data being analyzed for a particular Twitter screen name is also owned by the user of the account on my web application.
The proposed Twitter Solution
Here is a bunch of reference documentation I have been trying to follow.
https://developer.twitter.com/en/docs/basics/authentication/guides/log-in-with-twitter
https://developer.twitter.com/en/docs/basics/authentication/api-reference/request_token
https://oauth.net/core/1.0/#anchor9
https://oauth.net/core/1.0/#auth_step1
I have been trying to follow this and I have had different permutations to the code posted below (one without the callback URL as parameters, one with etc.) but at this point, not very different. I have not had any success and it's been more than a couple of days, which is killing me.
The code
This is my attempt to follow the OAuth specification proposed above in the documentation. Note that this is ASP.NET Core 2.2 + code. Also, this is the code for just Step 1 in the Twitter guide for OAuth authentication and authorization.
public async Task<string> GetUserOAuthRequestToken()
{
int timestamp = (Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;
string nonce = Convert.ToBase64String(Encoding.ASCII.GetBytes(timestamp.ToString()));
string consumerKey = twitterConfiguration.ConsumerKey;
string oAuthCallback = twitterConfiguration.OAuthCallback;
string requestString =
twitterConfiguration.EndpointUrl +
OAuthRequestTokenRoute;
string parameterString =
$"oauth_callback={WebUtility.UrlEncode(twitterConfiguration.OAuthCallback)}&" +
$"oauth_consumer_key={twitterConfiguration.ConsumerKey}&" +
$"oauth_nonce={nonce}&" +
$"oauth_signature_method=HMAC_SHA1&" +
$"oauth_timestamp={timestamp}" +
$"oauth_version=1.0";
string signatureBaseString =
"POST&" +
WebUtility.UrlEncode(requestString) +
"&" +
WebUtility.UrlEncode(parameterString);
string signingKey =
twitterConfiguration.ConsumerSecret +
"&" + twitterConfiguration.AccessTokenSecret;
byte[] signatureBaseStringBytes = Encoding.ASCII.GetBytes(signatureBaseString);
byte[] signingKeyBytes = Encoding.ASCII.GetBytes(signingKey);
HMACSHA1 hmacSha1 = new HMACSHA1(signingKeyBytes);
byte[] signature = hmacSha1.ComputeHash(signatureBaseStringBytes);
string authenticationHeaderValue =
$"oauth_nonce=\"{nonce}\", " +
$"oauth_callback=\"{WebUtility.UrlEncode(twitterConfiguration.OAuthCallback)}\", " +
$"oauth_signature_method=\"HMAC_SHA1\", " +
$"oauth_timestamp=\"{timestamp}\", " +
$"oauth_consumer_key=\"{twitterConfiguration.ConsumerKey}\", " +
$"oauth_signature=\"{Convert.ToBase64String(signature)}\", " +
$"oauth_version=\"1.0\"";
HttpRequestMessage request = new HttpRequestMessage();
request.Method = HttpMethod.Post;
request.RequestUri = new Uri(
baseUri: new Uri(twitterConfiguration.EndpointUrl),
relativeUri: OAuthRequestTokenRoute);
request.Content = new FormUrlEncodedContent(
new Dictionary<string, string>() {
{ "oauth_callback", twitterConfiguration.OAuthCallback }
});
request.Headers.Authorization = new AuthenticationHeaderValue("OAuth",
authenticationHeaderValue);
HttpResponseMessage httpResponseMessage = await httpClient.SendAsync(request);
if (httpResponseMessage.IsSuccessStatusCode)
{
return await httpResponseMessage.Content.ReadAsStringAsync();
}
else
{
return null;
}
}
Notes
I have tried to remove the callback URL from the parameters as well and that didn't work. I have tried all sort of slightly different permutations (urlencoded my signature, added the callback URL in the query string, removed it etc), but I have lost track at this point the one's I have tried and haven't (encodings, quotes etc.).
Ignore the fact that I am not serializing the response into a model yet as the goal is to first hit a success status code!
I have an integration test setup for this method as well and I keep getting 400 Bad Request with no additional information (which makes sense), but is absolutely not helping with debugging.
[Fact]
public async Task TwitterHttpClientTests_GetOAuthRequestToken_GetsToken()
{
var result = await twitterHttpClient.GetUserOAuthRequestToken();
Assert.NotNull(result);
}
As an aside I had some other questions as well:
Is there a way to verify a user's Twitter account without going
through the OAuth flow? The reason I ask this is because getting
through OAuth flow is proving to be difficult
Is it safe to do the first step of the Twitter login workflow on the back end and return the response to the front end? The response
would carry a sensitive token and token secret. (If I were to answer
this myself I would say you have to do it this way otherwise you
would have to hard code app secrets into front end configuration
which is worse). I ask this because this has been on my conscious
since I have started this and I'm worried a bit.
Is there an OAuth helper library for C# ASP.NET Core that can make this easier?

I solved this by writing unit tests and working through the Twitter documentation on Creating A Signature. Since that example provides keys and results, it's possible to verify that your code is correct.
Since you asked about libraries - I wrote LINQ to Twitter with the hope of helping others like myself with this difficult task.
In addition to to signature, the page navigation can be challenging as your code works through the OAuth flow. Please review the Twitter documentation on Obtaining user access tokens to understand this better. I've also documented this in the LINQ to Twitter Wiki on Securing your Applications. Here's how this will work with LINQ to Twitter:
First, I have an OAuthController with a Begin action to redirect a user to for kicking off the authentication process:
public async Task<ActionResult> Begin()
{
//var auth = new MvcSignInAuthorizer
var auth = new MvcAuthorizer
{
CredentialStore = new SessionStateCredentialStore(HttpContext.Session)
{
ConsumerKey = configuration["Twitter:ConsumerKey"],
ConsumerSecret = configuration["Twitter:ConsumerSecret"]
}
};
string twitterCallbackUrl = Request.GetDisplayUrl().Replace("Begin", "Complete");
return await auth.BeginAuthorizationAsync(new Uri(twitterCallbackUrl));
}
Notice that it's using an MvcSignInAuthorizer, passing in credentials via the CredentialStore property. If you were using your own raw code, you would be setting up the HTTP request with the Authorization header.
Next, notice that I'm modifying the current URL so that it will reference the same controller, but with the Complete endpoint. That is the oauth_callback that gets sent to Twitter authorization.
That process redirects the user to the Twitter web site, they authorize your app, and then it uses the oauth_callback to redirect the user back to your site. Here's how you handle that:
public async Task<ActionResult> Complete()
{
var auth = new MvcAuthorizer
{
CredentialStore = new SessionStateCredentialStore(HttpContext.Session)
};
await auth.CompleteAuthorizeAsync(new Uri(Request.GetDisplayUrl()));
// This is how you access credentials after authorization.
// The oauthToken and oauthTokenSecret do not expire.
// You can use the userID to associate the credentials with the user.
// You can save credentials any way you want - database,
// isolated storage, etc. - it's up to you.
// You can retrieve and load all 4 credentials on subsequent
// queries to avoid the need to re-authorize.
// When you've loaded all 4 credentials, LINQ to Twitter will let
// you make queries without re-authorizing.
//
//var credentials = auth.CredentialStore;
//string oauthToken = credentials.OAuthToken;
//string oauthTokenSecret = credentials.OAuthTokenSecret;
//string screenName = credentials.ScreenName;
//ulong userID = credentials.UserID;
//
return RedirectToAction("Index", "Home");
}
Again, you can see that I'm using MvcAuthorizer and completing the request. After completing the request, you'll be able to pull out the oauth_token and oauth_token_secret, as well as screen_name and user_id. You can save these artifacts and re-use them for all subsequent activity by this user, making their experience better because they don't have to log in every time you need to make a request.
On your question about verification, there is a Verify Credentials endpoint.
LINQ to Twitter has an ASP.NET Core Sample, API Samples with 100% API coverate, and full documentation on the Wiki if you want to learn more.

After hours and hours of going through the documentation I found the answer out. Turns out I missed some small details from the guides.
When making a request to oauth/request_token, when you sign the
request, you don't use the access token secret (for this specific request). Also, see the "Getting Signing Key" section of the signing a request guide and read the last few paragraphs. Therefore the signing key
does not have the access token secret
You must UrlEncode every single key and value. You must UrlEncode the authorization header as well.
I will post the updated code for you all here in case you need this in C#. Note that this code is not clean. You should separate OAuth functionality into some other class. This was my attempt to just get it to work.
public async Task<string> GetUserOAuthRequestToken()
{
int timestamp = (Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;
string nonce = Convert.ToBase64String(Encoding.ASCII.GetBytes(timestamp.ToString()));
string consumerKey = twitterConfiguration.ConsumerKey;
string oAuthCallback = twitterConfiguration.OAuthCallback;
string requestString =
twitterConfiguration.EndpointUrl +
OAuthRequestTokenRoute;
string parameterString =
$"oauth_callback={WebUtility.UrlEncode(twitterConfiguration.OAuthCallback)}&" +
$"oauth_consumer_key={WebUtility.UrlEncode(twitterConfiguration.ConsumerKey)}&" +
$"oauth_nonce={WebUtility.UrlEncode(nonce)}&" +
$"oauth_signature_method={WebUtility.UrlEncode(OAuthSigningAlgorithm)}&" +
$"oauth_timestamp={WebUtility.UrlEncode(timestamp.ToString())}&" +
$"oauth_version={WebUtility.UrlEncode("1.0")}";
string signatureBaseString =
"POST&" +
WebUtility.UrlEncode(requestString) +
"&" +
WebUtility.UrlEncode(parameterString);
string signingKey =
WebUtility.UrlEncode(twitterConfiguration.ConsumerSecret) +
"&";
byte[] signatureBaseStringBytes = Encoding.ASCII.GetBytes(signatureBaseString);
byte[] signingKeyBytes = Encoding.ASCII.GetBytes(signingKey);
HMACSHA1 hmacSha1 = new HMACSHA1(signingKeyBytes);
byte[] signature = hmacSha1.ComputeHash(signatureBaseStringBytes);
string base64Signature = Convert.ToBase64String(signature);
string authenticationHeaderValue =
$"oauth_nonce=\"{WebUtility.UrlEncode(nonce)}\", " +
$"oauth_callback=\"{WebUtility.UrlEncode(twitterConfiguration.OAuthCallback)}\", " +
$"oauth_signature_method=\"{WebUtility.UrlEncode(OAuthSigningAlgorithm)}\", " +
$"oauth_timestamp=\"{WebUtility.UrlEncode(timestamp.ToString())}\", " +
$"oauth_consumer_key=\"{WebUtility.UrlEncode(twitterConfiguration.ConsumerKey)}\", " +
$"oauth_signature=\"{WebUtility.UrlEncode(base64Signature)}\", " +
$"oauth_version=\"{WebUtility.UrlEncode("1.0")}\"";
HttpRequestMessage request = new HttpRequestMessage();
request.Method = HttpMethod.Post;
request.RequestUri = new Uri(
baseUri: new Uri(twitterConfiguration.EndpointUrl),
relativeUri: OAuthRequestTokenRoute);
request.Headers.Authorization = new AuthenticationHeaderValue("OAuth",
authenticationHeaderValue);
HttpResponseMessage httpResponseMessage = await httpClient.SendAsync(request);
if (httpResponseMessage.IsSuccessStatusCode)
{
string response = await httpResponseMessage.Content.ReadAsStringAsync();
return response;
}
else
{
return null;
}
}

Add subscriber to Mailchimp through API 3.0

I'm attempting to integrate version 3 of Mailchimp's API to add a subscriber to one of my mailing lists. The code below is what I have thus far, and my intention is to call it in my contact form method when the user fills out their email to subscribe. In theory it should get this email stored in emailAddress and POST it to MailChimp, but theorys are not practical. Below is my current method of POST data:
private string InsertIntoMailChimpGeneralList(string emailAddress)
{
var apiKey = ConfigurationManager.AppSettings["MailChimpAPIKeyGeneral"];
var dataCenter = ConfigurationManager.AppSettings["MailChimpDataCenterIDGeneral"];
var listId = ConfigurationManager.AppSettings["mailChimpListIDGeneral"];
var email_address = emailAddress;
var status = "subscribed";
using (var wc = new System.Net.WebClient())
{
// Data to be posted to add email address to list
var data = new { email_address, status };
// Serialize to JSON using Json.Net
var json = JsonConvert.SerializeObject(data);
// Base URL to MailChimp API
string apiUrl = "https://" + dataCenter + ".api.mailchimp.com/3.0/";
// Construct URL to API endpoint being used
var url = string.Concat(apiUrl, "lists/", listId, "/members?");
// Set content type
wc.Headers.Add("Content-Type", "application/json");
// Generate authorization header
string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(":" + apiKey));
// Set authorization header
wc.Headers[HttpRequestHeader.Authorization] = string.Format("Basic {0}", credentials);
// Post and get JSON response
string sendUrl = wc.UploadString(url, json);
return sendUrl;
}
}
In my contact form I have a check where I want the email address added in the contact form (emailAddress is the variable used here) to post that data to the list, when the form is submitted:
if (ConfigurationManager.AppSettings["UseMailChimpIntegration"].ToString().ToLower().Trim() == "true")
{
InsertIntoMailChimpGeneralList(emailAddress);
}
I feel I've implemented this wrong. I was able to get it working on v2 but upgrading to v3 has left me clueless at this point. My contact form runs fine and stored my values in my local database, but does not POST that data through to mailchimp.
I've triple checked my API/datacenter values, and would appreciate some assistance.

How do I update tokens in a JSON file using Rest API C#

Browserstack.com have a REST API that stores the results of Automated test sessions in a JSON file.
By default it sets the status token to "done" but you can update using the REST API e.g. "passed" or "failed".
It requires Basic authorization using your username & password.
They give an example of how to do this but it's a bit messy. So I wanted a simpler way of updating the file and only the tokens I was interested in.

I was able to do this using DalSoft's RestClient application:
public static async Task DalSoft(string SessionID, string TestStatus)
{
string Uri = "https://www.browserstack.com/automate/sessions/" + SessionID +
".json";
string AuthToken = "Basic " +
Convert.ToBase64String(Encoding.Default.GetBytes("username:password"));
dynamic client = new DalSoft.RestClient.RestClient(Uri);
var status = new { status=TestStatus };
var result = await client
.Headers(new { Authorization = AuthToken })
.Patch(status);
string Myresults = result.ToString();
}
https://github.com/DalSoft/DalSoft.RestClient/issues/40#issuecomment-334219145

Why do I keep getting 403 User does not have sufficient permission for this profile error from Analytics API

EDIT: Also I have read the following posts on Stack Overflow, but I don't think they have the solution I am looking for:
Google Analytics throws 403 error
Google Analytics API: "User does not have sufficient permissions for this account."
I am creating an installed application in C# to access and display my Google Analytics Data.
I have read Google's documentation for OAuth v2.0 and the Analytics v3 API, and I cannot retrieve my analytics data. Here is what I have done so far.
Navigate to the following URL in a Web Browser where I am prompted to log in to my Google Account (the account that owns the Analytics Account and has full ownership and permission) or if my browser has saved my login, an accept screen comes up asking me to confirm that I want to allow the app to access my analytics data. Here is the URL:
https://accounts.google.com/o/oauth2/auth?redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&client_id=XXXXXX.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fanalytics.readonly&approval_prompt=force&access_type=offline");
After the code is successfully returned and retrieved from the browser title window as the OAuth 2.0 documentation specifies for installed applications, I take this code and Create the following request which successfully returns an access token:
WebRequest request = WebRequest.Create("https://accounts.google.com/o/oauth2/token");
string body = String.Format("code={0}&client_id=XXXXXXXXXXX.apps.googleusercontent.com&client_secret=XXXXXXXXXXX&redirect_uri=urn:ietf:wg:oauth:2.0:oob&grant_type=authorization_code"
,browser.OAuthCode);
request.Method = "POST";
byte[] reqBytes = Encoding.UTF8.GetBytes(body);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = reqBytes.Length;
request.GetRequestStream();
Stream stream = request.GetRequestStream();
stream.Write(reqBytes, 0, (int)request.ContentLength);
WebResponse response = request.GetResponse();
Stream s = response.GetResponseStream();
StreamReader sr = new StreamReader(s);
string json = sr.ReadToEnd();
OAuthResponse tokenHolder = new OAuthResponse();
tokenHolder = Newtonsoft.Json.JsonConvert.DeserializeObject<OAuthResponse>(json);
return tokenHolder.AccessToken;
Finally, after successfully retrieving an access token, I create another Request to retrieve my analytics data like so:
public WebRequest ApiRequest()
{
string oAuthToken = OAuthToken();
//need to change this so people can select different ones
string idParam = "ids=ga:XXXXXX";
startDate = "start-date=" + startDate;
endDate = "end-date=" + endDate;
string totalEventsMetric = "ga:totalEvents";
string uniqueEventsMetric = "ga:uniqueEvents";
string categoryDimension = "ga:eventCategory";
string actionDimension = "ga:eventAction";
string labelDimension = "ga:eventLabel";
string parameters = "";
if ((bool)this._showTotalEvents.IsChecked)
parameters += "metrics=" + totalEventsMetric;
if ((bool)this._shwoUniqueEvents.IsChecked)
if (parameters != "")
parameters += "," + uniqueEventsMetric;
else
parameters += "metrics=" + uniqueEventsMetric;
if ((bool)this._showCategory.IsChecked)
if (parameters != "")
parameters += "&dimensions=" + categoryDimension;
else
parameters += "dimensions=" + categoryDimension;
if ((bool)this._showAction.IsChecked)
if (parameters != "" & parameters.Contains("dimensions"))
parameters += "," + actionDimension;
else if (parameters != "" & !parameters.Contains("dimensions"))
parameters += "&dimensions=" + actionDimension;
else
parameters += "dimensions=" + actionDimension;
if ((bool)this._showLabel.IsChecked)
if (parameters != "" & parameters.Contains("dimensions"))
parameters += "," + labelDimension;
else if (parameters != "" & !parameters.Contains("dimensions"))
parameters += "&dimensions=" + labelDimension;
else
parameters += "dimensions=" + labelDimension;
if (parameters != "")
{
parameters += "&" + idParam;
parameters += "&" + startDate;
parameters += "&" + endDate;
}
else
{
parameters += idParam;
parameters += "&" + startDate;
parameters += "&" + endDate;
parameters += "&metrics=" + totalEventsMetric;
parameters += "," + uniqueEventsMetric;
}
string url = string.Format("https://www.googleapis.com/analytics/v3/data/ga?{0}", parameters);
WebRequest request = WebRequest.Create(url);
request.Method = "GET";
request.Headers.Add("Authorization: Bearer " + oAuthToken);
return request;
}
My url ends up looking something like this:
https://www.googleapis.com/analytics/v3/data/ga?metrics=ga:totalEvents,ga:uniqueEvents&dimensions=ga:eventCategory,ga:eventAction,ga:eventLabel&ids=ga:XXXXX&start-date=2013-12-01&end-date=2014-01-01
And my Header:
{Authorization: Bearer oAuthTokenGoesHere}
And the error I get every time:
{"error":{"errors":[{"domain":"global","reason":"insufficientPermissions","message":"User does not have sufficient permissions for this profile."}],"code":403,"message":"User does not have sufficient permissions for this profile."}}
I cannot figure out why I am getting this error when this is an installed program. I log into the actual account in the web browser that opens up before I click accept and retrieve the OAuth code to exchange for a token. I have tried adding the App Engine and Compute Engine email address form the developer's console to my analytics account using the web interface, but this does not help. There is no email address associated with client ids for installed applications either, presumably because you have to log in in a browser before you can get a code.
I also tried passing the token in as a parameter instead of a header, but that did not work either.
I am not sure what to do from here.

I was providing the wrong id number to access my Analytics data. I was using the id contained in the table id (I am pretty sure this is what it is called) that looks like UA-XXXXXX-1 when this is the account id number. Eventually after going back and rereading all of the documentation I saw where it said to use the profile (view) Id.
Actually this post:
Google Analytics throws 403 error
mentioned that you need to make sure you are using the correct id, but the reason I did not think this referred to me was because it called the number the profile id, but when I was looking in Google Analytics, I could not find a profile id. In the analytics web interface, it is called the view id. I must have gotten lost in the sea of documentation and forgot about this part:
https://developers.google.com/analytics/devguides/reporting/core/v3/reference#ids
where it specifically says to use the "view (profile) id".

This is a simple error. You are trying to access a profile for which your Google Account has no authorisation. Kindly log in to the GA account, and navigate to the View/Profile you're trying to access, and go to Admin -> View -> User Management -> Add Permissions For:. This is the error which occurs only when you try to access a profile for which you don't have authorisation.
Also remember, sometimes you might have a few Google Accounts, only one of which has access to the Analytics profile in question (for example, home and work accounts). Sometimes when you're already logged in to a Google Account which does not have access to the Analytics profile in question, but you have logged in for the OAuth process using that account (which has no access) and given the C# application the authority to use those credentials, it will not ask you to log in. Say you're logged in to your home account which doesn't have access, and you use your C# application. During the OAuth authentication process, all it asks for now is whether you authorise this application to use your Google Account credentials. If you don't sign out of your home account and re-sign-in with your work account before this OAuth Authentication, then this error will occur infinitely, since your Account really does not have access to the profile in question at all.

Google Analytics API - Programmatically fetch page views in server side

We have a web application that consists of several pages. We registered our web app domain to Google Analytics and page views tracking works as expected (In the Analytics panel we can see page views for each page). Now we want this page views info to be stored in the back-end inside our DB. So we want to create a back-end process that will run once each day, and fetch the page views from Analytics API.
This is of course need to be done in code. From initial research it seems that in order to access Analytics API an authentication process must take place, meaning a human user must type in an id and password.
The question is, can it be done with code only ?

//-------------- Get Auth Token -------------------
WebClient webClient = new WebClient();
NameValueCollection data = new NameValueCollection();
data.Add("accountType", "GOOGLE");
data.Add("Email", "xxxx#gmail.com");
data.Add("Passwd", "xxxx");//Passwd, not a misspell.
data.Add("service", "analytics");
data.Add("source", "xxxx-xxxx-xx");//Could be anything.
byte[] bytes = webClient.UploadValues("https://www.google.com/accounts/ClientLogin", "POST", data);
string tokens = Encoding.UTF8.GetString(bytes);
string authToken = extractAuthToken(tokens);
//-------------- Get page views -------------------
string feed = "https://www.google.com/analytics/feeds/data";
//Required:
string ids = "ga:xxxx";
string metrics = "ga:pageviews";
string startDate = "2011-06-25";
string endDate = "2011-07-25";
//Optional:
string dimensions = "ga:pagePath";
string sort = "-ga:pageviews";
string feedUrl = string.Format("{0}?ids={1}&dimensions={2}&metrics={3}&sort={4}&start-date={5}&end-date={6}",
feed, ids, dimensions, metrics, sort, startDate, endDate);
webClient.Headers.Add("Authorization", "GoogleLogin " + authToken);
string result = webClient.DownloadString(feedUrl);
//-------------- Extract data from xml -------------------
XDocument xml = XDocument.Parse(result);
var ns1 = "{http://www.w3.org/2005/Atom}";
var ns2 = "{http://schemas.google.com/analytics/2009}";
var q = from entry in xml.Descendants()
where entry.Name == ns1 + "entry"
select new
{
PagePath = entry.Element(ns2 + "dimension").Attribute("value").Value,
Views = entry.Element(ns2 + "metric").Attribute("value").Value
};
//-------------- Do something with data -------------------
foreach (var page in q)
{
Debug.WriteLine(page.PagePath + " " + page.Views);
}
//-------------- Help Method -------------------
private string extractAuthToken(string data)
{
var tokens = data.Split(new string[] { "\n" }, StringSplitOptions.RemoveEmptyEntries);
return tokens.Where(token => token.StartsWith("Auth=")).Single();
}