We Keep Coding

Azure KeyPhrase API returning 400 at times

I'm getting mixed results with the Azure KeyPhrase API - sometimes successful (by that I mean 200 result) and others I'm getting 400 bad request. To test the service, I'm sending the contents from a Azure PDF on their NoSQL service.
The documentation says that each document may be upto 5k characters. So as to rule that out, (I started off with 5k) I'm limiting each to at most 1k characters.
How can I can get more info on what is the cause of the failure? I've already checked the Portal, but there's not much detail there.
I am using this endpoint: https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases
Some sample failures:
{"documents":[{"language":"en","id":1,"text":"David Chappell Understanding NoSQL on Microsoft Azure Sponsored by Microsoft Corporation Copyright © 2014 Chappell & Associates"}]}
{"documents":[{"language":"en","id":1,"text":"3 Relational technology has been the dominant approach to working with data for decades. Typically accessed using Structured Query Language (SQL), relational databases are incredibly useful. And as their popularity suggests, they can be applied in many different situations. But relational technology isn’t always the best approach. Suppose you need to work with very large amounts of data, for example, too much to store on a single machine. Scaling relational technology to work effectively across many servers (physical or virtual) can be challenging. Or suppose your application works with data that’s not a natural fit for relational systems, such as JavaScript Object Notation (JSON) documents. Shoehorning the data into relational tables is possible, but a storage technology expressly designed to work with this kind of information might be simpler. NoSQL technologies have been created to address problems like these. As the name suggests, the label encompasses a variety of storage"}]}
** added my quick/dirty poc code ***
List<string> sendRequest(object data)
{
string url = "https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases";
string key = "api-code-here";
string hdr = "Ocp-Apim-Subscription-Key";
var wc = new WebClient();
wc.Headers.Add(hdr, key);
wc.Headers.Add(HttpRequestHeader.ContentType, "application/json");
TextAnalyticsResult results = null;
string json = JsonConvert.SerializeObject(data);
try
{
var bytes = Encoding.Default.GetBytes(json);
var d2 = wc.UploadData(url, bytes);
var dataString = Encoding.Default.GetString(d2);
results = JsonConvert.DeserializeObject<TextAnalyticsResult>(dataString);
}
catch (Exception ex)
{
var s = ex.Message;
}
System.Threading.Thread.Sleep(125);
if (results != null && results.documents != null)
return results.documents.SelectMany(x => x.keyPhrases).ToList();
else
return new List<string>();
}
Called by:
foreach (var k in vals)
{
data.documents.Clear();
int countSpaces = k.Count(Char.IsWhiteSpace);
if (countSpaces > 3)
{
if (k.Length > maxLen)
{
var v = k;
while (v.Length > maxLen)
{
var tmp = v.Substring(0, maxLen);
var idx = tmp.LastIndexOf(" ");
tmp = tmp.Substring(0, idx).Trim();
data.documents.Add(new
{
language = "en",
id = data.documents.Count() + 1,
text = tmp
});
v = v.Substring(idx + 1).Trim();
phrases.AddRange(sendRequest(data));
data.documents.Clear();
}
data.documents.Add(new
{
language = "en",
id = data.documents.Count() + 1,
text = v
});
phrases.AddRange(sendRequest(data));
data.documents.Clear();
}
else
{
data.documents.Add(new
{
language = "en",
id = 1,
text = k
});
phrases.AddRange(sendRequest(data));
data.documents.Clear();
};
}
}

I manually created some requests using the document samples that you indicated had errors and they were processed by the service correctly and returned key phrases. So an encoding issue looks likely.
In the future, you can also look at the inner error returned by the service. Usually you'll see some more details like in the response sample below.
{
"code": "BadRequest",
"message": "Invalid request",
"innerError": {
"code": "InvalidRequestContent",
"message": "Request contains duplicated Ids. Make sure each document has a unique Id."
}
}
Also, there is a .NET SDK for Text Analytics that can help simplify calling the service.
https://github.com/Azure/azure-rest-api-specs/tree/current/specification/cognitiveservices/data-plane/TextAnalytics

Try changing this line
var bytes = Encoding.Default.GetBytes(json);
to
var bytes = Encoding.UTF8.GetBytes(json);

How to Check a input text is Norwegian language or not in c# [duplicate]

What's the best way to detect the language of a string?

If the context of your code have internet access, you can try to use the Google API for language detection.
http://code.google.com/apis/ajaxlanguage/documentation/
var text = "¿Dónde está el baño?";
google.language.detect(text, function(result) {
if (!result.error) {
var language = 'unknown';
for (l in google.language.Languages) {
if (google.language.Languages[l] == result.language) {
language = l;
break;
}
}
var container = document.getElementById("detection");
container.innerHTML = text + " is: " + language + "";
}
});
And, since you are using c#, take a look at this article on how to call the API from c#.
UPDATE:
That c# link is gone, here's a cached copy of the core of it:
string s = TextBoxTranslateEnglishToHebrew.Text;
string key = "YOUR GOOGLE AJAX API KEY";
GoogleLangaugeDetector detector =
new GoogleLangaugeDetector(s, VERSION.ONE_POINT_ZERO, key);
GoogleTranslator gTranslator = new GoogleTranslator(s, VERSION.ONE_POINT_ZERO,
detector.LanguageDetected.Equals("iw") ? LANGUAGE.HEBREW : LANGUAGE.ENGLISH,
detector.LanguageDetected.Equals("iw") ? LANGUAGE.ENGLISH : LANGUAGE.HEBREW,
key);
TextBoxTranslation.Text = gTranslator.Translation;
Basically, you need to create a URI and send it to Google that looks like:
http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=hello%20worled&langpair=en%7ciw&key=your_google_api_key_goes_here
This tells the API that you want to translate "hello world" from English to Hebrew, to which Google's JSON response would look like:
{"responseData": {"translatedText":"שלום העולם"}, "responseDetails": null, "responseStatus": 200}
I chose to make a base class that represents a typical Google JSON response:
[Serializable]
public class JSONResponse
{
public string responseDetails = null;
public string responseStatus = null;
}
Then, a Translation object that inherits from this class:
[Serializable]
public class Translation: JSONResponse
{
public TranslationResponseData responseData =
new TranslationResponseData();
}
This Translation class has a TranslationResponseData object that looks like this:
[Serializable]
public class TranslationResponseData
{
public string translatedText;
}
Finally, we can make the GoogleTranslator class:
using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.Net;
using System.IO;
using System.Runtime.Serialization.Json;
namespace GoogleTranslationAPI
{
public class GoogleTranslator
{
private string _q = "";
private string _v = "";
private string _key = "";
private string _langPair = "";
private string _requestUrl = "";
private string _translation = "";
public GoogleTranslator(string queryTerm, VERSION version, LANGUAGE languageFrom,
LANGUAGE languageTo, string key)
{
_q = HttpUtility.UrlPathEncode(queryTerm);
_v = HttpUtility.UrlEncode(EnumStringUtil.GetStringValue(version));
_langPair =
HttpUtility.UrlEncode(EnumStringUtil.GetStringValue(languageFrom) +
"|" + EnumStringUtil.GetStringValue(languageTo));
_key = HttpUtility.UrlEncode(key);
string encodedRequestUrlFragment =
string.Format("?v={0}&q={1}&langpair={2}&key={3}",
_v, _q, _langPair, _key);
_requestUrl = EnumStringUtil.GetStringValue(BASEURL.TRANSLATE) + encodedRequestUrlFragment;
GetTranslation();
}
public string Translation
{
get { return _translation; }
private set { _translation = value; }
}
private void GetTranslation()
{
try
{
WebRequest request = WebRequest.Create(_requestUrl);
WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadLine();
using (MemoryStream ms = new MemoryStream(Encoding.Unicode.GetBytes(json)))
{
DataContractJsonSerializer ser =
new DataContractJsonSerializer(typeof(Translation));
Translation translation = ser.ReadObject(ms) as Translation;
_translation = translation.responseData.translatedText;
}
}
catch (Exception) { }
}
}
}

Fast answer: NTextCat (NuGet, Online Demo)
Long answer:
Currently the best way seems to use classifiers trained to classify piece of text into one (or more) of languages from predefined set.
There is a Perl tool called TextCat. It has language models for 74 most popular languages. There is a huge number of ports of this tool into different programming languages.
There were no ports in .Net. So I have written one: NTextCat on GitHub.
It is pure .NET Framework DLL + command line interface to it. By default, it uses a profile of 14 languages.
Any feedback is very appreciated!
New ideas and feature requests are welcomed too :)
Alternative is to use numerous online services (e.g. one from Google mentioned, detectlanguage.com, langid.net, etc.).

A statistical approach using digraphs or trigraphs is a very good indicator. For example, here are the most common digraphs in English in order: http://www.letterfrequency.org/#digraph-frequency (one can find better or more complete lists). This method may have a better success rate than word analysis for short snippets of text because there are more digraphs in text than there are complete words.

If you mean the natural (ie human) language, this is in general a Hard Problem. What language is "server" - English or Turkish? What language is "chat" - English or French? What language is "uno" - Italian or Spanish (or Latin!) ?
Without paying attention to context, and doing some hard natural language processing (<----- this is the phrase to google for) you haven't got a chance.
You might enjoy a look at Frengly - it's a nice UI onto the Google Translate service which attempts to guess the language of the input text...

Make a statistical analyses of the string: Split the string into words. Get a dictionary for every language you want to test for. And then find the language that has the highest word count.
In C# every string in memory will be unicode, and is not encoded. Also in text files the encoding is not stored. (Sometimes only an indication of 8-bit or 16-bit).
If you want to make a distinction between two languages, you might find some simple tricks. For example if you want to recognize English from Dutch, the string that contains the "y" is mostly English. (Unreliable but fast).

CLD3 (Compact Language Detector v3) library from Google's Chromium browser
You could wrap the CLD3 library, which is written in C++.

We can use Regex.IsMatch(text, "[\\uxxxx-\\uxxxx]+") to detect an specific language. Here xxxx is the 4 digit Unicode id of a character.
To detect Arabic:
bool isArabic = Regex.IsMatch(yourtext, #"[\u0600-\u06FF]+")

You may use the C# package for language identification from Microsoft Research:
This package implements several algorithms for language
identification, and includes two sets of pre-compiled language
profiles. One set covers 52 languages and was trained on Wikipedia
(i.e. a well-written corpus); the other covers 26 languages and was
constructed from Twitter (i.e. a highly colloquial corpus). The
language identifiers are packaged up as a C# library, and be easily
embedded into other C# projects.
Download the package from the above link.

One alternative is to use 'Translator Text API' which is
... part of the Azure Cognitive Services API collection of machine
learning and AI algorithms in the cloud, and is readily consumable in
your development projects
Here's a quickstart guide on how to detect language from text using this API

C# check if a string is a valid WebProxy format

so after searching both on here and Google I have been unable to find a solution. Basically I want to allow the user to add a list of proxies from a text file, but I want to check that what gets passed in is a valid Proxy format before I make the WebProxy. I know using try catch like try {var proxy = new WebProxy(host + ":" + port;} catch{} will work, but as you may already know using try catch is slow, even more so when doing it in bulk.
So what would be a good and fast way to test a string to see if it's in a valid WebProxy format?
Thanks

This is how I do it:
var ValidIpAddressPortRegex = #"^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]):[\d]+$";
if (System.Text.RegularExpressions.Regex.IsMatch(proxy_string_to_validate, ValidIpAddressPortRegex ))
{
do_stuff_with_proxy();
}
else
{
//MessageBox.Show("IP:PORT url not valid!","Information");
}

How to work with the BING REST Api

How exactly do you use the BING REST api (specifically the ROUTES part) to get a driving distance in ASP.NET.
I have searched high and low on Google for this answer and none is forthcoming.
I have found url strings such as:
http://dev.virtualearth.net/REST/v1/Routes/Driving?waypoint.0=redmond&heading=90&waypoint.1=seattle&du=mi&key=BingMapsKey
That's great! But how to call it from ASP?
I have also found this code:
private void GetResponse(Uri uri, Action<HttpResponse> callback)
{
WebClient wc = new WebClient();
wc.OpenReadCompleted += (o, a) =>
{
if (callback != null)
{
DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(HttpResponse));
callback(ser.ReadObject(a.Result) as HttpResponse);
}
};
wc.OpenReadAsync(uri);
}
Which is a "generic method to make web requests". But, again, how do you call it? I find it confusing that it doesn't require a return type.
In order to call it, I have found code like this:
string key = "YOUR_BING_MAPS_KEY or SESSION_KEY";
string query = "1 Microsoft Way, Redmond, WA";
Uri geocodeRequest = new Uri(string.Format("http://dev.virtualearth.net/REST/v1/Locations?q={0}&key={1}", query, key));
GetResponse(geocodeRequest, (x) =>
{
Console.WriteLine(x.ResourceSets[0].Resources.Length + " result(s) found.");
Console.ReadLine();
});
But when I add this to the project, I get every error under the sun coming up. So, I am stuck.
I am a total ASP beginner and haven't found any online documentation any help at all.
p.s. I do have a BING api key and do use it in the code above.

I am not an expert in this, but the below compiles for me. Also make sure to add the data constructs as mentioned in the BING documentation:
protected void Page_Load(object sender, EventArgs e)
{
string key = "YOUR KEY";
string query = "ADDRESS";
Uri geocodeRequest = new Uri(string.Format("http://dev.virtualearth.net/REST/v1/Locations?q={0}&key={1}", query, key));
GetResponse(geocodeRequest, (x) =>
{
Console.WriteLine(x.ResourceSets[0].Resources.Length + " result(s) found.");
Console.ReadLine();
});
}
Quoting from another stackoverflow question:
The bottom of the documentation you are using points to the Data contracts you need for the REST services which are available here: http://msdn.microsoft.com/en-us/library/jj870778.aspx
Simply create a empty C# file and copy in paste in the C# Data Contracts. Then add the namespace to this class:
using BingMapsRESTService.Common.JSON;

Google Analytics API in C# -Execution of request failed: https://www.google.com/analytics/feeds/accounts/default

i want to access Google analytic data and i got samples from Google data API SDK. but these coding does not working and throws exception
Execution of request failed: https://www.google.com/analytics/feeds/accounts/default
so i found the reason for this is Google updated it's to v3.0. i searched updated coding for the C#, but i couldn't find solution for this.
i have same problem as this, but with C#.
Exception thrown when using GData .NET Analytics API
i tried coding with doing changes as follows as it says in Google developer - https://developers.google.com/analytics/resources/articles/gdata-migration-guide#appendix_a
string userName = this.Username.Text;
string passWord = this.Password.Text;
AnalyticsService service = new AnalyticsService("AnalyticsSampleApp");
service.setUserCredentials(userName, passWord);
string googleAccountWebId = "AIXXXXXXXXXXXXXXXXXXXXXXXXXXX";
string profileFeedUrl = "https://www.googleapis.com/analytics/v2.4/data?key=" + googleAccountWebId;
DataQuery query2 = new DataQuery(profileFeedUrl);
query2.Ids = "12345678";
query2.Metrics = "ga:visits";
query2.Sort = "ga:visits";
query2.GAStartDate = DateTime.Now.AddMonths(-1).AddDays(-2).ToString("2011-08-01");
query2.GAEndDate = DateTime.Now.ToString("2013-09-01");
query2.StartIndex = 1;
DataFeed data = service.Query(query2);
foreach (DataEntry entry in data.Entries)
{
string st=entry.Metrics[0].Value;
}
but even i change this it throws exception in
DataFeed data = service.Query(query2);
this line. exception is as follows:
Execution of request failed: https://www.googleapis.com/analytics/v2.4/data?key=AIXXXXXXXXXXXXXXXXXXXXXX-8&start-index=1&end-date=2013-09-01&ids=12345678&metrics=ga:visits&sort=ga:visits&start-date=2011-08-01
i'm using following DLL
Google.GData.Analytics.dll
Google.GData.Client.dll
Google.GData.Extensions.dll
My Questions :
how can i correct this error?
how can i access Google analytic data? is this correct? or else what is the way to doing it??
for a example i want to get available ProfileId and their values. (Title and Page views)

Analytics Account:
I am assuming you have an analytics account already if you don't then create one, and sign up your domain here:
http://www.google.com/intl/en/analytics/
To get your API Key do this:
Follow the instructions on https://developers.google.com/analytics/resources/articles/gdata-migration-guide (Create a Project in the Google APIs Console) to generate your key Once you have it set it as part of the querystring to request to Google Analytics service, in this case:
YourAPIkEStringabcdefghijklmno
To get the profileId (Ids on the code) you should do this:
Log into your analytics account, select the desired domain on your list (blue link) click on the administrator button and on the profiles tab find the profile
configuration subtab, right there you will find the profile id in this case the eight characters long id:
12345678
Here you have some C# code to help you getting the number of visits for that Id:
public string VisitsNumber()
{
string visits = string.Empty;
string username = "youremailuser#domain.com";
string pass = "yourpassword";
string gkey = "?key=YourAPIkEYYourAPIkEYYourAPIkEYYourAPIkE";
string dataFeedUrl = "https://www.google.com/analytics/feeds/data" + gkey;
string accountFeedUrl = "https://www.googleapis.com/analytics/v2.4/management/accounts" + gkey;
AnalyticsService service = new AnalyticsService("WebApp");
service.setUserCredentials(username, pass);
DataQuery query1 = new DataQuery(dataFeedUrl);
query1.Ids = "ga:12345678";
query1.Metrics = "ga:visits";
query1.Sort = "ga:visits";
//You were setting 2013-09-01 and thats an invalid date because it hasn't been reached yet, be sure you set valid dates
//For start date is better to place an aprox date when you registered the domain on Google Analytics for example January 2nd 2012, for an end date the actual date is enough, no need to go further
query1.GAStartDate = new DateTime(2012, 1, 2).ToString("yyyy-MM-dd");
query1.GAEndDate = DateTime.Now.ToString("yyyy-MM-dd");
query1.StartIndex = 1;
DataFeed dataFeedVisits = service.Query(query1);
foreach (DataEntry entry in dataFeedVisits.Entries)
{
string st = entry.Title.Text;
string ss = entry.Metrics[0].Value;
visits = ss;
}
return visits;
}
protected void Page_Load(object sender, EventArgs e)
{
if (!Page.IsPostBack)
{
Response.Write("Visits:" + this.VisitsNumber());
}
}
Since the 2.4 API is not so flexible anymore, I have another post here hacking it to get the profile Id:
Getting an specific ProfileId from registered Accounts using GData .NET Analytics API 2.4 if you need to convert the code to C# you can use the Telerik converter: http://converter.telerik.com/
I think this suffice to use the 2.4 API. If you need extra help let me know.