Write JSON to Solr via HTTP/Post - c#

I am writing to Solr using JSON via HTTP/POST from a C#/Winforms/.NET4.0 app to speed up indexing and using the code below. I write a document to solr (based on these instructions) but keep getting a '400 bad request'. The JSON appears to be clean and no issues.
It seems to be a syntax issue but I have been wrestling with this for last many hours to no avail. Any ideas on what is awry? All help appreciated.
Here is the URI string being posted
"http://localhost:8080/solr/update/json -H 'Content-type:application/json' -d ' [ {\"UUID\":\"d2a174e4-81d6-487f-b68d-392be5d3d47a\",\"Extension\":\".AVI\",\"VideoFileName\":\"Clip 1.avi\"} ' ]"
string uri = http://localhost:8080/solr/update/json;
public bool WriteJSONToSolr(string uri, string json)
{
WebRequest request = WebRequest.Create(uri + " -H 'Content-type:application/json' -d ' [ " + json + " ' ]" );
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(json);
Stream stream = null;
try
{ // send the Post
request.ContentLength = bytes.Length; //Count bytes to send
stream = request.GetRequestStream();
stream.Write(bytes, 0, bytes.Length); //Send it
}
catch
{
return false;
}
finally
{
if (stream != null)
{
stream.Close();
}
}
System.Net.WebResponse response = request.GetResponse();
if (response == null) return false;
return true;
}

If you are inserting then you will need to use add and doc in your json element. You'll also want to add a commit so that the index is updated. You can also remove the parameters from your uri as you can add them in the web request object. Lastly, you should have your collection name in the uri.
string json = "{\"add\":{\"doc\":{"
+ "\"UUID\":\"d2a174e4-81d6-487f-b68d-392be5d3d47a\","
+ "\"Extension\":\".AVI\","
+ "\"VideoFileName\":\"Clip 1.avi\"}},";
+ "\"commit\":{}}";
string uri = "http://localhost:8080/solr/collection/update";
WebRequest request = WebRequest.Create(uri);
request.ContentType = "application/json";
request.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(json);
Stream stream = null;
try {
request.ContentLength = bytes.Length;
stream = request.GetRequestStream();
stream.Write(bytes, 0, bytes.Length);
}
catch {
return;
}
finally {
if (stream != null) {
stream.Close();
}
}
System.Net.WebResponse response = request.GetResponse();
if (response == null) {
return;
}
If you are inserting multiple objects to solr you can add multiple add objects or doc objects to the json. For example...
json = "{add:{doc:{keys:values}}, add:{doc:{keys:values}}, commit:{}}"
or
json = "{add:{doc:{keys:values}, doc:{keys:values}}, commit:{}}"
While your debugging this watch the log for solr. It will alert you to any problems with that may be happening on the solr side.

First of all, I don't think you are passing a correct json data with -d option.. Look at the following formatting code in your code.
" -H 'Content-type:application/json' -d ' [ " + json + " ' ]"
Assume, your json data is {"name":"sam"} Then, the above formatting results to
-H 'Content-type:application/json' -d ' [{"name":"sam"} ' ]
You are passing a json data with missing ].
Apart from that, your approach for updating a document in solr index is wrong. Take a look at the following simple code.
[BTW: You may pass the 'commit' argument in the url].
public async Task PostAsync()
{
string json = "{\"add\": {\"doc\": {"
+ "\"id\":\"12345\","
+ "\"firstname\":\"Sam\","
+ "\"lastname\":\"Wills\","
+ "\"dob\":\"2016-12-14T00:00:00Z\""
+ "}}}";
using (var client = new HttpClient())
{
string uri = "http://localhost:8983/solr/people/update/json?wt=json&commit=true";
var jsonContent = new StringContent(json);
await client.PostAsync(new Uri(uri), jsonContent);
}
}
If you want to update a specific field instead of whole document [partial update], use the following code snippet.
public async Task PartialPostAsync()
{
string json = "{\"add\": {\"doc\": {"
+ "\"id\":\"12345\","
+ "\"lastname\":{\"set\":\"George\"}"
+ "}}}";
using (var client = new HttpClient())
{
string uri = "http://localhost:8983/solr/people/update/json?wt=json&commit=true";
var jsonContent = new StringContent(json);
await client.PostAsync(new Uri(uri), jsonContent);
}
}
'id' field is an unique field.

did you forget putting a space char before -H?

The reasong that your code isn't working, is because you are using cURL syntax in .Net.
cURL is an executable that sends and receives HTTP requests and .Net is a framework for programming applications.
They are not the same.
To make it work with .Net you first should post to the correct uri, and you need to set the correct ContentType property like such:
var uri = "http://localhost:8080/solr/update/json";
using (var r = WebRequest.Create(uri))
{
r.ContentType = "application/json";
r.Method = "POST";
using (var rs = r.GetRequestStream())
rs.Write // your data
// get response
// return response data
}
That said, why inflict pain upon yourself? Just use a SolR connector that already has a typed API for SolR operations!
https://code.google.com/p/solrnet/ for example!
But if you don't want to use that, then at least use a modern HTTP API like https://nuget.org/packages/RestSharp

I ran into this same problem today.
Try these two things
1) Keep in mind that you can NOT send a json string like
[{"A":"1","B":"0","C":"","D":"Washington"}]
instead you may have to massage the json to be more like
[{"A":"1","B":"0","D":"Washington"}]
Solr does not like empty values.
2) This second trick that helps (when sending data to solr via 'curl') : Try replacing all Double Quotes in your json string with two Double Quotes before you send the request to solr.
json = json.Replace(#"""", #"""""");

Please try below code after convert json to stream bytes
protected override void Append(LoggingEvent loggingEvent)
{
byte[] bodyBytes;
try
{
string body = BodyFormatter.CreateBody(loggingEvent, _parameters);
bodyBytes = Encoding.UTF8.GetBytes(body);
}
catch (Exception e)
{
ErrorHandler.Error("Failed to create body", e);
return;
}
HttpWebRequest request = BuildRequest();
request.BeginGetRequestStream(r =>
{
try
{
using (Stream stream = request.EndGetRequestStream(r))
{
stream.BeginWrite(bodyBytes, 0, bodyBytes.Length, c =>
{
try
{
stream.EndWrite(c);
request.BeginGetResponse(a =>
{
try
{
var response = request.EndGetResponse(a);
if (((HttpWebResponse)response).StatusCode != HttpStatusCode.OK)
ErrorHandler.Error("Got failed response: " + ((HttpWebResponse)response).StatusDescription);
response.Close();
}
catch (Exception e)
{
ErrorHandler.Error("Failed to get response", e);
}
}, null);
}
catch (Exception e)
{
ErrorHandler.Error("Failed to write", e);
}
}, null);
}
}
catch (Exception e)
{
ErrorHandler.Error("Failed to connect", e);
}
}, null);
}

Related

POST or Get Request with Json response in C# windows Form App

I made an API in server side with PHP + Laravel framework which accept both GET & Post requests with some special parameters.
It's address is : http://beresun.ir/API/Orders/0
and it gets these parameters :
token > string ,
restaurant_id > integer ,
admin_id > integer ,
token_id > integer .
if we send a request with GET method with these parameters, for example it will be :
http://beresun.ir/API/Orders/0?token=2JEuksuv86DcFmLrQa7nna4QDeowuGTqpyUK0pf9wSlbe6D5hLtEVxvzMT5gAZG0xBKy00HxS3J79mcr8F54dBD0uIg5HX5fzPOAP&restaurant_id=1&admin_id=2&token_id=40 which returns a json data , you can click on the link to see the results .
the response json data includes some information about customers and it's products.
now I want to make a windows application for this service with C# and request data from this API with POST or GET methods :
I want to use this API to get Json data from Web server and save them in my Windows application , So I created two functions in one of my Form Classes :
using Newtonsoft.Json;
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Windows.Forms;
public partial class MainActivity : Form
{
string token = "2JEuksuv86DcFmLrQa7nna4QDeowuGTqpyUK0pf9wSlbe6D5hLtEVxvzMT5gAZG0xBKy00HxS3J79mcr8F54dBD0uIg5HX5fzPOAP";
int restaurant_id = 1;
int admin_id = 2;
int token_id = 40;
private void SendWebrequest_Get_Method()
{
try
{
String postData = "token=" + token +
"&restaurant_id=" + restaurant_id +
"&admin_id=" + admin_id +
"&token_id=" + token_id;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://beresun.ir/API/Orders/0?" + postData);
request.ContentType = "application/json; charset=utf-8";
request.Accept = "application/json";
request.Method = WebRequestMethods.Http.Get;
WebResponse response = request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
String json_text = sr.ReadToEnd();
dynamic stuff = JsonConvert.DeserializeObject(json_text);
if (stuff.error != null)
{
MessageBox.Show("problem with getting data", "Error");
}
else
{
MessageBox.Show(json_text, "success");
}
sr.Close();
}
catch (Exception ex)
{
MessageBox.Show("Wrong request ! " + ex.Message, "Error");
}
}
private void SendWebrequest_POST_Method()
{
try
{
// Create a request using a URL that can receive a post.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://beresun.ir/API/Orders/5");
// Set the Method property of the request to POST.
request.Method = "POST";
request.Credentials = CredentialCache.DefaultCredentials;
((HttpWebRequest)request).UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
// Create POST data and convert it to a byte array.
string postData = "token=" + token +
"&restaurant_id=" + restaurant_id +
"&admin_id=" + admin_id +
"&token_id=" + token_id;
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/json; charset=utf-8";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
WebResponse response = request.GetResponse();
// Display the status.
MessageBox.Show(((HttpWebResponse)response).StatusDescription);
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
// Display the content.
MessageBox.Show(responseFromServer);
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
}
catch (Exception ex)
{
MessageBox.Show("Wrong request ! " + ex.Message, "Error");
}
}
}
Now Here is the problem , when I test the API it works fine , but when I request data from my application , it returns error and not working .
Can anyone explain me how I should request data from this API , to get data , I searched a lot , and I used many different methods , but none of them worked for me . maybe because this API returns very much Json data or maybe request timeout happen. I don't know , I couldn't find the problem . So I asked it Here.
I don't know what I should do .
Thanks
Oki so i run your code :
private string TestURL = "http://beresun.ir/API/";
string token = "2JEuksuv86DcFmLrQa7nna4QDeowuGTqpyUK0pf9wSlbe6D5hLtEVxvzMT5gAZG0xBKy00HxS3J79mcr8F54dBD0uIg5HX5fzPOAP";
int restaurant_id = 1;
int admin_id = 2;
int token_id = 40;
public async Task<string> test()
{
try
{
using (var Client = new HttpClient())
{
Client.BaseAddress = new Uri(TestURL);
Client.DefaultRequestHeaders.Accept.Clear();
Client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
string postData = "token=" + token +
"&restaurant_id=" + restaurant_id +
"&admin_id=" + admin_id +
"&token_id=" + token_id;
HttpResponseMessage responce = await Client.GetAsync("Orders/0?" + postData);
if (responce.IsSuccessStatusCode)
{
var Json = await responce.Content.ReadAsStringAsync();
// !
return Json;
}
else
{
// deal with error or here ...
return null;
}
}
}
catch (Exception e)
{
return null;
}
}
and its working am getting the json file ,, i think your mistake is in postData is string Not String ! a simple type can amazing harm !
try this:
private string URL = "Your Base domain URL";
public async Task<YourModel> getRequest()
{
using (var Client = new HttpClient())
{
Client.BaseAddress = new Uri(URL);
Client.DefaultRequestHeaders.Accept.Clear();
Client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage responce = await Client.GetAsync("Your Method or the API you callig");
if (responce.IsSuccessStatusCode)
{
var Json = await responce.Content.ReadAsStringAsync();
var Items= JsonConvert.DeserializeObject<YourModel>(Json);
// now use you have the date on Items !
return Items;
}
else
{
// deal with error or here ...
return null;
}
}
}

Posting a json to a distant server(REST)

I want to send some json data using this code to a distant server (Rest, outside my control), following the way I'm sending it :
I create the url and the request method first :
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(gUrlDot);
request.Method = "POST";
Dictionary<String, String> lDictionary = new Dictionary<String, String>();
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
Dot lDot= new Dot();
serviceContext lserviceContext = new serviceContext();
items litems = new items();
lDot.rad = pClient;
lDictionary.Add("companyId", "00230");
lDictionary.Add("treatmentDate", lGlobals.FormatDateYYYYMMDD());
lDictionary.Add("country", "FR");
lDictionary.Add("language", "fr");
lDictionary.Add("Id", "test");
litems.contextItem = lDictionary;
lDot.serviceContext = lserviceContext;
lDot.serviceContext.items = litems;
String Input=_Tools.JsonSerializer(lDot);
log.Debug("Input Dot " + Input);
Byte[] byteArray = encoding.GetBytes(Input);
request.ContentLength = byteArray.Length;
request.ContentType = #"application/json";
using (Stream dataStream = request.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
}
long length = 0;
When i'm getting here it crashes with an execption : Error 500 !
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
length = response.ContentLength;
string output = response.ToString();
lTrace.AppendLine("-Flux Json recu => " + response.StatusCode + " " + length);
log.Debug("Output Dot " + output);
}
log.Info(LogsHelper.LogHeader("End processing Get list ", pClient, Service.SrvGetList, "", lResponse.StatusCode, lResponse.StatusLabel, lResponse.ResponseObject, ref lTrace));
}
catch (Exception ex)
{
lResponse.StatusCode = StatusCodes.ERROR_COMMUNICATION;
log.Error(LogsHelper.LogHeader("End processing Get list", pClient, Service.SrvGetList, "", lResponse.StatusCode, ex.Message, lResponse.ResponseObject, ref lTrace));
}
return lResponse;
}
what am i missing here?
A 500 error means there's a problem on the server you're making the request to. You'll need to check to make sure two things;
1) Make sure your request is properly formatted, and doesn't have any surprises or invalid values for the requested resource.
2) that the server you want to talk to CAN get requests, and that its up-to-date (if you control the client).
In either case, the most common reason for a 500 is that something outside of your control has gone wrong on the client server.

Unable to Post on Pinterest using their API

I'm receiving a 400 Bad Request error message when posting a pin on Pinterest. It works using Postman, but doesn't work programmatically. Using C#, has anyone been able to successfully post a pin on Pinterest without using the pinsharp wrapper?
private void postPinterest(string messages, string id, string usertoken, string image, string boardname, string username)
{
string link = null;
boardname = boardname.Replace(" ", "-");
string board = username + "/" + boardname;
string url = "https://api.pinterest.com/v1/pins?access_token=" + usertoken;
StringBuilder sb = new StringBuilder();
if (!string.IsNullOrEmpty(board))
sb.Append("&board=" + HttpUtility.UrlEncode(board));
if (!string.IsNullOrEmpty(messages))
sb.Append("&note=" + HttpUtility.UrlEncode(messages));
if (!string.IsNullOrEmpty(link))
sb.Append("&image_url=" + HttpUtility.UrlEncode(link));
string postdata = sb.ToString().Substring(1);
PostData(url, postdata);
}
private object PostData(string url, string postdata)
{
object json=null;
try
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
// req.Accept = "application/json";
using (var stream = req.GetRequestStream())
{
byte[] bindata = Encoding.ASCII.GetBytes(postdata);
stream.Write(bindata, 0, bindata.Length);
}
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
string response = new StreamReader(resp.GetResponseStream()).ReadToEnd();
json = JsonConvert.DeserializeObject<dynamic>(response);
return json;
}
catch (WebException wex)
{
if (wex.Response != null)
{
using (var errorResponse = (HttpWebResponse)wex.Response)
{
using (var reader = new StreamReader(errorResponse.GetResponseStream()))
{
string error = reader.ReadToEnd();
return json;
}
}
}
}
return json;
}
EDIT:
It doesn't work using the JSON format or x-www-form-urlencoded format.
I changed the content type to application/x-www-form-urlencoded and now I'm receiving the error message below. I receive 400 Bad Request error using JSON format:
"{\n \"message\": \"405: Method Not Allowed\",\n \"type\": \"http\"\n}"
The problem is the the parameter that you are posting.
In the Api i could find board as a parameter but both note and image comes under field parameter which specifies the return type JSON.
As per documentation on this page you can post in this format
https://api.pinterest.com/v1/boards/anapinskywalker/wanderlust/pins/?
access_token=abcde&
limit=2&
fields=id,link,counts,note
So I tried the following and its getting response
https://api.pinterest.com/v1/boards/?access_token="YourTokenWithoutQuotes"&fields=id%2Ccreator
Would suggest you to first test the Api you are hitting putting a breakpoint inside the PostData function and check if the passed url is in the correct format and compare it with Pininterest API Explorer.
As you might have already received authorization code and access token so I am assuming your post function should be working fine.
public string postPinterest(string access_token,string boardname,string note,string image_url)
{
public string pinSharesEndPoint = "https://api.pinterest.com/v1/pins/?access_token={0}";
var requestUrl = String.Format(pinSharesEndPoint, accessToken);
var message = new
{
board = boardname,
note = note,
image_url = image_url
};
var requestJson = new JavaScriptSerializer().Serialize(message);
var client = new WebClient();
var requestHeaders = new NameValueCollection
{
{"Content-Type", "application/json" },
{"x-li-format", "json" }
};
client.Headers.Add(requestHeaders);
var responseJson = client.UploadString(requestUrl, "POST", requestJson);
var response = new JavaScriptSerializer().Deserialize<Dictionary<string, object>>(responseJson);
return response;
}

C# HttpWebRequest Content-type not Changing

In C# i need to POST some data to a web server using HTTP. I keep getting errors returned by the web server and after sniffing throught the data I dound that the problem is that thew Content-type header is still set to "text/html" and isn't getting changed to "application/json; Charset=UTF-8" as in my program. I've tried everything I can think of that might stop it getting changed, but am out of ideas.
Here is the function that is causing problems:
private string post(string uri, Dictionary<string, dynamic> parameters)
{
//Put parameters into long JSON string
string data = "{";
foreach (KeyValuePair<string, dynamic> item in parameters)
{
if (item.Value.GetType() == typeof(string))
{
data += "\r\n" + item.Key + ": " + "\"" + item.Value + "\"" + ",";
}
else if (item.Value.GetType() == typeof(int))
{
data += "\r\n" + item.Key + ": " + item.Value + ",";
}
}
data = data.TrimEnd(',');
data += "\r\n}";
//Setup web request
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(Url + uri);
wr.KeepAlive = true;
wr.ContentType = "application/json; charset=UTF-8";
wr.Method = "POST";
wr.ContentLength = data.Length;
//Ignore false certificates for testing/sniffing
wr.ServerCertificateValidationCallback = delegate { return true; };
try
{
using (Stream dataStream = wr.GetRequestStream())
{
//Send request to server
dataStream.Write(Encoding.UTF8.GetBytes(data), 0, data.Length);
}
//Get response from server
WebResponse response = wr.GetResponse();
response.Close();
}
catch (WebException e)
{
MessageBox.Show(e.Message);
}
return "";
}
The reason i'm getting problems is because the content-type stays as "text/html" regardless of what I set it as.
Thanks in advence.
As odd as this might sound, but this worked for me:
((WebRequest)httpWebRequest).ContentType = "application/json";
this changes the internal ContentType which updates the inherited one.
I am not sure why this works but I would guess it has something to do with the ContentType being an abstract property in WebRequest and there is some bug or issue in the overridden one in HttpWebRequest
A potential problem is that you're setting the content length based on the length of the string, but that's not necessarily the correct length to send. That is, you have in essence:
string data = "whatever goes here."
request.ContentLength = data.Length;
using (var s = request.GetRequestStream())
{
byte[] byteData = Encoding.UTF8.GetBytes(data);
s.Write(byteData, 0, data.Length);
}
This is going to cause a problem if encoding your string to UTF-8 results in more than data.Length bytes. That can happen if you have non-ASCII characters (i.e. accented characters, symbols from non-English languages, etc.). So what happens is your entire string isn't sent.
You need to write:
string data = "whatever goes here."
byte[] byteData = Encoding.UTF8.GetBytes(data);
request.ContentLength = byteData.Length; // this is the number of bytes you want to send
using (var s = request.GetRequestStream())
{
s.Write(byteData, 0, byteData.Length);
}
That said, I don't understand why your ContentType property isn't being set correctly. I can't say that I've ever seen that happen.

Trouble in fetching results from server in windows phone

I'm communicating with my server with the following code,
private void Save_Click(object sender, RoutedEventArgs e)
{
var request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Method = "POST";
request.BeginGetResponse(new AsyncCallback(GotResponse), request);
}
private void GotResponse(IAsyncResult asynchronousResult)
{
try
{
string data;
HttpWebRequest myrequest = (HttpWebRequest)asynchronousResult.AsyncState;
using (HttpWebResponse response = (HttpWebResponse)myrequest.EndGetResponse(asynchronousResult))
{
System.IO.Stream responseStream = response.GetResponseStream();
using (var reader = new System.IO.StreamReader(responseStream))
{
data = reader.ReadToEnd();
}
responseStream.Close();
}
this.Dispatcher.BeginInvoke(() =>
{
MessageBox.Show(data);
});
}
catch (Exception e)
{
var we = e.InnerException as WebException;
if (we != null)
{
var resp = we.Response as HttpWebResponse;
var code = resp.StatusCode;
this.Dispatcher.BeginInvoke(() =>
{
MessageBox.Show("Message :" + we.Message + " Status : " + we.Status);
});
}
else
throw;
}
}
I'm giving date and amount as my input value,it is url encoded. If all my data's are valid then everything works fine. And so my server will give the data as
{
"code":0,
"message":"Success",
"data":{
"date":xxxx,
"amount":123
}
}
But in case if give an invalid value,(For eg: abcd for 'amount'), then my server would reply as
{
"code":2,
"message":"Invalid value passed"
}
In this case, after Executing the line
using (HttpWebResponse response = (HttpWebResponse)myrequest.EndGetResponse(asynchronousResult))
It jumps to catch, and it display
Message:The remote server returned an error:NotFound. Status:UnKnown Error
Required Solution: It should fetch the result as it did in the previous case.
What sholud i do to fix it?
Well presumably the HTTP status code is 404. You're already accessing the response in your error code though - all you need to do is try to parse it as JSON, just as you are in the success case, instead of using we.Message to show an error message. You should probably be ready for the content to either be empty or not include valid JSON though, and only do this on specific status codes that you expect to still return JSON.

Categories