We Keep Coding

Multipart POST to the Joplin REST API using C# and Flurl

I am currently working on a console app to import data into Joplin for Windows 10, using C# and Flurl.
Joplin's API description can be found here.
I am trying to create a new resource in Joplin for a file on my system, so it can be attached to a Joplin note.
With CURL I can create the resource using command:
curl -F "data=#c:\\temp\\Test.pptx" -F "props={\"title\":\"my resource title\"}" http://localhost:41184/resources?token=MyToken
(note: it only works with "data=#c:\temp\Test.pptx", NOT with "data=c:\temp\Test.pptx")
When I try this with Flurl in c# I get a 400 response from Joplin, in the log I find:
Error: Resource cannot be created without a file
at Api.action_resources (C:\Program Files\Joplin\resources\app.asar\lib\services\rest\Api.js:351:37)
at Api.route (C:\Program Files\Joplin\resources\app.asar\lib\services\rest\Api.js:140:42)
at execRequest (C:\Program Files\Joplin\resources\app.asar\lib\ClipperServer.js:157:39)
at C:\Program Files\Joplin\resources\app.asar\lib\ClipperServer.js:185:8
at C:\Program Files\Joplin\resources\app.asar\node_modules\multiparty\index.js:136:9
at C:\Program Files\Joplin\resources\app.asar\node_modules\multiparty\index.js:115:9
at processTicksAndRejections (internal/process/task_queues.js:75:11)"
I have tried this so far:
try
{
var url = BaseUrl
.WithHeader("User_Agent", browserUserAgent)
.AppendPathSegment("resources")
.SetQueryParam("token", Token);
using (var fs = new FileStream("c:\\temp\\Test.pptx", FileMode.Open, FileAccess.Read))
{
var resource = url.PostMultipartAsync(mp => mp
.AddJson("props", new { title = "test title" })
.AddFile("data", fs, "Test.pptx", "application/octet-stream")
)
.ReceiveJson<JoplinResource>()
.Result;
}
}
and:
try
{
var url = BaseUrl
.WithHeader("User_Agent", browserUserAgent)
.AppendPathSegment("resources")
.SetQueryParam("token", Token);
var resource = url.PostMultipartAsync(mp => mp
.AddJson("props", new { title = "test title" })
.AddFile("data", "c:\\temp\\Test.pptx")
)
.ReceiveJson<JoplinResource>()
.Result;
}
I hooked up fiddler to see what is the difference between my application and CURL.
Curl:
POST http://127.0.0.1:41184/resources?token=MyToken HTTP/1.1
Host: 127.0.0.1:41184
User-Agent: curl/7.70.0
Accept: */*
Connection: Keep-Alive
Content-Length: 33648
Content-Type: multipart/form-data; boundary=------------------------91ab181cbb0247ba
--------------------------91ab181cbb0247ba
Content-Disposition: form-data; name="props"
{"title":"my resource title"}
--------------------------91ab181cbb0247ba
Content-Disposition: form-data; name="data"; filename="Test.pptx"
Content-Type: application/octet-stream
...
My Console app:
POST http://localhost:41184/resources?token=MyToken HTTP/1.1
User_Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36
Content-Type: multipart/form-data; boundary="f603841b-5c32-4e77-985a-69c2ffb6eed0"
Host: localhost:41184
Content-Length: 33612
Expect: 100-continue
Accept-Encoding: gzip, deflate
--f603841b-5c32-4e77-985a-69c2ffb6eed0
Content-Disposition: form-data; name=props
{"title":"My Resource"}
--f603841b-5c32-4e77-985a-69c2ffb6eed0
Content-Disposition: form-data; name=data; filename=Test.pptx; filename*=utf-8''Test.pptx
...
NOTE the differences:
props and data are in quotes when using CURL, not with FLURL
FLURL sends a second file name: filename*=utf-8''Test.pptx
How do I get this to work properly?

The issue was in the missing quotes for the "data" and "props":
try
{
var url = BaseUrl
.WithHeader("User_Agent", browserUserAgent)
.AppendPathSegment("resources")
.SetQueryParam("token", Token);
var resource = url.PostMultipartAsync(mp => mp
.AddJson("\"props\"", new { title = "My Resource" })
.AddFile("\"data\"", "c:\\temp\\Test.pptx")
)
.ReceiveJson<JoplinResource>()
.Result;
}
Raw request header is now:
POST http://localhost:41184/resources?token=MyToken HTTP/1.1
User_Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36
Content-Type: multipart/form-data; boundary="c6b2377a-1240-4ae3-872f-fa24b643d3e0"
Host: localhost:41184
Content-Length: 33616
Expect: 100-continue
Accept-Encoding: gzip, deflate
--c6b2377a-1240-4ae3-872f-fa24b643d3e0
Content-Disposition: form-data; name="props"
{"title":"My Resource"}
--c6b2377a-1240-4ae3-872f-fa24b643d3e0
Content-Disposition: form-data; name="data"; filename=Test.pptx; filename*=utf-8''Test.pptx
...
And the Joplin REST service creates a new resource...

c# GetResponse() timeout, but works on browser

I'm trying to read the response i get from nyc.gov. I used Fiddler to construct the WebRequest and it keeps timing out.
Important: this works if the url is https://www.google.com so it's got to be something from the nyc.gov server. But how can it know the difference between my code and Chrome?
I tried settings the KeepAlive to true/false/none.
I tried using Http1.0
I tried setting request.ServicePoint.Expect100Continue to false
I tried setting request.ContentLength = 0;
I tried enclosing in "using"
I added to app.config
<system.net>
<connectionManagement>
<add address="*" maxconnection="1000" />
</connectionManagement>
</system.net>
Here is my code:
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://a810-bisweb.nyc.gov/bisweb/bispi00.jsp");
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", #"1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9");
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException e)
{
if (e.Status == WebExceptionStatus.ProtocolError) response = (HttpWebResponse)e.Response;
else return false;
}
catch (Exception)
{
if (response != null) response.Close();
return false;
}
Here is the RAW request (provided by Fiddler) from Chrome - WORKS:
GET http://a810-bisweb.nyc.gov/bisweb/bispi00.jsp HTTP/1.1
Host: a810-bisweb.nyc.gov
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
And this is the RAW request from my code - HANGS (and eventually times out)
GET http://a810-bisweb.nyc.gov/bisweb/bispi00.jsp HTTP/1.1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Host: a810-bisweb.nyc.gov
Connection: Keep-Alive

C# Http WebRequest JSON

I'm new to WebRequests and I tried this:
string json = #"{""Gebruikersnaam"":""user"",""Wachtwoord"":""pass"",""IngelogdBlijven"":true}";
var httpWebRequest = (HttpWebRequest)WebRequest.Create("https://pontessg.magister.net/api/sessies");
httpWebRequest.ContentType = "application/json";
httpWebRequest.ContentLength = json.Length;
Console.WriteLine(json.Length);
Console.WriteLine(json);
Console.Read();
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
Console.WriteLine(result);
}
My code is keep returning me ERROR (400) and I don't really know why I think I forgot something, but I don't know what
(The Headers)
Request URL:https://pontessg.magister.net/api/sessies
Request Method:POST
Status Code:200 OK
Remote Address:[...]
Response Headers
view parsed
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 76
Content-Type: application/json; charset=utf-8
Expires: -1
Date: Sat, 25 Feb 2017 21:45:33 GMT
X-Frame-Options: DENY
Strict-Transport-Security: max-age=[...]
Request Headers
view parsed
POST /api/sessies HTTP/1.1
Host: pontessg.magister.net
Connection: keep-alive
Content-Length: 88
Accept: application/json, text/plain, */*
X-API-Client-ID: 12D8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Origin: https://pontessg.magister.net
Content-Type: application/json;charset=UTF-8
DNT: 1
Referer: https://pontessg.magister.net/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: SESSION_ID=[...]; M6UserName=user; username=pass; Magister.UserName=user; [...]
Request Payload
{"Gebruikersnaam":"user","Wachtwoord":"pass","IngelogdBlijven":true}
When I try the JS code below in the console of my browser, it returns 200 OK
$.ajax({
url: "https://pontessg.magister.net/api/sessies",
type: "POST",
data: "{\"Gebruikersnaam\":\"user\",\"Wachtwoord\":\"pass\",\"IngelogdBlijven\":true}",
contentType: "application/json;charset=utf-8",
success: action_Succeeded,
error: action_Failed
});
function action_Succeeded(r) {
console.log("succes");
}
function action_Failed(r1, r2, r3) {
alert("fail");
}
If someone could help me with this, I would like that

Ajax success function won't execute

I have an apache cordova project where I'm trying to post form data from the client to the database. I can see the json I send going out to my API which then sends the data to the DAL and then to the database. The entry correctly updates the table and then returns a 1 or -1 based on whether or not a previous entry exists. This value is then sent to the client. I can see the exact response being sent back, but my ajax success function won't fire.
Ajax:
$("#submit").click(function () {
var info = {
Username: $("#username").val(),
Password: $("#password").val(),
Firstname: $("#firstname").val(),
Lastname: $("#lastname").val(),
Email: $("#email").val()
};
info = JSON.stringify(info);
event.preventDefault();
$.ajax({
type: "POST",
dataType: "json",
async: false,
url: "http://localhost:57207/api/User/RegisterUser",
data: info,
success: function(data) {
console.log(data);
},
error: function(input) {
console.log(JSON.stringify(input));
}
})
});
Controller:
[System.Web.Http.HttpPost]
public JObject RegisterUser(JObject obj)
{
RegisterUsers user = new RegisterUsers();
user = JsonConvert.DeserializeObject<RegisterUsers>(((JProperty)obj.First).Name);
var temp = DataAccessLayer.RegisterUser(user.Username, user.Password, user.Firstname, user.Lastname, user.Email, null, null);
JObject jobj = new JObject();
jobj.Add("output", temp.ToString());
return jobj;
}
I have been using fiddler to check my response and request, and here is the raw data:
Request:
POST http://localhost:57207/api/User/RegisterUser HTTP/1.1
Host: localhost:57207
Connection: keep-alive
Content-Length: 125
Accept: text/html, */*; q=0.01
Origin: http://evil.com/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://localhost:4400/test.html
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
{"Username":"johndoe","Password":"password!!!","Firstname":"john","Lastname":"doe","Email":"johndoe#random.com"}
Response:
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
Access-Control-Allow-Origin: *
X-AspNet-Version: 4.0.30319
X-SourceFiles: =?UTF-8?B?RDpcRGV2ZWxvcG1lbnRcVGV4dDJQaG9uaWNzXFJlc3RBUElcYXBpXFVzZXJcUmVnaXN0ZXJVc2Vy?=
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,POST,OPTIONS
Access-Control-Allow-Headers: Content-Type, soapaction
Date: Wed, 08 Feb 2017 19:52:43 GMT
Content-Length: 15
{"output":"-1"}
My error function is the one that fires, and here is the output:
console.log:
{"readyState":0,"status":0,"statusText":"NetworkError: Failed to execute 'send' on 'XMLHttpRequest': Failed to load 'http://localhost:57207/api/User/RegisterUser'."}
Update:
I was able to set the async to true and my new error message is this:
{"readyState":0,"status":0,"statusText":"error"}

Try instead returning an IHttpActionResult or IActionResult, Ok() for a 200 response.
public IActionResult RegisterUser(JObject obj)
{
RegisterUsers user = new RegisterUsers();
user = JsonConvert.DeserializeObject<RegisterUsers>(((JProperty)obj.First).Name);
var temp = DataAccessLayer.RegisterUser(user.Username, user.Password, user.Firstname, user.Lastname, user.Email, null, null);
JObject jobj = new JObject();
jobj.Add("output", temp.ToString());
return Ok(jobj);
}
If it continues to fail, try just return a simple string like return Ok("Hello World!");. If it succeeds with simple string, then the serialization of the object may be failing or is coming out as invalid JSON.

C# .NET - HttpWebRequest - System.Net.WebException - Too many automatic redirections were attempted

I have a problem with HttpWebRequest class.
I am trying to get source code of website:
http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983
but I am always getting an error:
System.Net.WebException occurred
HResult=-2146233079
Message=Too many automatic redirections were attempted.
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at ProjectName.ClassName.MethodName(String urlAddress)
InnerException:
That is my code:
Uri uri = new Uri(#"http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
I have used a Fiddler Web Debugger tool to compare Firefox request with my C# .NET request, but still have no answer.
Firefox:
GET http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983 HTTP/1.1
Host: www.filmweb.pl
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate, proxy-revalidate
Content-Type: text/html;charset=UTF-8
Content-Language: pl-PL
Transfer-Encoding: chunked
Date: Wed, 07 Oct 2015 13:36:31 GMT
X-Cache: HIT from blade110.non.3dart.com
X-Cache-Hits: 116
Server: Apache
C# .NET:
GET http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci:+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983 HTTP/1.1
Host: www.filmweb.pl
Connection: Keep-Alive
HTTP/1.1 301 Moved Permanently
Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate, proxy-revalidate
Content-Type: text/html;charset=UTF-8
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Language: pl-PL
Location: /film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983
Content-Length: 0
Accept-Ranges: bytes
Date: Wed, 07 Oct 2015 13:34:51 GMT
X-Cache: MISS from blade712.non.3dart.com
Server: Apache
I have read other posts and update my code by different things, eg.
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.TransferEncoding = "gzip, deflate";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0";
request.Referer = "http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983";
request.KeepAlive = true;
request.AllowAutoRedirect = true;
request.MaximumAutomaticRedirections = 250;
request.Proxy = null;
request.UseDefaultCredentials = true;
CookieContainer cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;
but nothing works :-/
Can anybody help me with this problem?

You need to have the initial cookies when the website load before you fetch a deep-link.
The following code works for me:
// cookies
CookieContainer cookieContainer = new CookieContainer();
// make one call to the root of the website
// to get the cookies set
Uri uri = new Uri(#"http://www.filmweb.pl");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.CookieContainer = cookieContainer;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using(var s = response.GetResponseStream())
{
using(var sr = new StreamReader(s))
{
// linqpad
sr.ReadToEnd().Dump(); // to check for errors
}
}
// we have cookies now
// do the deep link fetch
uri = new Uri(#"http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983");
request = (HttpWebRequest)WebRequest.Create(uri);
request.CookieContainer = cookieContainer;
response = (HttpWebResponse)request.GetResponse();
//store the result
using(var f = File.Create("C:\\temp\\pl.txt"))
{
response.GetResponseStream().CopyTo(f);
}
Make sure that if you scrape a website that you adhere to their license and usage policies. Don't do anything that goes beyond fair use or against any copy-righted materials.