I am trying to get the header "Set-Cookie" or access the cookie container, but the Set-Cookie header is not available.
The cookie is in the response header, but it's not there in the client request object.
I am registering the ClientHttp stack using
bool httpResult = WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
Here's the response:
HTTP/1.1 200 OK
Content-Type: application/xml; charset=utf-8
Connection: keep-alive
Status: 200
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.0.pre4
ETag: "39030a9c5a45a24e485e4d2fb06c6389"
Client-Version: 312, 105, 0, 0
X-Runtime: 44
Content-Length: 1232
Set-Cookie: _CWFServer_session=[This is the session data]; path=/; HttpOnly
Cache-Control: private, max-age=0, must-revalidate
Server: nginx/0.7.67 + Phusion Passenger 3.0.0.pre4 (mod_rails/mod_rack)
<?xml version="1.0" encoding="UTF-8"?>
<user>
...
</user>
My callback code contains something like:
var webRequest = (HttpWebRequest)result.AsyncState;
raw = webRequest.EndGetResponse(result) as HttpWebResponse;
foreach (Cookie c in webRequest.CookieContainer.GetCookies(webRequest.RequestUri))
{
Console.WriteLine("Cookie['" + c.Name + "']: " + c.Value);
}
I've also tried looking at the headers but Set-Cookie header isn't present in the response either.
Any suggestions on what may be the problem?
Try explicitly passing a new CookieContainer:
CookieContainer container = new CookieContainer();
container.Add(new Uri("http://yoursite"), new Cookie("name", "value"));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://yoursite");
request.CookieContainer = container;
request.BeginGetResponse(new AsyncCallback(GetData), request);
You are receiving HttpOnly cookies:
Set-Cookie: _CWFServer_session=[This is the session data]; path=/; HttpOnly
For security reasons, those cookies can't be accessed from code, but you still can use them in your next calls to HttpWebRequest. More on this here : Reading HttpOnly Cookies from Headers of HttpWebResponse in Windows Phone
With WP7.1, I also had problems reading non HttpOnly cookies. I found out that they are not available if the response of the HttpWebRequest comes from the cache. Making the query unique with a random number solved the cache problem :
// The Request
Random random = new Random();
// UniqueQuery is used to defeat the cache system that destroys the cookie.
_uniqueQuery = "http://my-site.somewhere?someparameters=XXX"
+ ";test="+ random.Next();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_uniqueQuery);
request.BeginGetResponse(Response_Completed, request);
Once you get the response, you can fetch the cookie from the response headers:
void Response_Completed(IAsyncResult result)
{
HttpWebRequest request = (HttpWebRequest)result.AsyncState;
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result);
String header = response.Headers["Set-Cookie"];
I never managed to get the CookieContainer.GetCookies() method to work.
Is the cookie httponly? If so, you won't be able to see it, but if you use the same CookieContainer for your second request, the request will contain the cookie, even though your program won't be able to see it.
You must edit the headers collection directly. Something like this:
request.Headers["Set-Cookie"] = "name=value";
request.BeginGetResponse(myCallback, request);
Related
I'm working on a .net website that uses cookies for forms authentication and I wanted to add another secure cookie to hold access and refresh tokens. The cookie is being added to the Response.Cookies and everything seems fine. I used fiddler to inspect the response headers to make sure the Set-Cookie header was being set.
HTTP/1.1 302 Found
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Location: /
Set-Cookie:
TestTokenCookie=(truncated for brevity);
domain=local.foobar.com;
expires=Sun, 18-Nov-2018 14:42:56 GMT;
path=/
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: IE=Edge,chrome=1
Date: Mon, 20 Aug 2018 13:42:59 GMT
Content-Length: 118
The response looks correct but no cookie appears in the browser for it. I'm using Edit This Cookie chrome extension to view what cookies have been set. Here's the code for setting the cookie that I used.
public void CreateTokenCookie(TokenCookieData tokenCookieData, HttpContextBase currentContext, bool createPersistentTicket = true)
{
var ticket = new FormsAuthenticationTicket(1,
tokenCookieData.Username,
DateTime.Now,
DateTime.Now.AddDays(90),
createPersistentTicket,
tokenCookieData.ToString());
CreateCookieFromTicket(ticket, TOKEN_COOKIE_NAME, true, currentContext);
}
private void CreateCookieFromTicket(FormsAuthenticationTicket ticket, string cookieName, bool httpOnly, HttpContextBase currentContext)
{
var encryptedTicket = FormsAuthentication.Encrypt(ticket);
var cookie = new HttpCookie(cookieName, encryptedTicket)
{
HttpOnly = httpOnly,
Secure = FormsAuthentication.RequireSSL,
Path = FormsAuthentication.FormsCookiePath,
Expires = ticket.Expiration
};
var domain = GetCookieDomain();
if (domain != null)
{
cookie.Domain = domain;
}
if (currentContext.Response.Cookies[cookieName] != null)
{
currentContext.Response.Cookies.Remove(cookieName);
}
currentContext.Response.Cookies.Add(cookie);
}
Any ideas why the cookie is not being set in the browser?
I think the issue may be your doman "domain=local.test.com;"
See this https://stackoverflow.com/a/24071239/10241547 for more details
test.com seems to be part of that restricted list
or may be it is the com
// com : https://en.wikipedia.org/wiki/.com
com
See: https://publicsuffix.org/list/public_suffix_list.dat
I believe the issue ended up being the length of the value of the cookie. The raw value ended up being about 4105 characters which exceeds the max size of a cookie 4093 bytes.
Sending cookie received from httpwebrequest is not giving correct result however if i copy paste cookie value from browser cookie than it returns correct result. why i am not getting result from httpwebrequest but works perfectly fine from browser?
CookieContainer cookieContainer = new CookieContainer();
var targetUri = new Uri("URL1");
HttpWebRequest myHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(targetUri);
myHttpWebRequest.Method = "GET";
myHttpWebRequest.CookieContainer = cookieContainer;
//Get Response
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
//Create Request
targetUri = new Uri("URL2");
myHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(targetUri);
myHttpWebRequest.Method = "GET";
myHttpWebRequest.CookieContainer = cookieContainer;
//Get Response
myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
using (StreamReader reader = new StreamReader(myHttpWebResponse.GetResponseStream()))
{
string html = reader.ReadToEnd();
}
Following is second request(URL2) fiddler with cookie received through first request(URL1):
Request:
GET URL2
HTTP/1.1
Host: www.xyz.com
Cookie: JSESSIONID=Mn7qJwrRnxLn1NNfT1PNr1L2Gr2KCkfFVCRS1wfsT4zfzWJhT62J!-876337174
Response:
HTTP/1.1 200 OK
Date: Fri, 27 Feb 2015 13:03:52 GMT
Content-Length: 13
Content-Type: text/html;charset=UTF-8
X-Powered-By: Servlet/2.5 JSP/2.1
Now if i copy paste first url(URL1) in browser and use cookie value from browser then it returns correct result:
Request:
GET URL2
HTTP/1.1
Host: www.xyz.com
Cookie: JSESSIONID=PPPHJwmKQNh2ykVXytlcfTDH2YWNbtv76vPBzZTG3Dfdm9Mx0J74!-876337174
Response:
HTTP/1.1 200 OK
Date: Fri, 27 Feb 2015 13:06:15 GMT
Content-Type: text/html;charset=UTF-8
X-Powered-By: Servlet/2.5 JSP/2.1
Content-Length: 21417
Weblogic is returning the cookies in the response, and you need to send it back in the next request.
cookieContainer.Add(response.Cookies);
(Your cookieContainer is on your machine and is empty).
I'm trying to automate configuring a wireless router's SSID and Password via c# webclient. The router has no API that I know of. It's an unbranded chinese router. The web config seems to be the only option for configuration. It uses http-basic-authentication (you browse to the IP address of the router and get a generic dialog asking for username and password).
I've used Wireshark to get the headers and form fields that the http-post requests use when I manually update the SSID and Password (two separate forms). I then attempted to use webclient to emulate those post requests.
Here is a snippet of code that I am using to attempt to save a new SSID (NameValueCollection is defined elsewhere):
private const string FORM_SSID = "http://192.168.1.2/formWlanSetup.htm";
private const string REF_SSID = "http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0";
private NameValueCollection mFields = HttpUtility.ParseQueryString(string.Empty, Encoding.ASCII);
public string SaveConfigResponse()
{
try
{
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.Accept] = "text/html, application/xhtml+xml, */*";
wc.Headers[HttpRequestHeader.Referer] = REF_SSID;
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-US";
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
wc.Headers[HttpRequestHeader.Host] = "192.168.1.2";
wc.Headers[HttpRequestHeader.Connection] = "Keep-Alive";
wc.Headers[HttpRequestHeader.ContentLength] = Encoding.ASCII.GetBytes(mFields.ToString()).Length.ToString();
wc.Headers[HttpRequestHeader.CacheControl] = "no-cache";
string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(config_user + ":" + config_pass));
wc.Headers[HttpRequestHeader.Authorization] = string.Format("Basic {0}", credentials);
//wc.Credentials = new NetworkCredential("admin", "admin");
return Encoding.ASCII.GetString(wc.UploadValues(FORM_SSID, "POST", mFields));
}
}
catch (Exception ex)
{
return ex.Message;
}
}
This results in an http-status-code-401 not authorized response. Is what I'm trying to do just impossible?
UPDATE
Here are the HTTP headers of both the browser post/response and the WebClient post/response. Again, I tried to match what I saw the browser posting as well as I could with my WebClient post.
Browser:
POST /formWlanSetup.htm HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Referer: http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host: 192.168.1.2
Content-Length: 524
Connection: Keep-Alive
Cache-Control: no-cache
Authorization: Basic YWRtaW46YWRtaW4=
HTTP/1.1 302 Found
Location: wlbasic.htm
Content-Length: 183
Date: Thu, 23 Oct 2014 18:18:27 GMT
Server: eCos Embedded Web Server
Connection: close
Content-Type: text/html
Transfer-Encoding: chunked
Cache-Control: no-cache
WebClient:
POST /formWlanSetup.htm HTTP/1.1
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Cache-Control: no-cache
Authorization: Basic YWRtaW46YWRtaW4=
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
Referer: http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: 192.168.1.2
Content-Length: 524
Connection: Keep-Alive
HTTP/1.1 401 Not Authorized
WWW-Authenticate: Basic realm="AP"
Date: Thu, 23 Oct 2014 18:18:41 GMT
Server: eCos Embedded Web Server
Connection: close
Content-Type: text/html
Transfer-Encoding: chunked
Cache-Control: no-cache
Again, that was all gleaned from Wireshark. I'm not very familiar with Wireshark, but I was able to get this far. If I knew how to properly extract the raw packet data and pastebin it, I would.
Important New Observations
The Wireshark captures of the post packets from both Browser and WebClient obviously differ in the order of the headers. I don't know how significant that might or might not be, though, as the data for each header is clearly the same.
One stark difference between the packets that I noticed is that Wireshark reports the Browser packet to be significantly larger than the WebClient packet. Looking at the itemized view, I couldn't find any obvious differences. I assume posting raw data for comparison would reveal a lot, but again, I don't really know how to do that.
I had a bewildering revelation. Despite the response clearly stating '(401) Unauthorized', the post is in fact being accepted by the router! Driving in to the router's web config after my WebClient post shows that the settings were accepted and saved.
That last one is a biggie. I find myself in a situation where I can get my config to save with a WebClient post, but I have to ignore a 401 response in order to do so. Obviously, this is far from ideal. So close, yet so far!
FINAL UPDATE (RESOLUTION)
I've solved the issue of failing basic authentication, though not with WebClient. I used the suggestion from #caesay and went with HttpWebRequest (together with WebResponse). My form posts result in redirects, so I had to allow for that.
This is essentially what I went with:
private bool ConfigureRouter()
{
bool passed = false;
string response = "";
HttpWebRequest WEBREQ = null;
WebResponse WEBRESP = null;
// Attempt to POST form to router that saves a new SSID.
try
{
var uri = new Uri(FORM_SSID); // Create URI from URL string.
WEBREQ = HttpWebRequest.Create(uri) as HttpWebRequest;
// If POST will result in redirects, you won't see an "OK"
// response if you don't allow those redirects
WEBREQ.AllowAutoRedirect = true;
// Basic authentication will first send the request without
// creds. This is protocol standard.
// When the server replies with 401, the HttpWebRequest will
// automatically send the request again with the creds when
// when PreAuthenticate is set.
WEBREQ.PreAuthenticate = true;
WEBREQ.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
// Mimic all headers known to satisfy the request
// as discovered with a tool like Wireshark or Fiddler
// when the form was submitted from a browser.
WEBREQ.Method = "POST";
WEBREQ.Accept = "text/html, application/xhtml+xml, */*";
WEBREQ.Headers.Add("Accept-Language", "en-US"); // No AcceptLanguage property built-in to HttpWebRequest
WEBREQ.UserAgent = USER_AGENT;
WEBREQ.Referer = REF_SSID;
WEBREQ.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
WEBREQ.KeepAlive = true;
WEBREQ.Headers.Add("Pragma", "no-cache"); // No Pragma property built-in to HttpWebRequest
// Use a cached credential so that the creds are properly
// submitted with subsequent redirect requests.
CredentialCache creds = new CredentialCache();
creds.Add(uri, "Basic", new NetworkCredential(config_user, config_pass));
WEBREQ.Credentials = creds;
// Submit the form.
using (Stream stream = WEBREQ.GetRequestStream())
{
SSID ssid = new SSID(ssid_scanned); // Gets predefined form fields with new SSID inserted (NameValueCollection PostData)
stream.Write(ssid.PostData, 0, ssid.PostData.Length);
}
// Get the response from the final redirect.
WEBRESP = WEBREQ.GetResponse();
response = ((HttpWebResponse)WEBRESP).StatusCode.ToString();
if (response == "OK")
{
StatusUpdate("STATUS: SSID save was successful.");
passed = true;
}
else
{
StatusUpdate("FAILED: SSID save was unsuccessful.");
passed = false;
}
WEBRESP.Close();
}
catch (Exception ex)
{
StatusUpdate("ERROR: " + ex.Message);
return false;
}
return passed;
}
Is what I'm trying to do just impossible?
No, its not impossible. I have had many headaches with web scraping like this over the years because some web servers are picky, and your router interface is likely a custom web server implementation that isnt as forgiving as apache or iis.
I would do a wireshark capture and get the raw packet data that chrome sends (w/ payload etc), and then do the same capture for your application. Make sure the packets are as similar as you can get them. If you still have issues, post the packet captures to pastebin or something so we can have a look.
EDIT::
Instead of using the limited WebClient API, try using some lower level items, I wonder if the following code will work for you:
var uri = new Uri("http://192.168.1.2/formWlanSetup.htm");
var cookies = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.CookieContainer = cookies;
request.ServicePoint.Expect100Continue = false;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
request.Referer = "http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0";
request.Credentials = new NetworkCredential(config_user, config_pass);
request.PreAuthenticate = true;
var response = request.GetResponse();
var reader = new StreamReader(response.GetResponseStream());
string htmlResponse = reader.ReadToEnd();
I attempting to POST data to the following using HttpWebRequest. However I am not sure what the URI should be for the following task
POST /v1/session
Host: developer.messenger.yahooapis.com
Authorization: < Standard OAuth credentials >
Content-Type: application/json;charset=utf-8
Content-Length: 38
{
"presenceState" : 0,
"presenceMessage" : "I am now logged in"
}
I have something like this
string uri = "https://developer.messenger.yahooapis.com/v1/session ";
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(uri);
Is that uri correct for the required post.
Has been using Service Reference without any success:
Web service return only XML
Now I am using the raw SOAP message to do it:
XmlDocument doc = new XmlDocument();
doc.Load("Service.xml");
// create the request to your URL
Uri wsHost = new Uri("http://www.rrr.net/services/Connect");
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(wsHost);
// add the headers
// the SOAPACtion determines what action the web service should use
request.Headers.Add("SOAPAction", "act");
// set the request type
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
// add our body to the request
Stream stream = request.GetRequestStream();
doc.Save(stream);
stream.Close();
// get the response back
using( HttpWebResponse response = (HttpWebResponse)request.GetResponse() )
{
Stream dataStream = response.GetResponseStream();
StreamReader dataReader = new StreamReader(dataStream);
// Use Linq to read the xml response
using (XmlReader reader = XmlReader.Create(dataStream))
{
The post is correct, but response always give me text/plain empty result, the reponse header:
Headers = {Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain
Date: Thu, 06 Sep 2012 15:59:28 GMT
}
The SOAP message is, act is the function:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webService">
<soapenv:Header/>
<soapenv:Body>
<web:act>
<web:d1>1</web:d1>
<web:d2>14</web:d2>
</web:act>
</soapenv:Body>
</soapenv:Envelope>
I use SoapUI, below is the raw request from SoapUI, it return a xml result:
POST http://www.rrr.net/services/Connect HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: ""
Content-Length: 516
Host: www.rrr.net
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
Thank you.
SOAP webservices require action/method to be specified (and NOT empty). If you don't know which action you want you can look at webservice WSDL by invoking the webservice with queryString "?WSDL". I.e. www.yourSite.com/your/Web/Service/URL?WSDL
You must specify an action in both the request and also the service interface. You can set the action value on the interface member using the attributes shown below and then in the request using the method you have used but by specifying the action name you used in the contract:
The attributes on the interface member
[OperationContract Name="YourActionName"]
[WebInvoke (Method = "POST", UriTemplate = "YourActionName")]
Message YourServiceFunction();
One method of specifying the action on the message
Message inputMessage = Message.CreateMessage (MessageVersion.Soap, "YourActionName", reader);