If I have a URL like http://popurls.com/go/msn.com/l4eba1e6a0ffbd9fc915948434423a7d5, how do I expand it back to the original URL programmatically? Obviously I could use an API like expandurl.com but that will limits me to 100 requests per hour.
Make a request to the URL and check the status code returned. If 301 or 302, look for a Location header, which will contain the "expanded URL":
string url = "http://popurls.com/go/msn.com/l4eba1e6a0ffbd9fc915948434423a7d5";
var request = (HttpWebRequest) WebRequest.Create(url);
request.AllowAutoRedirect = false;
var response = (HttpWebResponse) webRequest.GetResponse();
if ((int) response.StatusCode == 301 || (int) response.StatusCode == 302)
{
url = response.Headers["Location"];
}
Note: This solution presumes that only one redirect occurs. This may or may not be want you need. If you simply want to deobfuscate URLs from obfuscators (bit.ly et al) this solution should work well.
Managed to find an answer.
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://popurls.com/go/msn.com/l4eba1e6a0ffbd9fc915948434423a7d5");
req.AllowAutoRedirect = true;
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
ServicePoint sp = req.ServicePoint;
Console.WriteLine("End address is " + sp.Address.ToString());
Related
I have a CLR function getting data from cookie authorized website. The first request gets a login cookies and the second request gets xml data I need. The problem is in that I am always getting 401 unauthorized on a second request when run it from SQL Server as a function. The testing console app using the same DLL is working fine. Looks like the second request has no cookies but I checked in exception the amount of cookie container of the second request, it is not empty.
String encoded = Convert.ToBase64String(Encoding.UTF8.GetBytes(UserName + ":" + Password));
try
{
HttpWebRequest loginrequest = (HttpWebRequest)WebRequest.Create(string.Format("{0}", BaseOrdersAddress));
CookieContainer logincookies = new CookieContainer();
loginrequest.Headers.Add(HttpRequestHeader.Authorization, "Basic " + encoded);
loginrequest.AllowAutoRedirect = false;
loginrequest.CookieContainer = logincookies;
loginrequest.Method = WebRequestMethods.Http.Get;
HttpWebResponse loginresponse = (HttpWebResponse)loginrequest.GetResponse();
loginresponse.Close();
if (loginresponse.StatusCode == HttpStatusCode.Found)
{
location = loginresponse.Headers[HttpResponseHeader.Location];
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(location);
request.CookieContainer = logincookies;
request.Method = WebRequestMethods.Http.Get;
HttpWebResponse response1 = request.GetResponse() as HttpWebResponse;
var xresult = new XmlSerializer(typeof(Local_Response));
r = (Local_Response)xresult.Deserialize(response1.GetResponseStream());
}
Solved.
The problem was in using HttpWebRequest.CookieContainer, don't know why but it does not work while running as a CLR function, no cookies are sent. Have to do it manually adding HttpRequestHeader.Cookie to request headers collection.
Don't forget, your SQLCLR code executes in the context of SQL Server. I see you have a username, password in the code - what does that do and where is the username/password retrieved from. My bet is that there is something wrong with this based on what I said earlier.
I have a string with 300 rows, there is a way to do it POST?
Here is my code, is currently working on a limited amount of short letters:
WebRequest req = (HttpWebRequest)WebRequest.Create(
"http://thisisurl/test.php?ad=test&f=" + information_data);
req.Method = "POST";
WebResponse res = req.GetResponse();
I'm going to explain your problem right now and go ahead and give you a possible solution at the end.
You are hitting the character limit for url length / query parameter length. IE limits it as low as 2,083.
The data you are providing should be sent in the body of the http request, not the URL parameters.
A Post Request Normally is done in the following format(Code from the link).
using (var wb = new WebClient())
{
var data = new NameValueCollection();
data["username"] = "myUser";
data["password"] = "myPassword";
var response = wb.UploadValues(url, "POST", data);
}
This thread should have enough info if you want to use the WebRequest class instead:
HTTP request with post
After a long day in searching and attempts I could not find a solution for my problem - getting multiple "Set-Cookie" headers.
I was trying to get them via Headers["Set-Cookie"] but it contains the only the first cookie.
Then my next try was to get them via Cookies (which is always with Count 0 and the purpose because I think they are malformed).
Here is the code to the last sample:
var request = WebRequest.Create(resourceUrl) as HttpWebRequest;
request.Method = "GET";
var response = request.GetResponse() as HttpWebResponse;
var cookiesCount = response.Cookies.Count;
And these are the cookies:
Set-Cookie:vjfmrii=67ea0de93a423ab17d168ee8327617b0
Set-Cookie:alpocjengi=dcf10w329x5d7e503ffb9f28123c7492f1c2deb4
Set-Cookie:vjfmrii=4abf7b9e97fff1a61fbcf5e11899ce71
Before I began, I thought that this would be one of the easiest parts, but unfortunately I was wrong. These cookies which have the same names and have several... have left me devastated.
I appreciate any help. Thanks in advance.
You need to add a CookieContainer to your request when you make it or the cookies will not be added to the response. See http://msdn.microsoft.com/en-us/library/dd920298(v=vs.95).aspx.
var request = WebRequest.Create(resourceUrl) as HttpWebRequest;
request.Method = "GET";
request.CookieContainer = new CookieContainer(); // <-- Add this
var response = request.GetResponse() as HttpWebResponse;
var cookiesCount = response.Cookies.Count;
I am able to get numbers with enum as suggested by dtb in Getting Http Status code number (200, 301, 404, etc.) from HttpWebRequest and HttpWebResponse. However, for moved permanently site also i am getting 200 (OK). What I want to see is 301 instead. Please help. My code is below. What could be wrong/needs to be corrected?
public int GetHeaders(string url)
{
//HttpStatusCode result = default(HttpStatusCode);
int result = 0;
var request = HttpWebRequest.Create(url);
request.Method = "HEAD";
try
{
using (var response = request.GetResponse() as HttpWebResponse)
{
if (response != null)
{
result = (int)response.StatusCode; // response.StatusCode;
response.Close();
}
}
}
catch (WebException we)
{
if (we.Response != null)
{
result = (int)((HttpWebResponse)we.Response).StatusCode;
}
}
return result;
}
The tool where i am using this code is capable of showing 404, not existing domains but it is ignoring the redirects and shows the details about the redirected url. e.g if i put my older domain easytipsandtricks.com in the text field, it shows the results for tipscow.com (if you check easytipsandtricks.com in any checker tool online, you will notice that it is giving the correct redirect message - 301 Moved). Please help.
You need to set HttpWebRequest.AllowAutoRedirect to false (default is true) for it to not automatically follow redirects (30x responses).
If AllowAutoRedirect is set to false, all responses with an HTTP status code from 300 to 399 is returned to the application.
Sample:
var request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Method = "HEAD";
request.AllowAutoRedirect = false;
Everything I find is either about a POST request or does not assume cookies.
I have an URL like this:
http://page.com/find/1,1,1,find.html?advanced=1¶m1=val1¶m2[]=val2
When put into a browser, this will direct me to a search results page. Now I would like to replicate it in a C# program. I have this so far:
WebRequest req = WebRequest.Create(url);
((HttpWebRequest)req).UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2";
req.Method = "GET";
WebResponse response = req.GetResponse();
When I run it, it returns a "please login" page, as expected. But there is a problem with one of the parameters. This is the response URL:
http://page.com/login.html?ref=find/1,1,1,find.html?advanced=1¶m1=val1¶m2=Array
So, two questions: what might have happened to param2? And how do I add cookies to this?
EDIT: Managed to set the cookies by casting to HttpWebRequest.
As devio said you should use HttpWebRequest. I did dirty test to check it.
Prepare cookies to send. I made available for whole localhost:
HttpWebRequest rq = (HttpWebRequest)WebRequest.Create("http://localhost/test.php");
rq.CookieContainer = new CookieContainer();
rq.CookieContainer.Add(new Cookie("test", "xxxx", "/", "localhost"));
Your script should set cookies to make them available in response. And you could use them.
HttpWebResponse resp = (HttpWebResponse)rq.GetResponse();
foreach(var c in resp.Cookies)
{
Debug("{0}: {1}", c.Name, c.Value);
}
You can use this code piece:
Cookie SessionCookie = new Cookie("{CookieName}", {Cookievalue})
{
Domain = "{domain you want to hit}", Path = "/", Expired = false, HttpOnly = true
};
CookieContainer SessionCookieHolder = new CookieContainer();
SessionCookie.Add(SessionCookie);
try
{
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create("{URL}");
WebReq.CookieContainer = SessionCookie;
WebReq.Method = "GET/POST/HEAD"; //depending on the request type//
WebReq.KeepAlive = true;
HttpWebResponse Resp = (HttpWebResponse)WebReq.GetResponse();
}
catch(Exception ex)
{
string ExceptionReader = ex.Message;
}
Cookies are stored in the HttpWebRequest.CookieContainer and HttpWebResponse.Cookies properties.
For subsequent requests, you need to add the response's cookies to the request's cookie container.