Google Finance, how to get the JSON data streamed? - c#

I tried to explain this earlier, but obviously failed!
So, if you have a google finance graph open, for instance:
http://www.google.com/finance?q=INDEXNASDAQ:.IXIC
I would like to somehow use the (HttpWebRequest) object in C# so that I can grab the small data which google sends to the page to update the graph.
A friend mentioned this was JSON?
I was trying to use the following code example, but even when i set the keep alive property to 'true', it still wouldnt work:
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.keepalive.aspx#Y369

You also need to change the example's line that sets the Connection property to Close. Comment out this line (along with keeping the keep-alive property set to true):
myHttpWebRequest2.Connection = "Close";
You do that and your example should run fine.
Regarding getting the data and using HttpWebRequest to work with it, you can do that. The data returned isn't JSON - it looks like straight text and I'm guessing Google's javascript is parsing it out. (I haven't inspected the javascript on Google Finance's page, but that's my guess.)
Using Fiddler, the response from this URL:
http://www.google.com/finance/getprices?q=.IXIC&x=INDEXNASDAQ&i=120&p=10m&f=d,c,v,o,h,l&df=cpct&auto=1&ts=1307994768643
looks like this:
EXCHANGE%3DINDEXNASDAQ
MARKET_OPEN_MINUTE=570
MARKET_CLOSE_MINUTE=960
INTERVAL=120
COLUMNS=DATE,CLOSE,HIGH,LOW,OPEN,VOLUME
DATA=
TIMEZONE_OFFSET=-240
a1307994120,2641.12,2641.12,2639.96,2640.01,0
1,2638.76,2642.14,2638.76,2641.13,0
2,2638.95,2640.54,2638.74,2638.79,0
3,2639.85,2640.01,2638.08,2638.95,0
4,2640.07,2640.87,2639.31,2639.88,0
5,2640.31,2640.48,2639.42,2640.08,0
6,2641.09,2641.09,2640.3,2640.31,0
A little cryptic, but you can see how the COLUMNS line lines up with the data at the bottom. Also, the f querystring parameter seems to be indicating which columns to return (d=date, c=close,v=volume,o=open,h=high,l=low).
EDIT: I should mention that the URL I used is being sent from the finance graph page to get updated data - you can see this URL being requested at regular intervals using a tool like Fiddler. The response data that I pasted above is also output by the sample application from MSDN.
But commenting out that one line in the example from MSDN and a little fiddling with Fiddler should give you the data and clues you need to parse the return that comes from that URL.
I hope this helps!
PS - my first line in my modified MSDN example looks like this:
HttpWebRequest myHttpWebRequest1 = (HttpWebRequest)WebRequest.Create("http://www.google.com/finance/getprices?q=.IXIC&x=INDEXNASDAQ&i=120&p=10m&f=d,c,v,o,h,l&df=cpct&auto=1&ts=1307994768643");
I made a similar change to the other WebRequest call a little further down in the example...other than that, I didn't change anything else in the example.

Related

Trying to send pagination information in response object but only passing through data property c#

I have been attempting to send pagination information to the front end from a c# backend. I have found this tutorial that I have followed directly: https://codewithmukesh.com/blog/pagination-in-aspnet-core-webapi/
I am using ASP.NET from my understanding and the class I'm working with extends ControllerBase.
I am able to get my backend setup and create the Response and PagedResponse wrapper classes that have been introduced. I don't go further as I don't have a need for the additional features the guide provides.
Just to clarify what I'd like, a response object looks like this when I sent it to the frontend:
Snippet of Response Object
RESPONSE OBJECT
{
config
data
headers
request
status
statusText
}
Usually inside of the data property of this object you would see the actual data that the front end is interested. By following the tutorial I was expecting to learn how to add a property to the response object "pageCount", and result in something like this:
RESPONSE OBJECT
{
config
data
headers
request
status
statusText
pageCount
}
However, the actual result has the same response object, but with a different data property. Inside of the data property, this is what it looks like:
DATA PROPERTY
{
data
errors
message
pageCount
succeeded
}
I don't like this approach as the data property of the response object has additional information inside of it that's not the data. I'd instead like to be able to add the page count to response. Ideally the front end would be able to get pages from assigning a variable like 'response' to the response object and doing something like response.pageCount to determine the available pages as well as response.data to get the actual data.
I have been searching for a way to add the information to the response object and haven't found any luck. I did find a solution talking about adding it to headers but that's not where I want to add it. Additionally, I did find some outdated code that no longer works on my version.
Ideally, once I find a solution to send the pageCount back, the whole response and data structure will look like this:
RESPONSE OBJECT
{
config
data:
[Array of Items] (No additional information inside of data)
headers
request
status
statusText
pageCount
}
Please let me know if any additional information is needed. Any help is appreciated! Thank you in advance!
EDIT Additional Code & Information:
[HttpGet("[action]")]
public IActionResult GetAll([FromHeader(Name = "itemsPerPage")] int? itemsPerPage, [FromHeader(Name = "sortBy")] string? sortBy, [FromHeader(Name = "page")] int? page) {
List<Things> things = repo.GetAll(itemsPerPage, sortBy, page);
return new JsonResult(new PagedResponse<List<Things>>(things, things.Count));
}
If you were to console.log() a regular response from the backend on the front end without any pagination information, the response object would look like this
Snippet of Items
This is what I'm calling the "Response" Object.
As you can see there is a data property on this Response Object:
enter image description here
And as you can see from the picture, it is an array of items. There is no additional information inside of the data property. An example of something extra would be pagination information like "PageCount". I do not want the "Page Count" property inside of the data property. However, when following the guide I linked above, that is the result I get.
Here is what I would like my response object to look like:
Snippet of Ideal Response Object
As you can see the PageCount property is a property of the Response Object, and not inside of the data property.
Now this is what my response object looks like after I what I've attempted through the tutorial:
Snippet of Response after finishing tutorail
As you can see inside of the data property, there is another data property. You can see that the pagesCount is inside of the top level data property. And the additional attributes such as errors and message are additional properties that the tutorial added.
Hopefully that gives you an idea of what I've done and how the result is not what I would've expected.
So my direct question right now is either How can I overwrite the Response Object completely so that I can directly control what it looks like including adding a pageCount property. Or Alternatively, how can I add a single property pageCount to the already existing response object?
#Andrew Williamson was the one who really answered my question. He pointed out that the response Object I was assuming was coming from the backend was actually not the response. And the information inside of the data property was what was actually being returned by the backend.
I've discovered that the reason I'm seeing that wrapper object is actually because I'm using axios through npm to access responses.
Basically the Axios NPM package describes the Response Schema:
https://www.npmjs.com/package/axios#response-schema
This shows the response object I was looking at.
The solution that I settled on was using response headers to add the page count there.
To add the response headers, I did this code:
public List<Things> GetAll([FromHeader(Name = "itemsPerPage")] int? itemsPerPage, [FromHeader(Name = "sortBy")] string? sortBy, [FromHeader(Name = "page")] int? page) {
List<Things> things = repo.GetAll(itemsPerPage, sortBy, page);
Response.Headers.Add("Pages", things.Count.ToString());
return things;
}
And this is how I accessed the pages header on the front end using AXIOS
useEffect(() => {
endPointServiceMethod(query).then((response: any) => {
console.log('Pages Headers');
console.log(response.headers['pages']);
console.log('Data');
console.log(response.data);
setItems(response);
});
}, []);
Once again, thank you Andrew Williamson for pointing out the answer!

Gathering and Filtering Necessary Post Data for a HTTP Post Requests

I hope you're all well. I am trying to use C# and the request.post command to autofill a login page. I've shown the snippet of code I'm dealing with below.
My main concern currently is the 'POST DATA GOES HERE' .I'm unsure how I can gather the post-data as well as what syntax / format i would need to put it in.
I've already tried using Fiddler and Charles Proxy (HTTP Debuggers) to find anything about PostData . I did find some things I needed : User-Agent / Success Keys but those currently aren't relevant.
//Send a Post request to the (URL, postdata, contenttype) - store the HTTP Response into Response
var Response = Request.Post(LOGIN_URL, "{post data goes here}", "content type goes here `application/java`");

How can i edit a HTTP a request C# using fiddlercore

What I want to be able to do: Edit HTTP Requests before they are sent off to the server
User navigates to a webpage of their choice in their browser > They encounter a request they wish to edit > they edit the request and then that gets sent to the server instead of the original one.
What I have done so far: I have captured the request, now I need help finding the code to edit it. Here is my code for capturing the request so far:
Fiddler.FiddlerApplication.BeforeRequest += sess =>
{
//Code to detect user specified URL here
}
Is it possible for me to edit the request before it is actually sent? If it can be done using the FiddlerCore API only then I'd be grateful, although I am willing to download more binaries if required.
Additional notes: I have tried streamwriters, binary writers, copy the respose into a memory stream edit it then copy it back, none of those methods work for me. Also when I try some methods my app just hangs and doesn't respond to things like pressing the X.
Maybe I'm just bad at explaining what I'm trying to achieve seems the only good answer I have has been about reponses :/
If the request reads the string "hello world" then I'd like the user to be able to change the REQUEST to say "hello there"
Such a noobish mistake I made, I thought that RequestBody was read only! Turns out I could have simply edited the response like this:
session.RequestBody = myBytes;
Really annoyed at myself for this!
In the demo app, adding the delegate is shown as:
Fiddler.FiddlerApplication.BeforeResponse += delegate(Fiddler.Session oS) {
// Console.WriteLine("{0}:HTTP {1} for {2}", oS.id, oS.responseCode, oS.fullUrl);
// Uncomment the following two statements to decompress/unchunk the
// HTTP response and subsequently modify any HTTP responses to replace
// instances of the word "Microsoft" with "Bayden". You MUST also
// set bBufferResponse = true inside the beforeREQUEST method above.
//
//oS.utilDecodeResponse(); oS.utilReplaceInResponse("Microsoft", "Bayden");
};

An image calls an ASP utility. How do I do that without image in ASP.NET (C#)

I have a store (C#) that calls our "back office(ASP classic)" via a one-pixel image with some parameters to record a sale. It looks like this:
<asp:Image ID="BOImageLink" runat="server" ImageUrl="https://backoffice.mysite.com/Import.asp?TicketType=Import"></asp:Image>
The store's code-behind appends other parameters to the ImageUrl as needed.
About once per day someone gets through the system without triggering the image tag link. My task is to make the call to that script from our store which is ASP.NET C# code without using an image tag and I just don't know how to do it. All help appreciated, answering accordingly knowing I am new to .NET is doubly so.
That's a really random way of doing that!
Why don't you do a web request in code instead?
http://msdn.microsoft.com/en-us/library/5t9y35bd.aspx
Would be something like this:
WebRequest request = WebRequest.Create ("https://backoffice.mysite.com/Import.asp?TicketType=Import");
using this to get back any response:
HttpWebResponse response = (HttpWebResponse)request.GetResponse ();

Determine Final Destination of a Shortened URL

I'm trying to find the best way (in code) to determine the final destination of a shortened URL. For instance http://tinyurl.com redirects to an eBay auction. I'm trying to get the URL for the eBay auction. I'm trying to do this from within .NET so I can compare multiple URLs to ensure that there is no duplicates.
TIA
While I spent a minute writing the code to ensure that it worked the answer was already delivered, but I post the code anyway:
private static string GetRealUrl(string url)
{
WebRequest request = WebRequest.Create(url);
request.Method = WebRequestMethods.Http.Head;
WebResponse response = request.GetResponse();
return response.ResponseUri.ToString();
}
This will work as long as the short url service does a regular redirect.
You should issue a HEAD request to the url using a HttpWebRequest instance. In the returned HttpWebResponse, check the ResponseUri.
Just make sure the AllowAutoRedirect is set to true on the HttpWebRequest instance (it is true by default).
One way would be to read the URL and get the result code from it. If it's a 301 (permanent redirect) then follow where it's taking you. Continue to do this until you reach a 200 (OK). When using tinyurl it could happen that you will go through several 301 until you reach a 200.
Assuming you don't want to actually follow the link, for TinyURL you can append /info to the end of the url:
http://tinyurl.com/unicycles/info
and it gives you a page showing where that tinyurl links to, which I assume would be easy to parse using xpath or similar.
Most other URL shortening services have similar features, but they all work differently.

Categories