We Keep Coding

C# Facing problem to read ajax data using web browser control

This is a web site https://www.wsj.com/news/types/newsplus whose data is getting loaded by ajax at runtime. i have to read all article title text. from morning i tried lots of code but still no code worked because data is getting load by ajax.
This is my code which i tried.
HtmlDocument hd = GetHtmlAjax(new Uri("https://www.wsj.com/news/types/newsplus"), 300, true);
ParseData(hd);
HtmlElementCollection main_element = hd.GetElementsByTagName("h3");
if (main_element != null)
{
foreach (HtmlElement element in main_element)
{
string cls = element.GetAttribute("className");
if (String.IsNullOrEmpty(cls) || !cls.Equals("WSJTheme--headline--unZqjb45 undefined WSJTheme--heading-3--2z_phq5h typography--serif-display--ZXeuhS5E"))
continue;
HtmlElementCollection childDivs = element.Children.GetElementsByName("a");
foreach (HtmlElement childElement in childDivs)
{
//grab links and other stuff same way
string linktxt = childElement.InnerText;
}
}
}
WebBrowser wb = null;
public HtmlDocument GetHtmlAjax(Uri uri, int AjaxTimeLoadTimeOut,bool loadurl)
{
if (loadurl)
{
wb = new WebBrowser();
wb.ScriptErrorsSuppressed = true;
wb.Navigate(uri);
}
while (wb.ReadyState != WebBrowserReadyState.Complete)
Application.DoEvents();
Thread.Sleep(AjaxTimeLoadTimeOut);
Application.DoEvents();
return wb.Document;
}
i follow many links to handle this issue but fail. these are the links i followed.
htmlagilitypack and dynamic content issue
Get HTML in C# from page that Loads Dynamic Data
Retrieve ajax/JavaScript return results from webpage in c#
How to extract dynamic ajax content from a web page
please some tell me what to change in my code to parse title link text. thanks
Post code from #aepot
private static HttpClient client = new HttpClient();
private static async Task<T> GetJsonPageAsync<T>(string url)
{
using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
{
response.EnsureSuccessStatusCode();
string text = await response.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<T>(text);
}
}
private async void button1_Click(object sender, EventArgs e)
{
try
{
dynamic newsList = await GetJsonPageAsync<dynamic>("https://www.wsj.com/news/types/newsplus?id={%22query%22:%22type:=\\%22NewsPlus\\%22%22,%22db%22:%22wsjie,blog,interactivemedia%22}&type=search_collection");
List<Task<dynamic>> tasks = new List<Task<dynamic>>();
foreach (dynamic item in newsList.collection)
{
string strUrl = "https://www.wsj.com/news/types/newsplus?id=" + item.id + "&type=article";
tasks.Add(GetJsonPageAsync<dynamic>(strUrl));
//tasks.Add(GetJsonPageAsync<dynamic>($"https://www.wsj.com/news/types/newsplus?id={item.id}&type=article"));
}
dynamic[] newsDataList = await Task.WhenAll(tasks);
foreach (dynamic newItem in newsDataList)
{
//Console.WriteLine(newItem.data.headline);
//Console.WriteLine(newItem.data.url);
txtData.Text += newItem.data.headline + System.Environment.NewLine;
txtData.Text += new string('-', 200); + System.Environment.NewLine;
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}

AJAX is simple GET or POST request.
Using regular Browser dev tools I've found that page sends simple GET request and receive JSON data. JSON can be deserealized or explored via reader.
For JSON parsing i used Newtonsoft.Json NuGet package
Here's simple example based on WinForms app.
public partial class Form1 : Form
{
private static readonly HttpClient client = new HttpClient();
private async Task<T> GetJsonPageAsync<T>(string url)
{
using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
{
response.EnsureSuccessStatusCode();
string text = await response.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<T>(text);
}
}
public Form1()
{
InitializeComponent();
ServicePointManager.DefaultConnectionLimit = 10; // to make it faster
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
}
private async void button1_Click(object sender, EventArgs e)
{
try
{
dynamic newsList = await GetJsonPageAsync<dynamic>("https://www.wsj.com/news/types/newsplus?id={%22query%22:%22type:=\\%22NewsPlus\\%22%22,%22db%22:%22wsjie,blog,interactivemedia%22}&type=search_collection");
List<Task<dynamic>> tasks = new List<Task<dynamic>>();
foreach (dynamic item in newsList.collection)
{
tasks.Add(GetJsonPageAsync<dynamic>($"https://www.wsj.com/news/types/newsplus?id={item.id}&type=article"));
}
dynamic[] newsDataList = await Task.WhenAll(tasks);
foreach (dynamic newItem in newsDataList)
{
textBox1.Text += newItem.data.headline + Environment.NewLine;
textBox1.Text += new string('-', 200) + Environment.NewLine;
}
}
catch (Exception ex)
{
textBox1.Text = ex.Message;
}
}
}
UPD: Added fix for .NET Framework 4.5.2

Xamarin Forms Play a sound Asynchronously

I can successfully play sounds using Xamarin forms (Android and iOS) however I also need to achieve the following:
I need to await so that if multiple sounds are 'played', one will complete before the next.
I need to return a boolean to indicate whether operation was a success.
Here is my current simplified code (for the iOS platform):
public Task<bool> PlayAudioTask(string fileName)
{
var tcs = new TaskCompletionSource<bool>();
string filePath = NSBundle.MainBundle.PathForResource(
Path.GetFileNameWithoutExtension(fileName), Path.GetExtension(fileName));
var url = NSUrl.FromString(filePath);
var _player = AVAudioPlayer.FromUrl(url);
_player.FinishedPlaying += (object sender, AVStatusEventArgs e) =>
{
_player = null;
tcs.SetResult(true);
};
_player.Play();
return tcs.Task;
}
To test the method, I have tried calling it like so:
var res1 = await _audioService.PlayAudioTask("file1");
var res2 = await _audioService.PlayAudioTask("file2");
var res3 = await _audioService.PlayAudioTask("file3");
I had hoped to hear the audio for file1, then file2, then file3. However I only hear file 1 and the code doesn't seem to reach the second await.
Thankyou

I think your issue here is that the AVAudioPlayer _player was being cleared out before it was finished. If you were to add debugging to your FinsihedPlaying, you'll notice that you never hit that point.
Try these changes out, I made a private AVAudioPlayer to sit outside of the Task
(I used the following guide as a reference https://developer.xamarin.com/recipes/ios/media/sound/avaudioplayer/)
public async void play()
{
System.Diagnostics.Debug.WriteLine("Play 1");
await PlayAudioTask("wave2.wav");
System.Diagnostics.Debug.WriteLine("Play 2");
await PlayAudioTask("wave2.wav");
System.Diagnostics.Debug.WriteLine("Play 3");
await PlayAudioTask("wave2.wav");
}
private AVAudioPlayer player; // Leave the player outside the Task
public Task<bool> PlayAudioTask(string fileName)
{
var tcs = new TaskCompletionSource<bool>();
// Any existing sound playing?
if (player != null)
{
//Stop and dispose of any sound
player.Stop();
player.Dispose();
}
string filePath = NSBundle.MainBundle.PathForResource(
Path.GetFileNameWithoutExtension(fileName), Path.GetExtension(fileName));
var url = NSUrl.FromString(filePath);
player = AVAudioPlayer.FromUrl(url);
player.FinishedPlaying += (object sender, AVStatusEventArgs e) =>
{
System.Diagnostics.Debug.WriteLine("DONE PLAYING");
player = null;
tcs.SetResult(true);
};
player.NumberOfLoops = 0;
System.Diagnostics.Debug.WriteLine("Start Playing");
player.Play();
return tcs.Task;
}

Repeat async task by timer in xamarin forms

i'm new to xamarin.forms development and i'm still having my first steps from the few tutorials that are found on the net.
I have an async task that returns the time from date.jsontest.com
and i have a timer that decrements a text in a label.
i want to put the async task in the timer so that it repeats itself and displays the time on the label however im getting cannot convert async lamba to func
here's my code please help me, thanks
static async Task<string> RequestTimeAsync()
{
using (var client = new HttpClient())
{
var jsonString = await client.GetStringAsync("http://date.jsontest.com/");
var jsonObject = JObject.Parse(jsonString);
return jsonObject["time"].Value<string>();
}
}
protected override async void OnAppearing()
{
base.OnAppearing();
timeLabel.Text = await RequestTimeAsync();
Device.StartTimer(TimeSpan.FromSeconds(1), () => { // i want the taks to be put
//here so that it gets repeated
var number = float.Parse(button.Text) - 1;
button.Text = number.ToString();
return number > 0;
});
}
Reloading the Content Page in the timer would do the trick, so if anybody can please help me it would be appreciated

You just need to wrap the async method call in a Task.Run(), for example:
Device.StartTimer(TimeSpan.FromSeconds(1), () =>
{
Task.Run(async () =>
{
var time = await RequestTimeAsync();
// do something with time...
});
return true;
});

Solved
as simple as that
private async void ContinuousWebRequest()
{
while (_keepPolling)
{
timeLabel.Text = "call: "+counter+" "+ await RequestTimeAsync()+Environment.NewLine;
// Update the UI (because of async/await magic, this is still in the UI thread!)
if (_keepPolling)
{
await Task.Delay(TimeSpan.FromSeconds(5));
}
}
}
static async Task<string> RequestTimeAsync()
{
using (var client = new HttpClient())
{
var jsonString = await client.GetStringAsync("http://date.jsontest.com/");
var jsonObject = JObject.Parse(jsonString);
TimePage.counter++;
return jsonObject["time"].Value<string>();
}
}

A simple clock demonstrates a one second timer action:
Device.StartTimer(TimeSpan.FromSeconds(1), doitt);
bool doitt()
{
label1.Text = DateTime.Now.ToString("h:mm:ss");
return true;
}

or you can try this:
static async Task<string> RequestTimeAsync()
{
using (var client = new HttpClient())
{
var jsonString = await client.GetStringAsync("http://date.jsontest.com/");
var jsonObject = JObject.Parse(jsonString);
return jsonObject["time"].Value<string>();
}
}
protected override async void OnAppearing()
{
base.OnAppearing();
var Text = await RequestTimeAsync();
Device.StartTimer(TimeSpan.FromSeconds(1), () => {
// i want the taks to be put
//here so that it gets repeated
var jsonDateTsk = RequestTimeAsync();
jsonDateTsk.Wait();
var jsonTime = jsonDateTsk.Result;
var number = float.Parse(Text) - 1;
var btnText = $"{number}";
return number > 0;
});
}

Waiting on the results of GetGeopositionAsync()

I'm developing a WinPhone 8 App.
On this App there is a Button 'Send SMS'.
When the user clicks on this button two things should happen:
(Method A) Get the geo-coordinate of the current Location (using Geolocator and GetGeopositionAsync).
(Method B) Compose and send an SMS with the geo-coordinate as part of the body.
The Problem: GetGeopositionAsync is an asynchronous method. Before the coordinate is detected (which takes a few seconds) the SMS is sent (of course with no coordinates).
How can I tell Method 2 to wait until the coordinates are available?
OK, here is my code:
When the user presses the button, the coordinates are determined by the first method and the second method sends the SMS which includes the coordinates in its body:
private void btnSendSms_Click(object sender, RoutedEventArgs e)
{
GetCurrentCoordinate(); // Method 1
// -> Gets the coordinates
SendSms(); // Method 2
// Sends the coordinates within the body text
}
The first method GetCurrentCoordinate() looks as follows:
...
private GeoCoordinate MyCoordinate = null;
private ReverseGeocodeQuery MyReverseGeocodeQuery = null;
private double _accuracy = 0.0;
...
private async void GetCurrentCoordinate()
{
Geolocator geolocator = new Geolocator();
geolocator.DesiredAccuracy = PositionAccuracy.High;
try
{
Geoposition currentPosition = await geolocator.GetGeopositionAsync(
TimeSpan.FromMinutes(1),
TimeSpan.FromSeconds(10));
lblLatitude.Text = currentPosition.Coordinate.Latitude.ToString("0.000");
lblLongitude.Text = currentPosition.Coordinate.Longitude.ToString("0.000");
_accuracy = currentPosition.Coordinate.Accuracy;
MyCoordinate = new GeoCoordinate(
currentPosition.Coordinate.Latitude,
currentPosition.Coordinate.Longitude);
if (MyReverseGeocodeQuery == null || !MyReverseGeocodeQuery.IsBusy)
{
MyReverseGeocodeQuery = new ReverseGeocodeQuery();
MyReverseGeocodeQuery.GeoCoordinate = new GeoCoordinate(
MyCoordinate.Latitude,
MyCoordinate.Longitude);
MyReverseGeocodeQuery.QueryCompleted += ReverseGeocodeQuery_QueryCompleted;
MyReverseGeocodeQuery.QueryAsync();
}
}
catch (Exception)
{ // Do something }
}
private void ReverseGeocodeQuery_QueryCompleted(object sender,
QueryCompletedEventArgs<IList<MapLocation>> e)
{
if (e.Error == null)
{
if (e.Result.Count > 0)
{
MapAddress address = e.Result[0].Information.Address;
lblCurrAddress.Text = address.Street + " " + address.HouseNumber + ",\r" +
address.PostalCode + " " + address.City + ",\r" +
address.Country + " (" + address.CountryCode + ")";
}
}
}
}
And the Methode 'SendSms()':
private void SendSms()
{
SmsComposeTask smsComposeTask = new SmsComposeTask();
smsComposeTask.To = "0123456";
smsComposeTask.Body = "Current position: \rLat = " + lblLatitude.Text +
", Long = " + lblLongitude.Text +
"\r" + lblCurrAddress.Text;
// -> The TextBoxes are still empty!
smsComposeTask.Show();
}
The problem is, that all these TextBoxes (lblLatitude, lblLongitude, lblCurrAddress) are still empty when the method SendSms() sets the SmsComposeTask object.
I have to ensure that the TextBoxes are already set BEFORE the method SendSms() starts.

You should almost never mark a method async void unless it's a UI event handler. You're calling an asynchronous method without waiting for it to end. You are basically calling those 2 methods in parallel, so it's clear why the coordinates aren't available.
You need to make GetCurrentCoordinate return an awaitable task and await it, like this:
private async Task GetCurrentCoordinateAsync()
{
//....
}
private async void btnSendSms_Click(object sender, RoutedEventArgs e)
{
await GetCurrentCoordinateAsync();
// You'll get here only after the first method finished asynchronously.
SendSms();
}

This is one of the primary reasons you should avoid async void. void is a very unnatural return type for async methods.
First, make your GetCurrentCoordinate an async Task method instead of async void. Then, you can change your click handler to look like this:
private async void btnSendSms_Click(object sender, RoutedEventArgs e)
{
await GetCurrentCoordinate();
SendSms();
}
Your click handler is async void only because event handlers have to return void. But you should really strive to avoid async void in all other code.

There two things you're doing wrong here:
Using void returning async methods when you need to await on them. This is bad because you can't await on execution of these methods and should only be used when you can't make the method return Task or Task<T>. That's why you're not seeing anything on the text boxes when SendSmsis called.
Mixing UI and non-UI code. You should transfer data between UI and non-UI code to avoid tight coupling between code with different responsibilities. IT also makes it easy to read and debug the code.
ReverseGeocodeQuery does not have an awaitable async API but you can easily make your own:
private async Task<IList<MapLocation>> ReverseGeocodeQueryAsync(GeoCoordinate geoCoordinate)
{
var tcs = new TaskCompletionSource<IList<MapLocation>>();
EventHandler<QueryCompletedEventArgs<IList<MapLocation>>> handler =
(s, e) =>
{
if (e.Cacelled)
{
tcs.TrySetCancelled();
}
else if (e.Error != null)
{
tcs.TrySetException(e.Error);
}
else
{
tcs.TrySetResult(e.Result);
}
};
var query = new ReverseGeocodeQuery{ GeoCoordinate = geoCoordinate };
try
{
query.QueryCompleted += handler;
query.QueryAsync();
return await tcs.Task;
}
finally
{
query.QueryCompleted -= handler;
}
}
This way you'll get full cancellation and error support.
Now let's make the retrieval of the geo coordinate information all in one chunk:
private async Task<Tuple<Geocoordinate, MapLocation>> GetCurrentCoordinateAsync()
{
try
{
var geolocator = new Geolocator
{
DesiredAccuracy = PositionAccuracy.High
};
var currentPosition = await geolocator.GetGeopositionAsync(
TimeSpan.FromMinutes(1),
TimeSpan.FromSeconds(10))
.ConfigureAwait(continueOnCapturedContext: false);
var currentCoordinate = currentPosition.Coordinate;
var mapLocation = await this.ReverseGeocodeQueryAsync(
new GeoCoordinate(
currentCoordinate.Latitude,
currentCoordinate.Longitude));
return Tuple.Create(
currentCoordinate,
mapLocation.FirstOrDefault());
}
catch (Exception)
{
// Do something...
return Tuple.Create(null, null);
}
}
Now the button eventnt handler becomes much more readable:
private void btnSendSms_Click(object sender, RoutedEventArgs e)
{
var info = await GetCurrentCoordinate();
if (info.Item1 != nuil)
{
lblLatitude.Text = info.Item1.Latitude.ToString("0.000");
lblLongitude.Text = info.Item1.Longitude.ToString("0.000");
}
if (info.Item2 != null)
{
var address = info.Item2.Information.Address;
lblCurrAddress.Text = string.Format(
"{0} {1},\n{2} {3},\n{4} ({5})",
address.Street,
address.HouseNumber,
address.PostalCode,
address.City,
address.Country,
address.CountryCode);
}
SendSms(info.Item1, info.Item2);
}
Does this make sense?

Waiting for async operation to complete

I have a question about how asynchronous methods work on c#. I am trying to get the convertAddressToCoordinate method to set the myLocation variable to a GeoCoordinate. However, the compareDistance method is called even before the myLocation value is set. How can I ensure that the myLocation value is not null before I call compareDistance()?
public GeoCoordinate myLocation = null;
public void returnClosestCurrent(string address)
{
convertAddressToCoordinate(address);
compareDistance(myLocation);
}
public void convertAddressToCoordinate(string add)
{
WebClient wc = new WebClient();
wc.DownloadStringCompleted += wc_DownloadStringCompleted;
wc.DownloadStringAsync(new Uri("http://maps.googleapis.com/maps/api/geocode/json?address=1600+bay+st&sensor=false"));
}
void wc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
XDocument xdoc = XDocument.Parse(e.Result, LoadOptions.None);
var data = from query in xdoc.Descendants("location")
select new Location
{
lat = (string)query.Element("lat"),
lng = (string)query.Element("lng")
};
GeoCoordinate destinationGeo = new GeoCoordinate(Convert.ToDouble(data.ElementAt(0).lat), Convert.ToDouble(data.ElementAt(0).lng));
myLocation = destinationGeo;
}

you can use the await keyword for getting values of async task
http://msdn.microsoft.com/en-us/library/vstudio/hh156528.aspx

Looking at your code, seems to me that the compareDistance(myLocation); should be called only in the end of the wc_DownloadStringCompleted method.
After that change, everything should work fine.

You would want to await the wc.DownloadStringAsync
public async Task returnClosestCurrent(string address)
{
await convertAddressToCoordinate(address)
.ContinueWith(t => compareDistance(myLocation));
}
public async Task convertAddressToCoordinate(string add)
{
WebClient wc = new WebClient();
wc.DownloadStringCompleted += wc_DownloadStringCompleted;
await wc.DownloadStringAsync(new Uri("http://maps.googleapis.com/maps/api/geocode/json?address=1600+bay+st&sensor=false"));
}
The pattern with async/await is the code tends to turn everything into async/await
It could be refactored to use WebClient.DownloadStringTaskAsync
public async Task ReturnClosestCurrent(string address)
{
await convertAddressToCoordinate(address)
.ContinueWith(t => compareDistance(t.Result));
}
public async Task<GeoCoordinate> ConvertAddressToCoordinate(string add)
{
WebClient wc = new WebClient();
wc.DownloadStringCompleted += wc_DownloadStringCompleted;
var content = await wc.DownloadStringTaskAsync(new Uri("http://maps.googleapis.com/maps/api/geocode/json?address=1600+bay+st&sensor=false"));
return ParseContent(content);
}
private GeoCoordinate ParseContent(string content)
{
XDocument xdoc = XDocument.Parse(content, LoadOptions.None);
var data = from query in xdoc.Descendants("location")
select new Location
{
lat = (string)query.Element("lat"),
lng = (string)query.Element("lng")
};
GeoCoordinate destinationGeo = new GeoCoordinate(Convert.ToDouble(data.ElementAt(0).lat), Convert.ToDouble(data.ElementAt(0).lng));
return destinationGeo;
}