I am working on an application which requires to fetch entities from one system (Contacts from Dynamics CRM). The app does some process on those contacts, basically convert to compatible payload for other system and then sync it back.
The Dynamics CRM has over 8000 contacts so no brainer, I went ahead with pagination approach.. The code looks something like this -
string nextPageUrl = string.Empty;
bool nextPageAvailable = true;
int totalContactsCount = 0;
bool firstCall = true;
JArray totalContacts = new JArray();
while (nextPageAvailable)
{
string contacts = string.Empty;
if (string.IsNullOrEmpty(nextPageUrl) && firstCall)
{
contacts = await _genericService.GetEntity("contacts", pageSize: 100);
firstCall = false;
}
else
{
contacts = await _genericService.GetEntityFromPaginationUrl(nextPageUrl, pageSize: 100);
}
var contactsObj = JObject.Parse(contacts);
var contactArr = (JArray)contactsObj["value"];
var contactsLength = contactArr.Count;
foreach(var contactObject in contactArr)
{
totalContacts.Add(contactObject);
}
var contactsCt = totalContacts.Count;
nextPageUrl = contactsObj["#odata.nextLink"].IsNullOrEmpty() ? string.Empty : contactsObj["#odata.nextLink"].ToString();
if(string.IsNullOrEmpty(nextPageUrl) && !firstCall)
{
nextPageAvailable = false;
}
Console.WriteLine(nextPageUrl);
}
This works perfectly but till I get 8000 contacts (100 at a time), the memory utilization is almost over 370mb. I am sure there is a better way to achieve this but can not think of any.
The memory problem will get more serious when I reiterate through totalContacts which will be having over 8000 contacts and process them.
Since the contacts coming from Dynamics CRM may have newly added properties, I can not strongly type the objects with models. This is why I have decided to go with JArray. Is there any better alternative to achieve this?
Thanks in advance.
Related
I am working on a small C# application using Woocommerce.net library to make RESP API calls. My goal here is to retrieve all products from my online store which I have done. However, while the products are being retrieved, I want to show the user progress with a progress bar preferably. Here is the code I am using for retrieving products.
RestAPI rest = new RestAPI("https://www.store.ca/wp-json/wc/v3/", "key", "key");
WCObject wc = new WCObject(rest);
//Get all products
products = new List<Product>();
Dictionary<string, string> dic1 = new Dictionary<string, string>();
dic1.Add("per_page", "100");
int pageNumber1 = 1;
dic1.Add("page", pageNumber1.ToString());
bool endWhile1 = false;
while (!endWhile1)
{
var productsTemp = await wc.Product.GetAll(dic1);
if (productsTemp.Count > 0)
{
products.AddRange(productsTemp);
pageNumber1++;
dic1["page"] = pageNumber1.ToString();
}
else
{
endWhile1 = true;
}
}
Is there any way possible to report back progress for such call. I couldn't find anything related to this. I came across IProgress<int> but I believe that would not work in this scenario.
Any ideas?
Any help is appreciated.
Thanks in advance.
You read objects 'per page'.
1.If you can get product count please do it first (version 3 had count method)
2. progress step will be (page number * 100 items) which you can notify right after addRange
I am a bit new to NAV and so far I have published web services on NAV and have been able to consume these SOAP web services using C#.
Now, the data has increased and its taking longer to load. I have an idea of querying the data in chunks (e.g chunks of 10) using Datatables, but this I am yet too figure out how to set limits and offsets.
Here is my C# code to read the NAV soap service
public string getItemCardList(itemCardService_Service itemCardServiceObj, List<itemCardService_Filter> filter)
{
serializer.MaxJsonLength = 50000000;
return serializer.Serialize(itemCardServiceObj.ReadMultiple(filter.ToArray(), null, 0));
}
After some several searches I have gotten the answer on the [MSDN Website][https://msdn.microsoft.com/en-us/library/ff477110.aspx] and modified it to work for me.
public string holder()
{
const int fetchSize = 10;
string bookmarkKey = null;
List<itemCardService> itemList = new List<itemCardService>();
//Read items data in pages of 10
itemCardService[] results = itemCardServiceObj.ReadMultiple(filter.ToArray(), bookmarkKey, fetchSize);
while(results.Length > 0)
{
bookmarkKey = results.Last().Key;
itemList.AddRange(results);
results = itemCardServiceObj.ReadMultiple(filter.ToArray(), bookmarkKey, fetchSize);
}
serializer.MaxJsonLength = 50000000;
return serializer.Serialize(itemList);
}
Hello SO folks and more specifically Google folks monitoring this tag per your support page. I am working from .NET and PlaylistItems.List("snippet,contentDetails") does not do a whole lot compared to the old RSS Feed search. In fact adding part contentDetails adds little value in that only the VideoID is now returned but it is already part of Snippet.ResourceId.VideoId
"kind": "youtube#playlistItem",
bla,
bla,
"contentDetails": {
"videoId": "DLME0PsJRnk"
}
Why add a "part" which is only going to return one bit of information?
How about supporting something like "snippet,contentDetails(duration,PublishedAt,Views)"
I feel this is kind of basic metadata (snippet) most apps would want to list to the users.
While you are at it please please remove this non-sense of Java casing of parameters. Why would you leak-out your language of choice into an API, that's really sad. Yes it is frustrating to keep checking whether I case-spelled them correctly.
Well, it looks like you are forcing "us" to build a list of VideoIds than turn around and make more API calls when I was doing it previously with fewer.
It also means, I will have to manage the 50 items max paging twice, once for the playlist if it is over 50 videos and then manage manually my list of VideosIds paging when I turn around to make Videos.List calls.
Let me know if I missed an All-In-One call type of API, thank you.
Here is what I have now working, let me know if there is a better way
// 20150802
public async Task<List<YouTubeInfo>> PlaylistVideosInfo(String PlaylistID)
{
var YoutubeService = YouTubeService();
//
List<YouTubeInfo> VideoInfos = new List<YouTubeInfo>();
//
var NextPageToken = "";
while (NextPageToken != null)
{
//
var SearchListRequest = YoutubeService.PlaylistItems.List("snippet");
SearchListRequest.PlaylistId = PlaylistID;
SearchListRequest.MaxResults = 50;
SearchListRequest.PageToken = NextPageToken;
// Call the search.list method to retrieve results matching the specified query term.
var SearchListResponse = await SearchListRequest.ExecuteAsync();
// Collect Video IDs from this page
var VideoIDsBatch = new List<string>(); // batch Video detail search by 50 max
foreach (var searchResult in SearchListResponse.Items)
{
VideoIDsBatch.Add(searchResult.Snippet.ResourceId.VideoId);
}
// Make API call for this batch - expect a single page :(
var VideoListRequest = YoutubeService.Videos.List("snippet,contentDetails");
VideoListRequest.Id = String.Join(",", VideoIDsBatch);
VideoListRequest.MaxResults = 50;
var VideoListResponse = await VideoListRequest.ExecuteAsync();
// Collect each Video details
foreach (var VideoResult in VideoListResponse.Items)
{
YouTubeInfoAdd(VideoInfos, VideoResult);
}
// request next page
NextPageToken = SearchListResponse.NextPageToken;
}
// Return All Videos' detail
return VideoInfos;
}
In ServiceNow, I am able to get only a maximum of 250 records in a SOAP request. How to get all the records?
Web Reference Url = https://*****.service-now.com/rm_story.do?WSDL
Code:
var url = "https://*****.service-now.com/rm_story.do?SOAP";
var userName = *****;
var password = *****;
var proxy = new ServiceNow_rm_story
{
Url = url,
Credentials = new NetworkCredential(userName, password)
};
try
{
var objRecord = new Namespace.WebReference.getRecords
{
// filters..
};
var recordResults = proxy.getRecords(objRecord);
}
catch (Exception ex)
{
}
In recordResults, I am getting only 250 records. How to get all the records ?
Also see this stack overflow answer which provides info.
Get ServiceNow Records Powershell - More than 250
Note that returning a large number of records can affect performance of the response and it may be more efficient to process your query in batches using offsets (i.e., get 1-100, then 101-200, ...). This can be achieved by using a sort order and offset. The ServiceNow REST Table API actually returns link headers from Get requests providing you links for the first, next and last set of records making it easy to know the url to query the next batch of records.
See: http://wiki.servicenow.com/index.php?title=Table_API#Methods
and look under 'Response Header'.
Have u tried to pass/override __limit parameter?
Google / wiki / Users manual / Release notes are always helpful
In your code snippet in line where it says //filter you should define __limit (and potentially __first_row and __last_row as explained in the example bellow)
int Skip = 0;
int Take = 250;
while (true)
{
using (var soapClient = new ServiceNowLocations.ServiceNow_cmn_location())
{
var cred = new System.Net.NetworkCredential(_user, _pass);
soapClient.Credentials = cred;
soapClient.Url = _apiUrl + "cmn_location.do?SOAP";
var getParams = new ServiceNowLocations.getRecords()
{
__first_row = Skip.ToString(),
__last_row = (Skip + Take).ToString(),
__limit = Take.ToString()
};
var records = soapClient.getRecords(getParams);
if (records != null)
{
if (records.Count() == 0)
{
break;
}
Skip += records.Count();
if (records.Count() != Take)
{
// last batch or everything in first batch
break;
}
}
else
{
// service now web service endpoint not configured correctly
break;
}
}
}
I made an library that handles interacting with ServiceNow Rest API much easier
https://emersonbottero.github.io/ServiceNow.Core/
I'm trying to use Tweetinvi to retrieve all the tweets made by a particular screen name.
I've tried GetUserTimeLine (see below) but it shows tweets from all the people I follow instead of just mine.
IUser user = Tweetinvi.User.GetUserFromScreenName("SCREEN_NAME");
// Create a parameter for queries with specific parameters
var timelineParameter = Timeline.CreateUserTimelineRequestParameter(user);
timelineParameter.ExcludeReplies = true;
timelineParameter.TrimUser = true;
timelineParameter.IncludeRTS = false;
var tweets = Timeline.GetUserTimeline(timelineParameter);
return tweets;
Thanks,
Travis
A little late, but maybe useful for others (using the Tweetinvi NuGet 0.9.12.1) :
Tweetinvi.Core.Interfaces.IUser user2 = Tweetinvi.User.GetUserFromScreenName("StackOverflow");
var userTimelineParam = new Tweetinvi.Core.Parameters.UserTimelineParameters
{
MaximumNumberOfTweetsToRetrieve = 100,
IncludeRTS=true
};
List<Tweetinvi.Core.Interfaces.ITweet> tweets2= new List<Tweetinvi.Core.Interfaces.ITweet>();
tweets2 = Timeline.GetUserTimeline(user2, userTimelineParam).ToList();
foreach (Tweetinvi.Core.Interfaces.ITweet prime2 in tweets2)
{
Debug.WriteLine(prime2.CreatedAt+" "+prime2.Text+" "+prime2.Id.ToString());
}
Twitter does not provide such endpoint. Therefore you will need to filter the tweets that have not been created by yourself.
Simply use linq (using System.Linq) to filter down the result after your code:
var tweetsPublishedByMyself = tweets.Where(x => x.Creator.Equals(user)).ToArray();