I'm scraping data from a variety of pages on a site, and I want to assign points to a total score based on whether or not elements (H1s, alt tags, meta titles; that sort of thing) exist, are unique or are duplicates. I'd like to create a method that can do this for every element I scrape, which I am using custom classes to store.
public class PageData
{
[Key]
[Required]
public int Id { get; set; }
[Required]
public string PageUrl { get; set; }
public string Analytics { get; set; }
public bool Paginated { get; set; }
public bool Flash { get; set; }
public bool Iframe { get; set; }
public bool NoIndexFollow { get; set; }
public bool SchemaTag { get; set; }
public virtual ICollection<Platform> Platforms { get; set; }
public virtual ICollection<AltTag> AltTags { get; set; }
public virtual ICollection<Canonical> Canonicals { get; set; }
public virtual ICollection<MetaTitle> MetaTitles { get; set; }
public virtual ICollection<MetaDesc> MetaDescs { get; set; }
public virtual ICollection<BlogLocation> BlogLocations { get; set; }
public virtual ICollection<H1> H1s { get; set; }
public virtual ICollection<H2> H2s { get; set; }
public virtual ICollection<H3> H3s { get; set; }
public virtual ICollection<ViewState> ViewStates { get; set; }
}
public class H1
{
public H1() { }
public H1(int id, string h1)
{
this.Id = id;
this.H1String = h1;
}
public override string ToString()
{
return H1String;
}
[Key]
public int KeyId { get; set; }
public string H1String { get; set; }
[ForeignKey("PageData")]
public int Id { get; set; }
[ScriptIgnore]
public virtual PageData PageData { get; set; }
}
Method to try allocate scores
public void pageCheck(SiteData site, dynamic pageObj, int lowAssignedScore, int highAssignedScore, int totalScore)
{
List<string> uniqueCheckList = new List<string>();
bool uniqueCheck = true;
foreach (PageData page in site.PageDatas)
{
if (pageObj.Count != 0)
{
foreach (var modelObj in pageObj)
{
if (uniqueCheckList.Contains(modelObj.ToString()))
{
totalScore =+ lowAssignedScore;
uniqueCheck = false;
break;
}
uniqueCheckList.Add(modelObj.ToString());
}
if (uniqueCheck)
{
totalScore += highAssignedScore;
}
}
}
I'm instantiating a new page object to pass in which element of the page I want:
PageData page = new PageData();
pageCheck(site, page.H1s, 4, 6, totalScore);
When I pass in page.H1s it is coming through as:
{System.Collections.Generic.List < Bescoured.Models.PageModels.H1 > }
Is there a way to do what I am trying to do in c#? Or will it not let me due to the static nature of the language? I started off by creating a method that was specific to H1 then trying to make it generic but it looks like I need to make a method for each element.
EDIT:
An example of how I would do it if I had make a method for each element
foreach (PageData page in site.PageDatas)
{
if (page.H1s.Count != 0)
{
foreach (H1 h1 in page.H1s)
{
if (h1s.Contains(h1.H1String))
{
totalScore += 4;
uniqueCheck = false;
break;
}
h1s.Add(h1.H1String);
}
if (uniqueCheck)
{
totalScore += 6;
}
}
}
I see you're only using the ToString() method of a element. Why not pass a IEnumerable< string> into it?
public void pageCheck(SiteData site, IEnumerable<string> pageObj, int lowAssignedScore, int highAssignedScore, int totalScore)
Usage:
pageCheck(site, page.H1s.Select(item => item.ToString()), 4, 6, totalScore);
pageCheck(site, page.AltTags.Select(item => item.ToString()), 4, 6, totalScore);
I'll always try to avoid dynamics, unless there's no other option.
Related
I want to know which approach is faster. I have the following class:
public class AccountBalance
{
public long AccountID { get; set; }
public string AccountNumber { get; set; }
[NotRequiredForDataTable]
public IEnumerable<AccountMediaBalance> AccountMediaBalances { get; set; }
}
public class AccountMediaBalance
{
public long AccountID { get; set; }
public long MediaID { get; set; }
public string MediaSerialNumber { get; set; }
public decimal? CashAmount { get; set; }
public int FareID { get; set; }
public bool ShowID { get; set; }
public bool ShareCash { get; set; }
public bool ShareToken { get; set; }
public bool SharePass { get; set; }
public IEnumerable<Token> Tokens { get; set; }
public IEnumerable<Pass> Passes { get; set; }
}
public class Token
{
public long TokenID { get; set; }
public long AccountID { get; set; }
public long MediaProductID { get; set; }
public long MediaID { get; set; }
public long? ActivateByMediaID { get; set; }
public string SerialNumber { get; set; }
public decimal? TransferValue { get; set; }
public DateTimeOffset CreateTime { get; set; }
public DateTimeOffset? ActivateTime { get; set; }
public DateTimeOffset? ExpiryTime { get; set; }
}
public class Pass
{
public long MediaProductID { get; set; }
public long AccountID { get; set; }
public long MediaID { get; set; }
public int ProductID { get; set; }
public long? ActivateByMediaID { get; set; }
public string SerialNumber { get; set; }
public DateTimeOffset CreateTime { get; set; }
public DateTimeOffset? ActivateTime { get; set; }
public DateTimeOffset? ExpiryTime { get; set; }
}
I have a list of AccountBalance data as List<AccountBAlance> and I want to transform data in a way that I want to separate AccountMediaBalance collection in one list, All tokens in one list and all passes in a separate list.
There are two approaches for doing this:
public void SaveAccountBalances(List<AccountBalance> accountBalances)
{
if (accountBalances != null)
{
var mediaBalances = accountBalances.SelectMany(x => x.AccountMediaBalances ??
Enumerable.Empty<AccountMediaBalance>()).ToList();
var tokens = mediaBalances.SelectMany(x => x.Tokens ?? Enumerable.Empty<Token>()).ToList();
var passes = mediaBalances.SelectMany(x => x.Passes ?? Enumerable.Empty<Pass>()).ToList();
}
}
The other approach would be like following:
public void SaveAccountBalances(List<AccountBalance> accountBalances)
{
var mediaBalances = new List<AccountMediaBalance>();
var tokens = new List<Token>();
var passes = new List<Pass>();
if (accountBalances != null)
{
foreach (var item in accountBalances)
{
mediaBalances.AddRange(item.AccountMediaBalances ?? Enumerable.Empty<AccountMediaBalance>());
}
foreach (var item in mediaBalances)
{
tokens.AddRange(item.Tokens ?? Enumerable.Empty<Token>());
passes.AddRange(item.Passes ?? Enumerable.Empty<Pass>());
}
}
}
Performance is a big concern. Can anyone put me in the right direction and let me know which approach is faster and why?
foreach loop uses GetEnumerator directly, while linq creates the query object first and then GetEnumerator. So it is naturally a little bit faster to use foreach loop for a single iteration but it still makes the code look better to use linq.
I have the following code that populates var data from a webpage using JsonConvert on clicking the Go button, what i need is to still access the data rootobject in a separate loop ie the generatealliancelist at the bottom but im not sure on how to declare data so it is visible from everywhere?
heres the code:
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
public class Rootobject
{
public Player[] players { get; set; }
public Alliance[] alliances { get; set; }
public Base[] bases { get; set; }
public Poi[] pois { get; set; }
}
public class Player
{
public string i { get; set; } //player ID
public string p { get; set; }
public string a { get; set; } //alliance ID
public string n { get; set; } //player name
public string f { get; set; } //faction (1 GDI, 2 NOD)
public string ps { get; set; }
public string pd { get; set; }
public string bc { get; set; }
}
public class Alliance
{
public string a { get; set; } //alliance ID
public string an { get; set; } //alliance name
public string p { get; set; }
public string c { get; set; } //player count
}
public class Base
{
public string pi { get; set; } //player ID
public string y { get; set; } //coordinates
public string x { get; set; } //coordinates
public string n { get; set; } //base name
public string i { get; set; } //base ID
public string l { get; set; } //base level
public string al { get; set; } //is base alerted
public string pr { get; set; } //has shield
public string cb { get; set; } //building condition in%
public string cd { get; set; } //defense condition in %
public string ps { get; set; } //time in ms before shield drop
}
public class Poi
{
public string x { get; set; } //coordinates
public string y { get; set; } //coordinates
public string t { get; set; } //type from I (Tiberium) to 7(defense) , 0 is tunnel exit
public string l { get; set; } //level
public string a { get; set; } //alliance id owner
}
private void button1_Click(object sender, EventArgs e)
{
System.Net.WebClient wc = new System.Net.WebClient();
string jsonData = wc.DownloadString("http://ccta.hodor.ninja/mapdata/13"); //WORLD4
Regex regex = new Regex("ccmapData = ({.*}]),\"timestamp\":\"(.*)\",\"world_size\":\"(.*)\"");
Match match = regex.Match(jsonData);
System.DateTime timestamp = UnixTimeStampToDateTime(Convert.ToDouble(match.Groups[2].Value));
var worldsize = match.Groups[3].Value;
var data = JsonConvert.DeserializeObject<Rootobject>(match.Groups[1].Value + "}");
}
public void generatealliancelist()
{
foreach (Alliance alliance in data.alliances) // ERROR the name data does not exist in the current context
{
}
}
}
Instead of making the variable "available anywhere", (which is considered bad design) you should pass it to the GenerateAllianceList method as a parameter:
public void GenerateAllianceList(IEnumerable<T> allicances) // Fill in the correct type instead of `T`
{
foreach (Alliance alliance in allicances)
{
}
}
}
The call the method as follows:
GenerateAllicanceList(data.Alliances);
Also, your property names are atrocious. Give them descriptive names instead of explaining them in comments.
Edit:
OP mentioned the properties are like that because that's how they're called in the Json data. In that case, one should use JsonProperty (or DataMember) attributes to serialize these unfortunate names to readable ones. See: How can I change property names when serializing with Json.net?
1.You can either make this a field in your class - I think that's a bad idea since we are talking about a local temp var that changes based on a button click.
2.You should pass the var as a method param - Better in this case:
public void generatealliancelist(Rootobject data)
{
foreach (Alliance alliance in data.alliances) // ERROR the name data does not exist in the current context
{
}
}
}
I am attempting to retrieve information using the Steam API. I created the classes offerStuff and itemsClass, offerStuff contains public List<itemsClass> items { get; set; }, however, whenever I attempt to access this list through os.items.Add(item), my program crashes with NullReferenceException. Is there some declaration I am missing? If so, how would I declare it so I can access it without the exception?
public static List<offerStuff> pollOffers()
{
using (dynamic tradeOffers = WebAPI.GetInterface("IEconService", config.API_Key))
{
List<offerStuff> OfferList = new List<offerStuff>();
offerStuff os = new offerStuff();
KeyValue kvOffers = tradeOffers.GetTradeOffers(get_received_offers: 1);//, active_only: 1
foreach (KeyValue kv in kvOffers["trade_offers_received"].Children)
{
os.tradeofferid = kv["tradeofferid"].AsInteger(); ;
os.accountid_other = Convert.ToUInt64(kv["accountid_other"].AsInteger());
os.message = kv["message"].AsString();
os.trade_offer_state = kv["trade_offer_state"].AsInteger();
foreach (KeyValue kv2 in kv["items_to_receive"].Children)
{
itemsClass item = new itemsClass();
item.appid = (kv2["appid"].AsInteger());
item.assetid = kv2["assetid"].AsInteger();
item.classid = kv2["classid"].AsInteger();
item.instanceid = kv2["instanceid"].AsInteger();
item.amount = kv2["amount"].AsInteger();
item.missing = kv2["missing"].AsInteger();
os.items.Add(item);
}
os.is_our_offer = kv["is_our_offer"].AsBoolean();
os.time_created = kv["time_created"].AsInteger();
os.time_updated = kv["time_updated"].AsInteger();
OfferList.Add(os);
}
return OfferList;
}
}
}
public class offerStuff
{
public int tradeofferid { get; set; }
public SteamID accountid_other { get; set; }
public string message { get; set; }
public int trade_offer_state { get; set; }
public List<itemsClass> items { get; set; }
public bool is_our_offer { get; set; }
public int time_created { get; set; }
public int time_updated { get; set; }
}
public class itemsClass
{
public int appid { get; set; }
public int assetid { get; set; }
public int classid { get; set; }
public int instanceid { get; set; }
public int amount { get; set; }
public int missing { get; set; }
}
The problem is probably that you're not initializing the collection items. You could do it on your contructor like this
public offerStuff()
{
items = new List<itemsClass>();
}
I've been playing around with Entity Framework for a while now. For some reason however the foreign keys all seem to be null untill I explicitely use the Include method to include them.
According to what I've read here:
Entity Framework 4.1 Virtual Properties
by making them virtual they should be lazy loaded (and not null).
This is my model:
class AltiGoalInfo
{
public int AltiGoalInfoId { get; set; }
public int ShotTime { get; set; } //Time it took from shooting till scoring
public virtual AltiPlayerInfo AltiPlayerInfo { get; set; }
public virtual AltiTeamEnum Team { get; set; }
public override string ToString()
{
double shotTime = ShotTime;
shotTime /= 1000;
shotTime = Math.Round(shotTime, 2);
return shotTime + " by " + AltiPlayerInfo;
}
}
class AltiPlayerInfo
{
public int AltiPlayerInfoId { get; set; }
public String VaporId { get; set; }
public String LastKnownNickname { get; set; }
public int LastKnownLevel { get; set; }
public int LastKnownAceRank { get; set; }
public Boolean LastKnownDemo { get; set; }
private IList<String> _knownIps = new List<String>();
public IList<String> KnownIps
{
get
{
return _knownIps;
}
set
{
_knownIps = value;
}
}
public string KnownIpsString
{
get
{
return String.Join(";", _knownIps);
}
set
{
_knownIps = value.Split(';').ToList();
}
}
public int Goals { get; set; }
public int MatchesPlayed { get; set; }
public int MatchesWon { get; set; }
public double CalcWinRate()
{
double mwon = MatchesWon;
double mplay = MatchesPlayed;
double result = mwon / mplay * 100.0;
return Math.Round(result, 0);
}
public DateTime FirstSeen { get; set; }
public DateTime LastSeen { get; set; }
public override string ToString()
{
return LastKnownNickname;
}
}
class AltiDbContext : DbContext
{
public DbSet<AltiPlayerInfo> AltiPlayerInfos { get; set; }
public DbSet<AltiGoalInfo> AltiGoalInfos { get; set; }
}
When I use the following code to acces the data the brrr variable is null:
var test = altiDbContext.AltiGoalInfos.First();
var brrr = test.AltiPlayerInfo;
When I however include in manually first, it does work:
var test = altiDbContext.AltiGoalInfos.Include(t => t.AltiPlayerInfo).First();
var brrr = test.AltiPlayerInfo;
Edit:
I've also verified the configuration of the DbContext, by default all settings in the configuration are set to true (which should enable lazy loading).
I found the solution, I had to make the model classes public
I want to use LINQ to pass data from one custom collection to another. Its complicated because the collection has 2 sub collections.
Want to copy data to:
public class Quote
{
public int Id { get; set; }
public string Type { get; set; }
public virtual ICollection<Rate> Rates { get; set; }
}
public class Rate
{
public int Id { get; set; }
public virtual ICollection<Option> Options { get; set; }
}
public class Option
{
public int Id { get; set; }
public decimal Price { get; set; }
}
from:
public class Quote
{
public int QuoteId { get; set; }
public string Type { get; set; }
public string Destination { get; set; }
public List<RateSet> RateSets { get; set; }
}
public class RateSet
{
public int Id { get; set; }
public decimal ValueMin { get; set; }
public decimal ValueMax { get; set; }
public List<Option> Options { get; set; }
}
public class Option
{
public int Id { get; set; }
public string Name { get; set; }
public decimal? Price { get; set; }
}
I was getting somewhere with this but keeping hitting problems...
newQuotes = Quotes
.Select(x => new Quote() {
Id = x.QuoteId,
Rates = x.RateSets.Select( y => new Rate() {
Id = y.Id,
Options = y.Options.Select(z => new Option() {
Id = z.Id,
Price = z.Price
}).ToList(),....
to
Compiled without any errors
// to
public class Quote2
{
public int Id { get; set; }
public string Type { get; set; }
public virtual ICollection<Rate> Rates { get; set; }
}
public class Rate
{
public int Id { get; set; }
public virtual ICollection<Option2> Options { get; set; }
}
public class Option2
{
public int Id { get; set; }
public decimal Price { get; set; }
}
// from
public class Quote1
{
public int QuoteId { get; set; }
public string Type { get; set; }
public string Destination { get; set; }
public List<RateSet> RateSets { get; set; }
}
public class RateSet
{
public int Id { get; set; }
public decimal ValueMin { get; set; }
public decimal ValueMax { get; set; }
public List<Option1> Options { get; set; }
}
public class Option1
{
public int Id { get; set; }
public string Name { get; set; }
public decimal? Price { get; set; }
}
void Main()
{
var Quotes = new List<Quote1>();
var newQuotes = Quotes
.Select(x => new Quote2 {
Id = x.QuoteId,
Rates = x.RateSets == null ? null : x.RateSets.Select( y => new Rate {
Id = y.Id,
Options = y.Options == null ? null : y.Options.Select(z => new Option2 {
Id = z.Id,
Price = z.Price.Value
}).ToList()}).ToList()}).ToList();
}
I would make it a bit more modular:
newQuotes = Quotes.Select(x => new Quote
{
ID = x.QuoteID,
Type = x.Type,
Rates = ConvertRates(x.RateSets)
});
ConvertRates would use the same approach to create its sub objects and could either be a method or a Func:
ICollection<Rate> ConvertRates(IEnumerable<RateSet> oldRates)
{
return oldRates.Select(x => new Rate
{
ID = x.ID,
Options = ConvertOptions(x.Options)
}).ToList();
}
Basically, this is the same approach you used, just split up and readable.
I think what you need to do is define casting between each two corresponding classes, then cast one list into the other.
A simpler way may be to create methods in each class that would convert itself to the other type. Or if you don't want that kind of coupling, create a factory class that will do the conversion for you, one item at a time. Then use link to loop through and convert each item.
Like so:
public class Quote
{
public int Id { get; set; }
public string Type { get; set; }
public virtual ICollection<Rate> Rates { get; set; }
public static Quote FromData(Data.Quote input){
if (input == null) return null;
Quote output = new Quote()
{
Id = input.QuoteId,
Type = input.Type
};
output.Rates = (from i in input.RateSets
select Rate.FromData(i)).ToList();
}
}
public class Rate
{
public int Id { get; set; }
public virtual ICollection<Option> Options { get; set; }
public static Rate FromData(Data.RateSet input)
{
if (input == null) return null;
Rate output = new Rate()
{
Id = input.Id
};
output.Options = (from i in input.Options
select Option.FromData(i)).ToList();
return output;
}
}
public class Option
{
public int Id { get; set; }
public decimal Price { get; set; }
public static Option FromData(Data.Option input)
{
if (input == null) return null;
Option output = new Option()
{
Id = input.Id,
Price = input.Price ?? 0m
};
return output;
}
}
namespace Data {
public class Quote
{
public int QuoteId { get; set; }
public string Type { get; set; }
public string Destination { get; set; }
public List<RateSet> RateSets { get; set; }
}
public class RateSet
{
public int Id { get; set; }
public decimal ValueMin { get; set; }
public decimal ValueMax { get; set; }
public List<Option> Options { get; set; }
}
public class Option
{
public int Id { get; set; }
public string Name { get; set; }
public decimal? Price { get; set; }
}
}