We Keep Coding

Mongo time-series inserts using c# produce exception after short time

I want to test if mongo db can have a collection up to 50 000 000 000.
So I insert 10K elements every second using method:
public async Task InsertManyAsync(List<DBRoutLine> list)
{
await _collRouts.InsertManyAsync(list);
}
Data looks like this:
namespace DbLayer.Models
{
public class DBRoutLineMeta
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public string id { get; set; }
public int counter { get; set; }
}
[BsonIgnoreExtraElements]
public class DBRoutLine
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public string id { get; set; }
public DBRoutLineMeta meta { get; set; } = new DBRoutLineMeta();
public DateTime timestamp { get; set; } = DateTime.UtcNow;
public string some_data { get; set; } = DateTime.Now.ToString();
}
}
id members not required actually but I have them, just for testing.
So I've got exception like this:
"A bulk write operation resulted in one or more errors. WriteErrors: [ { Category : "DuplicateKey", Code : 11000, Message : "E11000 duplicate key error collection: TSTest.system.buckets.TSTable dup key: { _id: ObjectId('634e87301297fa65b7df9923') }" } ]."
after sometime. It can be also like this:
"time-series insert failed: TSTest.TSTable :: caused by :: Expected
And it will never recover from error even if I recreate connection to mongo server. Only application restart helps to insert records again.
Test code:
using DbLayer.Models;
using DbLayer.Services;
using MongoDB.Bson;
Console.WriteLine("Hello, World!");
var service = new RoutService();
try
{
CancellationTokenSource tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
var list = new List<DBRoutLine>();
for (int i = 0; i < 10000; i++)
{
DBRoutLine line = new DBRoutLine();
list.Add(line);
}
Task task = Task.Run(async () => {
int max_counter = await service.GetMaxCount();
bool recover = false;
while (!token.IsCancellationRequested)
{
try
{
if (!recover)
{
foreach (DBRoutLine line in list)
{
line.meta.counter = ++max_counter;
line.id = ObjectId.GenerateNewId().ToString();
line.meta.id = line.id;
}
}
var t1 = DateTime.Now;
await service.InsertManyAsync(list);
var t2 = DateTime.Now;
max_counter = await service.GetMaxCount();
var t3 = DateTime.Now;
Console
.WriteLine(
$"{max_counter}->Insert:{(int)(t2 - t1).TotalMilliseconds}, GetMax:{(int)(t3 - t2).TotalMilliseconds}");
recover = false;
}
catch(Exception ex)
{
recover = true;
await Task.Delay(3000);
Console.WriteLine(ex.Message.ToString());
service = new RoutService();
max_counter = await service.GetMaxCount();
}
}
}, token);
Console.WriteLine("Press any key to stop emulation\n");
Console.ReadKey();
tokenSource.Cancel();
Task.WaitAll(task);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
Service code:
using DbLayer.Models;
using MongoDB.Bson;
using MongoDB.Driver;
namespace DbLayer.Services
{
public class RoutService:IDisposable
{
private readonly IMongoCollection<DBRoutLine> _collRouts;
private readonly MongoClient _mongoClient;
private readonly string CollName = "TSTable";
public RoutService(
)
{
var ConnectionString = "mongodb://mongoservice:27017";
_mongoClient = new MongoClient(
ConnectionString);
var mongoDatabase = _mongoClient.GetDatabase(
"TSTest");
var filter = new BsonDocument("name", CollName);
var options = new ListCollectionNamesOptions { Filter = filter };
if (!mongoDatabase.ListCollectionNames(options).Any())
{
var createOptions = new CreateCollectionOptions();
var timeField = nameof(DBRoutLine.timestamp);
var metaField = nameof(DBRoutLine.meta);
createOptions.TimeSeriesOptions =
new TimeSeriesOptions(timeField, metaField, TimeSeriesGranularity.Minutes);
mongoDatabase.CreateCollection(
CollName,
createOptions);
}
_collRouts =
mongoDatabase.GetCollection<DBRoutLine>(
CollName
);
CreateIndexes();
}
private void CreateIndexes()
{
{
IndexKeysDefinition<DBRoutLine> keys =
new IndexKeysDefinitionBuilder<DBRoutLine>()
.Descending(d => d.meta.counter);
var indexModel = new CreateIndexModel<DBRoutLine>(
keys, new CreateIndexOptions()
{ Name = "counter" }
);
_collRouts.Indexes.CreateOneAsync(indexModel);
}
////////////////////////////////////////////////
{
IndexKeysDefinition<DBRoutLine> keys =
new IndexKeysDefinitionBuilder<DBRoutLine>()
.Ascending(d => d.meta.id);
var indexModel = new CreateIndexModel<DBRoutLine>(
keys, new CreateIndexOptions()
{ Name = "id" }
);
_collRouts.Indexes.CreateOneAsync(indexModel);
}
}
public async Task InsertManyAsync(List<DBRoutLine> list)
{
await _collRouts.InsertManyAsync(list);
}
public async Task<int> GetMaxCount()
{
var last = await _collRouts
.Find(i=> i.meta.counter > 0)
.SortByDescending( i => i.meta.counter).FirstOrDefaultAsync();
if (last == null)
{
return 0;
}
return last.meta.counter;
}
public void Dispose()
{
}
}
}
project repository:
github.com/pruginkad/TestMongo

Ok, I found the bug. I changed timestamp only once when I created List of documents.
in this code:
foreach (DBRoutLine line in list)
{
line.meta.counter = ++max_counter;
line.id = ObjectId.GenerateNewId().ToString();
line.meta.id = line.id;
line.timestamp = DateTime.UtcNow;//missing line
}
I had to update timestamp, my mistake.
Anyway it's kind of strange that exception happen after every 17M documents and disappear after restart of mongo db

Subscribe to contract events using Nethereum

I need to get subscribe to Uniswap pair contract Sync event and get pair reserves. So here what I tried to do:
[Event("Sync")]
class PairSyncEventDTO : IEventDTO
{
[Parameter("uint112", "reserve0")]
public virtual BigInteger Reserve0 { get; set; }
[Parameter("uint112", "reserve1", 2)]
public virtual BigInteger Reserve1 { get; set; }
}
public async Task Start()
{
readonly string uniSwapFactoryAddress = "0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f";
string uniSwapFactoryAbi = GetAbi(Resources.IUniswapV2Factory);
string uniSwapPairAbi = GetAbi(Resources.IUniswapV2Pair);
var web3 = new Web3("https://mainnet.infura.io/v3/fff");
Contract uniSwapFactoryContract = web3.Eth.GetContract(uniSwapFactoryAbi, uniSwapFactoryAddress);
Function uniSwapGetPairFunction = uniSwapFactoryContract.GetFunction("getPair");
string daiAddress = "0x6b175474e89094c44da98b954eedeac495271d0f";
string wethAddress = "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2";
string pairContractAddress = await uniSwapGetPairFunction.CallAsync<string>(wethAddress, daiAddress);
Contract pairContract = web3.Eth.GetContract(uniSwapPairAbi, pairContractAddress);
Event pairSyncEvent = pairContract.GetEvent("Sync");
NewFilterInput pairSyncFilter = pairSyncEvent.EventABI.CreateFilterInput();
using (var client = new StreamingWebSocketClient("wss://mainnet.infura.io/ws/v3/fff"))
{
var subscription = new EthLogsObservableSubscription(client);
subscription.GetSubscriptionDataResponsesAsObservable().
Subscribe(log =>
{
try
{
EventLog<PairSyncEventDTO> decoded = Event<PairSyncEventDTO>.DecodeEvent(log);
if (decoded != null)
{
decimal reserve0 = Web3.Convert.FromWei(decoded.Event.Reserve0);
decimal reserve1 = Web3.Convert.FromWei(decoded.Event.Reserve1);
Console.WriteLine($#"Price={reserve0 / reserve1}");
}
else Console.WriteLine(#"Found not standard transfer log");
}
catch (Exception ex)
{
Console.WriteLine(#"Log Address: " + log.Address + #" is not a standard transfer log:", ex.Message);
}
});
await client.StartAsync();
await subscription.SubscribeAsync(pairSyncFilter);
}
}
string GetAbi(byte[] storedContractJson)
{
string json = Encoding.UTF8.GetString(storedContractJson);
JObject contractObject = JObject.Parse(json);
if (!contractObject.TryGetValue("abi", out JToken abiJson)) throw new KeyNotFoundException("abi object was not found in stored contract json");
return abiJson.ToString();
}
And it seems to subscribe, but never enters Subscribe lambda.
Also if I try to await subscription.SubscribeAsync(); without any filter, it also doesn't enter Subscribe lambda.
But after executing SubscribeAsync CPU is significantly loaded by the process.
What am I doing wrong? Why isn't Subscribe lambda called?
Why does it load CPU?

I don't see a major issue with your code, but as I don't have the abis, this is an example without them. The "Sync" event does not fire all the time, so that might have been the issue.
using Nethereum.ABI.FunctionEncoding.Attributes;
using Nethereum.Contracts;
using Nethereum.JsonRpc.WebSocketStreamingClient;
using Nethereum.RPC.Reactive.Eth.Subscriptions;
using System;
using System.Collections.Generic;
using System.Numerics;
using System.Text;
using System.Threading.Tasks;
using Nethereum.RPC.Eth.DTOs;
using Nethereum.RPC.Web3;
using Newtonsoft.Json.Linq;
namespace Nethereum.WSLogStreamingUniswapSample
{
class Program
{
[Event("Sync")]
class PairSyncEventDTO : IEventDTO
{
[Parameter("uint112", "reserve0")]
public virtual BigInteger Reserve0 { get; set; }
[Parameter("uint112", "reserve1", 2)]
public virtual BigInteger Reserve1 { get; set; }
}
public partial class GetPairFunction : GetPairFunctionBase { }
[Function("getPair", "address")]
public class GetPairFunctionBase : FunctionMessage
{
[Parameter("address", "tokenA", 1)]
public virtual string TokenA { get; set; }
[Parameter("address", "tokenB", 2)]
public virtual string TokenB { get; set; }
}
public static async Task Main()
{
string uniSwapFactoryAddress = "0x5C69bEe701ef814a2B6a3EDD4B1652CB9cc5aA6f";
var web3 = new Web3.Web3("https://mainnet.infura.io/v3/7238211010344719ad14a89db874158c");
string daiAddress = "0x6b175474e89094c44da98b954eedeac495271d0f";
string wethAddress = "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2";
var pairContractAddress = await web3.Eth.GetContractQueryHandler<GetPairFunction>()
.QueryAsync<string>(uniSwapFactoryAddress,
new GetPairFunction() {TokenA = daiAddress, TokenB = wethAddress});
var filter = web3.Eth.GetEvent<PairSyncEventDTO>(pairContractAddress).CreateFilterInput();
using (var client = new StreamingWebSocketClient("wss://mainnet.infura.io/ws/v3/7238211010344719ad14a89db874158c"))
{
var subscription = new EthLogsObservableSubscription(client);
subscription.GetSubscriptionDataResponsesAsObservable().
Subscribe(log =>
{
try
{
EventLog<PairSyncEventDTO> decoded = Event<PairSyncEventDTO>.DecodeEvent(log);
if (decoded != null)
{
decimal reserve0 = Web3.Web3.Convert.FromWei(decoded.Event.Reserve0);
decimal reserve1 = Web3.Web3.Convert.FromWei(decoded.Event.Reserve1);
Console.WriteLine($#"Price={reserve0 / reserve1}");
}
else Console.WriteLine(#"Found not standard transfer log");
}
catch (Exception ex)
{
Console.WriteLine(#"Log Address: " + log.Address + #" is not a standard transfer log:", ex.Message);
}
});
await client.StartAsync();
subscription.GetSubscribeResponseAsObservable().Subscribe(id => Console.WriteLine($"Subscribed with id: {id}"));
await subscription.SubscribeAsync(filter);
Console.ReadLine();
await subscription.UnsubscribeAsync();
}
}
}
To keep it alive in infura, you might want to ping it every so often
Example
while (true)
{
var handler = new EthBlockNumberObservableHandler(client);
handler.GetResponseAsObservable().Subscribe(x => Console.WriteLine(x.Value));
await handler.SendRequestAsync();
Thread.Sleep(30000);
}

ASP.Net ElasticSearch

Hello so I am trying to code a simple web view, with asp.net and NEST library, that will take my ElasticSearch database, and show it in textview on button click.
This is the code that I input when my button is clicked,
would you please look at it and tell me am I on a good path or something is not good.
using Elasticsearch.Net;
using Nest;
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
namespace ElasticsearchWeb
{
public class shekspir
{
public string type { get; set; }
public int line_id { get; set; }
public string play_name { get; set; }
public int speech_number { get; set; }
public float line_number { get; set; }
public string speaker { get; set; }
public string text_entry { get; set; }
}
public partial class Default : System.Web.UI.Page
{
public static Uri GetElasticHost()
{
var host = "http://localhost:9200";
return new Uri(host);
}
public static ElasticClient GetElasticClient(ConnectionSettings settings = null)
{
if (settings == null)
{
var node = GetElasticHost();
var pool = new SingleNodeConnectionPool(node);
settings = new ConnectionSettings(pool);
}
settings.DisableDirectStreaming(true);
var client = new ElasticClient(settings);
return client;
}
public static List<shekspir> GetAllShekspir(int ID)
{
var workОfShakespeare = GetElasticClient();
ISearchResponse<shekspir> result = null;
result = workОfShakespeare.Search<shekspir>(x => x
.Index("shekspir")
.Query(q => q
.MatchAll())
.Size(100)
);
List<shekspir> list = new List<shekspir>();
foreach (var r in result.Hits)
{
shekspir a = r.Source;
list.Add(a);
}
return list;
}
protected void Page_Load(object sender, EventArgs e)
{
}
protected void Button1_Click(object sender, EventArgs e)
{
List<shekspir> list = GetAllShekspir (1);
foreach (shekspir u in list)
{
litInfo.Text += u.play_name + ": " + u.text_entry + "<br>";
}
}
}
}

List<shekspir> list = new List<shekspir>();
foreach (var r in result.Hits)
{
shekspir a = r.Source;
list.Add(a);
}
If you just want the returned documents above can be replaced by
var list= result.Documents

Console doesn't write any value

I'm trying to scrape books.toscrape.com everything seems perfect, but it doesn't output anything to console.
I'm sure about the XPaths are correct, and syntax is right.
I don't see any errors, or warnings.
Don't have any clue what i can try for this problem.
using System;
using System.Windows;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using System.Data;
using System.Collections.Generic;
namespace book_scraping
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
DataTable table = new DataTable();
public MainWindow()
{
InitializeComponent();
}
string user_url;
class Book
{
public string Titlex { get; set; }
public string Price { get; set; }
public string Rate { get; set; }
}
public void Scrape()
{
var books = new List<Book>();
IWebDriver driver = new FirefoxDriver();
user_url = Textbox1.Text;
int.TryParse(Textbox2.Text, out var x);
for (int i = 1; i < x; i++)
{
driver.Url = "http://" + user_url + "/catalogue/" + "page-" + i + ".html";
var element = driver.FindElements(By.XPath("//article[#class='product_pod']"));
foreach (var elements in element) {
var book = new Book
{
Titlex = driver.FindElement(By.XPath("//h3/a")).Text,
Price = driver.FindElement(By.XPath("//p[#class='price_color']")).Text,
Rate = driver.FindElement(By.XPath("//article/p")).GetAttribute("class")?.Replace("star-rating ", ""),
};
foreach (var a in books)
{
Console.WriteLine($"{a.Titlex} {a.Price} {a.Rate}");
}
}
}
}
private void Button_Click(object sender, RoutedEventArgs e)
{
Scrape();
}
}
}
I expect the output of title,price,rate as text like
hello world 50 three, or something similar

Your Books list is empty, try this
books.Add(new Book
{
Titlex = driver.FindElement(By.XPath("//h3/a")).Text,
Price = driver.FindElement(By.XPath("//p[#class='price_color']")).Text,
Rate = driver.FindElement(By.XPath("//article/p")).GetAttribute("class")?.Replace("star-rating ", ""),
});
instead of:
var book = new Book
{
Titlex = driver.FindElement(By.XPath("//h3/a")).Text,
Price = driver.FindElement(By.XPath("//p[#class='price_color']")).Text,
Rate = driver.FindElement(By.XPath("//article/p")).GetAttribute("class")?.Replace("star-rating ", ""),
};

how to read items from file into a list of items and set the properties to the value in the file?

Hello I am trying to create a stock log system in c# winforms and i am a bit stuck for ideas on how to read the items back into a list and storing the data into the properties.
I will be reading in from a csv file where each line is 1 item and each property is separated by a comma.
the main Class
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
//using System.IO;
//using Microsoft.VisualBasic;
namespace stock_list
{
public partial class Form1 : Form
{
private List<item> itemlist = new List<item>((1));
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
}
private void btnSave_Click(object sender, EventArgs e)
{
saveitem(Convert.ToInt64(txtStallNumber.Text), Convert.ToInt64(txtStockNumber.Text), txtDescription.Text, Convert.ToDecimal(txtPaidprice.Text), Convert.ToDecimal(txtSoldPrice.Text));
}
private void btnItems_Click(object sender, EventArgs e)
{
readfromfile();
}
private void readfromfile()
{
var reader = new System.IO.StreamReader(#"file.csv", Encoding.UTF8, false);
while (!reader.EndOfStream)
{
//what todo here??
}
}
private void saveitem(long stallnumberpar, long stocknumberpar, string itemdiscriptionpar, decimal boughtpricepar, decimal soldpricepar, decimal profitorlosspar = 0)
{
itemlist.Add(new item { stallnumber = stallnumberpar, stocknumber = stocknumberpar, itemdescription = itemdiscriptionpar, boughtprice = boughtpricepar, soldprice = soldpricepar, profitorloss = soldpricepar - boughtpricepar});
txtDescription.Clear();
txtPaidprice.Clear();
txtSoldPrice.Clear();
txtStallNumber.Text = "";
txtStockNumber.Clear();
txtStallNumber.Focus();
MessageBox.Show("Item Saved");
}
private void btnQuery_Click(object sender, EventArgs e)
{
RunQueryDescription(Microsoft.VisualBasic.Interaction.InputBox("Enter Search Criteria", "Enter Search Criteria", "Default",0,0));
}
private void RunQueryDescription(string description)
{
//List<item> products = new List<item>((1));
var writer = new System.IO.StreamWriter(#"file.csv", true, Encoding.UTF8);
item[] productsarr = new item[itemlist.Count];
int index = 0;
foreach (item product in itemlist)
{
if (product.itemdescription.Contains(description))
{
productsarr[index] = product;
index++;
}
else
{
index++;
continue;
}
}
for (int i = 0; i < productsarr.Length; i++)
{
MessageBox.Show(productsarr[i].stallnumber.ToString() +
productsarr[i].stocknumber.ToString() +
productsarr[i].itemdescription.ToString() +
productsarr[i].boughtprice.ToString() +
productsarr[i].soldprice.ToString() +
productsarr[i].profitorloss.ToString());
writer.Write(productsarr[i].stallnumber.ToString() + "," +
productsarr[i].stocknumber.ToString() + "," +
productsarr[i].itemdescription.ToString() + "," +
productsarr[i].boughtprice.ToString() + "," +
productsarr[i].soldprice.ToString() + "," +
productsarr[i].profitorloss.ToString());
writer.Close();
writer.Dispose();
}
}
}
}
The items class
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Collections;
using System.IO;
namespace stock_list
{
class item
{
public long stallnumber { get; set; }
public long stocknumber { get; set; }
public string itemdescription { get; set; }
public decimal boughtprice { get; set; }
public decimal soldprice { get; set; }
public decimal profitorloss { get; set; }
}
}
EDIT:
Example File
1,1,Vase,1.00,2.00,1.00
Any help will be valued
Thanks in Advance!

You can use the File.ReadAllLines to read the file
private List<item> itemlist = new List<item>();
private void readfromfile()
{
var lines = System.IO.File.ReadAllLines("path");
foreach (string item in lines)
{
var values = item.Split(',');
itemlist.Add(new item()
{
stallnumber = long.Parse(values[0]),
stocknumber = long.Parse(values[1]),
itemdescription = values[2],
//and so on
});
}
}

Try splitting each line on the delimiter (most likely a comma), and then parse out the line and add a new instance of item to your list. Something like this should work inside your loop (untested):
var line = reader.ReadLine();
var values = line.Split(',');
itemlist.Add(new item { stallnumber = Convert.ToInt32(values[0]), ... });