I created an data collection app for our company which collect data from our remote devices.
The data is collected from a datamailbox which is comparable with an database that works like an 10 day buffer to store the data. this is all correctly working.
The data is collected through post api requests. for example :
var url = BuildUrl("syncdata");
var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
var data = DynamicJson.Parse(response);
transactionId = data.transactionId;
I've been trying to collect multiple devices at one time but the problem is that it starts running and collect the data from the first device which works. Than our second device will start collecting the data but it only starts from where device one ended so i've been losing 12hours of data each run. For performance we use transactionId's.(each set of data has its own Id)
The workflow should be like this :
When the data is retrieved for the first time, the user specifies only
the createTransaction filter. The DataMailbox returns all the data of
all devices gateways – with historical data – of the account along a
transaction ID. For the next calls to the API, the client specifies
both createTransaction and lastTransactionId filters. The
lastTransactionId is the ID of the transaction that was returned by
the latest request. The system returns all the historical
data that has been received by the DataMailbox since the last
transaction and a new transaction ID. deviceIds is an additional
filter on the returned result. You must be cautious when using the
combination of lastTransactionId, createTransaction and deviceIds.
lastTransactionId is first used to determine what set of data — newer
than this transaction ID and from all the Device gateways — must be
returned from the DataMailbox, then deviceIds filters this set of data
to send data only from the desired device gateways. If a first request
is called with lastTransactionId, createTransaction and deviceIds, the
following request — implying a new lastTransactionId — does not
contain values history from the previous lastTransactionId of the
device gateways that were not in the deviceId from previous request.
I'm really struggling with the data collection and have no clue how to use the TransactionId and the LastTransActionId.This is the code for now
try
{
CheckLogin();
using (var webClient = new MyWebClient())
{
bool moreDataAvailable;
int samplesCount = 0;
string transactionId = Properties.Settings.Default.TransactionId;
string lastTransactionId = Properties.Settings.Default.LastTransactionId;
do
{
var url = BuildUrl("syncdata");
var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
var data = DynamicJson.Parse(response);
transactionId = data.transactionId;
var talk2MMessage = getTalk2MMessageHeader(webClient);
if (talk2MMessage != null)
{
}
foreach (var ewon in data.ewons)
{
Directory.CreateDirectory(ewon.name);
foreach (var tag in ewon.tags)
{
try
{
Console.WriteLine(Path.Combine(ewon.name, tag.name + ""));
foreach (var sample in tag.history)
{
Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
Console.WriteLine(transactionId);
samplesCount++;
}
}
catch (RuntimeBinderException)
{ // Tag has no history. If it's in the transaction, it's most likely because it has alarm history
Console.WriteLine("Tag {0}.{1} has no history.", ewon.name, tag.name);
}
}
}
Console.WriteLine("{0} samples written to disk", samplesCount);
// Flush data received in this transaction
if (Properties.Settings.Default.DeleteData)
{
//Console.WriteLine("Flushing received data from the DataMailbox...");
url = BuildUrl("delete");
webClient.CallApi(url, new NameValueCollection() { { "transactionId", transactionId } });
Console.WriteLine("DataMailbox flushed.");
}
//save the transaction id for next run of this program
Properties.Settings.Default.LastTransactionId = lastTransactionId;
Properties.Settings.Default.Save();
// Did we receive all data?
try
{
moreDataAvailable = data.moreDataAvailable;
}
catch (RuntimeBinderException)
{ // The moreDataAvailable flag is not specified in the server response
moreDataAvailable = false;
}
if (moreDataAvailable)
Console.WriteLine("There's more data available. Let's get the next part...");
}
while (moreDataAvailable);
Here are my credentials for starting the collection like all parameters
static void CheckLogin()
{
if (string.IsNullOrEmpty(Properties.Settings.Default.Talk2MDevId))
{
Properties.Settings.Default.Talk2MDevId = Prompt("Talk2MDevId");
Properties.Settings.Default.APIToken = Prompt("API Token");
string deleteInputString = Prompt("Delete data after synchronization? (yes/no)");
Properties.Settings.Default.DeleteData = deleteInputString.ToLower().StartsWith("y");
Properties.Settings.Default.TransactionId = "";
Properties.Settings.Default.LastTransactionId = "";
Properties.Settings.Default.Save();
}
I think it's something with the transactionId and LastTransaction id but i have no clue.
More information can be found here: https://developer.ewon.biz/system/files_force/rg-0005-00-en-reference-guide-for-dmweb-api.pdf
As I understand your question, you problem is that for the first few transactionIds, you only get data from device 1 and then only data from device 2.
I'm assuming the following in my answer:
You didn't specify somewhere else in code the filter on "ewonid"
When you say you lose 12 hours of data , you are assuming it because "device 2" data are streamed after "device 1" data.
You did try without the /delete call with no change
/syncdata is an endpoint that returns a block of data for an account since a given transactionId (or oldest block if you didn't provide a transactionID). This data is sorted by storage date by the server, which depends on multiple factors:
when was the device last "vpn online"
at which frequency the device is pushing data to datamailbox
when was that device packet digested by datamailbox service
You could technically have 1 year old data pushed by a device that gets connected back to vpn now, and those data would be registered in the most recent blocks.
For those reasons, the order of data block is not the order of device recording timestamp. You always have to look at the field ewons[].tags[].history[].date to known when that measure was made.
foreach (var sample in tag.history)
{
Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
Console.WriteLine(sample.value + " at " + sample.date);
Console.WriteLine(transactionId);
samplesCount++;
}
In your case, I would assume both devices are configured to push their data once a day, one pushing it's backlog, let's say, at 6AM and the other at 6PM.
Related
I'm using MongoDB 4.0.8 with C# driver 2.8.1 and I'm trying to implement Transactions in my project.
I copy-pasted the following code sample:
static async Task<bool> UpdateProducts()
{
//Create client connection to our MongoDB database
var client = new MongoClient(MongoDBConnectionString);
//Create a session object that is used when leveraging transactions
var session = client.StartSession();
//Create the collection object that represents the "products" collection
var products = session.Client.GetDatabase("MongoDBStore").GetCollection<Product>("products");
//Clean up the collection if there is data in there
products.Database.DropCollection("products");
//Create some sample data
var TV = new Product { Description = "Television", SKU = 4001, Price = 2000 };
var Book = new Product { Description = "A funny book", SKU = 43221, Price = 19.99 };
var DogBowl = new Product { Description = "Bowl for Fido", SKU = 123, Price = 40.00 };
//Begin transaction
session.StartTransaction(new TransactionOptions(
readConcern: ReadConcern.Snapshot,
writeConcern: WriteConcern.WMajority));
try
{
//Insert the sample data
await products.InsertOneAsync(session, TV);
await products.InsertOneAsync(session, Book);
await products.InsertOneAsync(session, DogBowl);
var filter = new FilterDefinitionBuilder<Product>().Empty;
var results = await products.Find(filter).ToListAsync();
//Increase all the prices by 10% for all products
var update = new UpdateDefinitionBuilder<Product>().Mul<Double>(r => r.Price, 1.1);
await products.UpdateManyAsync(session, filter, update); //,options);
//Made it here without error? Let's commit the transaction
session.CommitTransaction();
//Let's print the new results to the console
Console.WriteLine("Original Prices:\n");
results = await products.Find<Product>(filter).ToListAsync();
foreach (Product d in results)
{
Console.WriteLine(String.Format("Product Name: {0}\tPrice: {1:0.00}", d.Description, d.Price));
}
}
catch (Exception e)
{
Console.WriteLine("Error writing to MongoDB: " + e.Message);
session.AbortTransaction();
}
return true;
}
But in the first Insert command, I'm getting this error:
Command insert failed:
Transaction numbers are only allowed on a replica set member or mongos.
The Documentation says that:
Starting in version 4.0, MongoDB provides the ability to perform multi-document transactions against replica sets.
I don't have replicas in my project, I have only one database instance which is my primary one. If there a solution or a work-around I can use to implement Transactions? I have methods that update more than one collection and I really think it could save me time to use it.
like the documentation says, transactions only work with replica sets. so you need to run your mongodb server as single node replica set. to achieve that, do the following steps...
step 1:
stop the mongodb server.
step 2:
add the replication setting to your mongod.cfg file. here's my own as an example
storage:
dbPath: C:\DATA
directoryPerDB: true
journal:
enabled: true
systemLog:
destination: file
logAppend: true
path: C:\DATA\log\mongod.log
net:
port: 27017
bindIp: 127.0.0.1
replication:
replSetName: MyRepSet
step 3: open up a mongodb shell and issue the following command to initiate the replica set.
rs.initiate()
step 4: restart mongod
on a side-note, if you'd like to write cleaner, more convenient transaction code like the following, check out my library MongoDB.Entities
using (var TN = new Transaction())
{
var author = new Author { Name = "one" };
TN.Save(author);
TN.Delete<Book>(book.ID);
TN.Commit();
}
I'm making a c# call to the Stripe.net API to fetch a balance history for a connected account. I'm trying to expand on the balance transaction object to see where the charge is coming from (ex. the customer who made the charge) as all the charges to connected accounts on my platform are from charge objects with a destination property to the connected account.
Here is my code and a screenshot of what the expanded source looks like, but think I should see a charge id or a customer or something refering me to the initial customer somewhere, but I don't...
var balanceService = new StripeBalanceService();
balanceService.ExpandSource = true;
var list = new List <string> () {
"data.source.source_transfer"
};
StripeList <StripeBalanceTransaction> balanceTransactions
= balanceService.List(
new StripeBalanceTransactionListOptions() {
Limit = 20,
Type = "payment",
Expand = list
},
new StripeRequestOptions() {
StripeConnectAccountId = accountId
}
);
foreach(var transaction in balanceTransactions) {
var test = transaction;
}
I feel like I should see a charge id (ex. ch_xxx) or a Customer value (which is null) all I see of any relevance is a payment id (ex. py_xxx)
It is possible to get the charge object(ch_xxx), it is just a little involved!
As you are using destination charges, the charge(ch_xxx) takes place on the platform account, and then a transfer(tr_xxx) is made to the connected account. That transfer creates a payment(py_xxx) on the connected account, which results in a balance transaction(txn_xxx).
As your code expands the source of those balance transactions, you get the payment(py_xxx). The payment is equivalent to a charge, so it has a source_transfer field. You can expand this field also! This will give you the transfer object(tr_xxx). Finally, the transfer has a source_transaction field, and this can be exapanded to give the original charge(ch_xxx)!
Putting that all together, you will want to expand on "data.source.source_transfer.source_transaction".
If you use a Stripe library in a dynamic language you can see this in action ... unfortunately, stripe-dotnet has an open issue right now which means that you can not do this directly. Instead, you will need to make the API calls manually by calling the various Retrieve functions on the IDs, instead of doing a single expansion. It would look something like this:
var paymentId = transaction.Source.Id;
var chargeService = new StripeChargeService();
var payment = chargeService.Get(
paymentId,
new StripeRequestOptions()
{
StripeConnectAccountId = accountId
}
);
var transferService = new StripeTransferService();
transferService.ExpandSourceTransaction = true;
var transfer = transferService.Get(payment.SourceTransferId);
var charge = transfer.SourceTransaction;
Console.WriteLine(charge.Id);
I am using stripe connect(destination payment) with the help of stripe.net library from Jaymedavis.
The problem that I am facing is that I am not able to retrieve the destination payment ID to update the metadata in the connected account. The following line returns a null preventing me from updating meta data on the connected account. But the strange thing is that when I log in to the dashboard the destination payment ID exists. I am not sure why I am not able to retreive it in code.
Is the charge creation asynchronous?. I am not sure. Stripe's connect documentation does not help either. The following line returns a null. My code is down below. Seeking help.
String deschargeID = result.Transfer.DestinationPayment;
Here is the code that I am using
var service = new StripeChargeService(ZambreroSecretKey);
var result = (Stripe.StripeCharge) null;
try {
result = service.Create(newCharge);
if (result.Paid) {
//get the chargeID on the newgen account and update the metadata.
//Returns null even though it exists in the dashboard
String deschargeID = result.Transfer.DestinationPayment;
var chargeService = new StripeChargeService(newgenSecretKey);
StripeCharge charge = chargeService.Get(deschargeID);
charge.Metadata = myDict;
Response.Redirect("PgeCustSuccess.aspx?OrderID=" + OrderID);
}
} catch (StripeException stripeException) {
Debug.WriteLine(stripeException.Message);
stripe.Text = stripeException.Message;
}
The charge object's transfer attribute is not expanded by default, meaning it's just a string with the ID of the transfer object ("tr_..."), not a full transfer object.
According to Stripe.net's documentation, you can expand the transfer attribute by adding this line:
service.ExpandTransfer = True
before sending the charge creation request.
I'm using the PayflowNETAPI class of the PayflowPro API (Payflow_dotNET.dll) to submit a transaction to store credit cards so my company doesn't have to (for PCI Compliant reassons).
I'm using the PNREF from the credit card store transaction to make reference transaction but I keep getting "RESULT=2&PNREF=&RESPMSG=Invalid tender"
I've tried Authorization, Capture, and Sale transactions and they all give the same result. What am I doing wrong?
I've read through the Payflow Gateway Developer Guide and Reference several times (https://developer.paypal.com/docs/classic/payflow/integration-guide/). All the examples for Authorization, Capture, and Sale transactions have the credit card information in the request. There is some small sections that explain and outline credit card uploads but never use the result of the request in a reference transaction.
Below is a sample application and the output
string creditCardUploadRequest = "TRXTYPE=L&TENDER=C&ACCT=4111111111111111&EXPDATE=1218&CVV2=250&BILLTOFIRSTNAME=Homer&BILLTOLASTNAME=Simpson&BILLTOSTREET=350 5th Ave&BILLTOCITY=New York&BILLTOSTATE=NY&BILLTOZIP=10118&BILLTOCOUNTRY=840&USER=<USER>&VENDOR=<VENDOR>&PARTNER=<PARTNER>&PWD=<PASSWORD>&VERBOSITY=HIGH";
var client = new PayPal.Payments.Communication.PayflowNETAPI(HostAddress: "pilot-payflowpro.paypal.com", HostPort: 443, Timeout: 90);
var ccUploadResponse = client.SubmitTransaction(ParamList: creditCardUploadRequest, RequestId: PayflowUtility.RequestId);
//place the responses into collection
var payPalCollection = new NameValueCollection();
foreach (string element in ccUploadResponse.Split('&'))
{
string[] Temp = element.Split('=');
payPalCollection.Add(Temp[0], Temp[1]);
}
Console.WriteLine("creditCardUploadRequest succeeded = {0}", payPalCollection.Get("RESPMSG") == "Approved");
string authorizationRequest = "TRXTYPE=A&ORIGID=" + payPalCollection.Get("PNREF") + "&INVNUM=ORD123456&AMT=50&COMMENT1=My Product Sale&USER=<USER>&VENDOR=<VENDOR>&PARTNER=<PARTNER>&PWD=<PASSWORD>&VERBOSITY=HIGH";
var authorizationResponse = client.SubmitTransaction(ParamList: authorizationRequest, RequestId: PayflowUtility.RequestId);
foreach (string element in authorizationResponse.Split('&'))
{
Console.WriteLine(element);
}
Console.WriteLine("\nDONE");
Console.ReadKey();
OUTPUT:
creditCardUploadRequest succeeded = True
RESULT=2
PNREF=A7X08AB571EC
RESPMSG=Invalid tender
DONE
In your second call you are missing the variable "TENDER=C" . Add that and it should be fine .
"string authorizationRequest = "TRXTYPE=A&ORIGID=" + payPalCollection.Get("PNREF") + "&INVNUM=ORD123456&AMT=50&COMMENT1=My Product Sale&USER=<USER>&VENDOR=<VENDOR>&PARTNER=<PARTNER>&PWD=<PASSWORD>&VERBOSITY=HIGH"
In ServiceNow, I am able to get only a maximum of 250 records in a SOAP request. How to get all the records?
Web Reference Url = https://*****.service-now.com/rm_story.do?WSDL
Code:
var url = "https://*****.service-now.com/rm_story.do?SOAP";
var userName = *****;
var password = *****;
var proxy = new ServiceNow_rm_story
{
Url = url,
Credentials = new NetworkCredential(userName, password)
};
try
{
var objRecord = new Namespace.WebReference.getRecords
{
// filters..
};
var recordResults = proxy.getRecords(objRecord);
}
catch (Exception ex)
{
}
In recordResults, I am getting only 250 records. How to get all the records ?
Also see this stack overflow answer which provides info.
Get ServiceNow Records Powershell - More than 250
Note that returning a large number of records can affect performance of the response and it may be more efficient to process your query in batches using offsets (i.e., get 1-100, then 101-200, ...). This can be achieved by using a sort order and offset. The ServiceNow REST Table API actually returns link headers from Get requests providing you links for the first, next and last set of records making it easy to know the url to query the next batch of records.
See: http://wiki.servicenow.com/index.php?title=Table_API#Methods
and look under 'Response Header'.
Have u tried to pass/override __limit parameter?
Google / wiki / Users manual / Release notes are always helpful
In your code snippet in line where it says //filter you should define __limit (and potentially __first_row and __last_row as explained in the example bellow)
int Skip = 0;
int Take = 250;
while (true)
{
using (var soapClient = new ServiceNowLocations.ServiceNow_cmn_location())
{
var cred = new System.Net.NetworkCredential(_user, _pass);
soapClient.Credentials = cred;
soapClient.Url = _apiUrl + "cmn_location.do?SOAP";
var getParams = new ServiceNowLocations.getRecords()
{
__first_row = Skip.ToString(),
__last_row = (Skip + Take).ToString(),
__limit = Take.ToString()
};
var records = soapClient.getRecords(getParams);
if (records != null)
{
if (records.Count() == 0)
{
break;
}
Skip += records.Count();
if (records.Count() != Take)
{
// last batch or everything in first batch
break;
}
}
else
{
// service now web service endpoint not configured correctly
break;
}
}
}
I made an library that handles interacting with ServiceNow Rest API much easier
https://emersonbottero.github.io/ServiceNow.Core/