We Keep Coding

datacollection program not correctly collecting

I created an data collection app for our company which collect data from our remote devices.
The data is collected from a datamailbox which is comparable with an database that works like an 10 day buffer to store the data. this is all correctly working.
The data is collected through post api requests. for example :
var url = BuildUrl("syncdata");
var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
var data = DynamicJson.Parse(response);
transactionId = data.transactionId;
I've been trying to collect multiple devices at one time but the problem is that it starts running and collect the data from the first device which works. Than our second device will start collecting the data but it only starts from where device one ended so i've been losing 12hours of data each run. For performance we use transactionId's.(each set of data has its own Id)
The workflow should be like this :
When the data is retrieved for the first time, the user specifies only
the createTransaction filter. The DataMailbox returns all the data of
all devices gateways – with historical data – of the account along a
transaction ID. For the next calls to the API, the client specifies
both createTransaction and lastTransactionId filters. The
lastTransactionId is the ID of the transaction that was returned by
the latest request. The system returns all the historical
data that has been received by the DataMailbox since the last
transaction and a new transaction ID. deviceIds is an additional
filter on the returned result. You must be cautious when using the
combination of lastTransactionId, createTransaction and deviceIds.
lastTransactionId is first used to determine what set of data — newer
than this transaction ID and from all the Device gateways — must be
returned from the DataMailbox, then deviceIds filters this set of data
to send data only from the desired device gateways. If a first request
is called with lastTransactionId, createTransaction and deviceIds, the
following request — implying a new lastTransactionId — does not
contain values history from the previous lastTransactionId of the
device gateways that were not in the deviceId from previous request.
I'm really struggling with the data collection and have no clue how to use the TransactionId and the LastTransActionId.This is the code for now
try
{
CheckLogin();
using (var webClient = new MyWebClient())
{
bool moreDataAvailable;
int samplesCount = 0;
string transactionId = Properties.Settings.Default.TransactionId;
string lastTransactionId = Properties.Settings.Default.LastTransactionId;
do
{
var url = BuildUrl("syncdata");
var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
var data = DynamicJson.Parse(response);
transactionId = data.transactionId;
var talk2MMessage = getTalk2MMessageHeader(webClient);
if (talk2MMessage != null)
{
}
foreach (var ewon in data.ewons)
{
Directory.CreateDirectory(ewon.name);
foreach (var tag in ewon.tags)
{
try
{
Console.WriteLine(Path.Combine(ewon.name, tag.name + ""));
foreach (var sample in tag.history)
{
Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
Console.WriteLine(transactionId);
samplesCount++;
}
}
catch (RuntimeBinderException)
{ // Tag has no history. If it's in the transaction, it's most likely because it has alarm history
Console.WriteLine("Tag {0}.{1} has no history.", ewon.name, tag.name);
}
}
}
Console.WriteLine("{0} samples written to disk", samplesCount);
// Flush data received in this transaction
if (Properties.Settings.Default.DeleteData)
{
//Console.WriteLine("Flushing received data from the DataMailbox...");
url = BuildUrl("delete");
webClient.CallApi(url, new NameValueCollection() { { "transactionId", transactionId } });
Console.WriteLine("DataMailbox flushed.");
}
//save the transaction id for next run of this program
Properties.Settings.Default.LastTransactionId = lastTransactionId;
Properties.Settings.Default.Save();
// Did we receive all data?
try
{
moreDataAvailable = data.moreDataAvailable;
}
catch (RuntimeBinderException)
{ // The moreDataAvailable flag is not specified in the server response
moreDataAvailable = false;
}
if (moreDataAvailable)
Console.WriteLine("There's more data available. Let's get the next part...");
}
while (moreDataAvailable);
Here are my credentials for starting the collection like all parameters
static void CheckLogin()
{
if (string.IsNullOrEmpty(Properties.Settings.Default.Talk2MDevId))
{
Properties.Settings.Default.Talk2MDevId = Prompt("Talk2MDevId");
Properties.Settings.Default.APIToken = Prompt("API Token");
string deleteInputString = Prompt("Delete data after synchronization? (yes/no)");
Properties.Settings.Default.DeleteData = deleteInputString.ToLower().StartsWith("y");
Properties.Settings.Default.TransactionId = "";
Properties.Settings.Default.LastTransactionId = "";
Properties.Settings.Default.Save();
}
I think it's something with the transactionId and LastTransaction id but i have no clue.
More information can be found here: https://developer.ewon.biz/system/files_force/rg-0005-00-en-reference-guide-for-dmweb-api.pdf

As I understand your question, you problem is that for the first few transactionIds, you only get data from device 1 and then only data from device 2.
I'm assuming the following in my answer:
You didn't specify somewhere else in code the filter on "ewonid"
When you say you lose 12 hours of data , you are assuming it because "device 2" data are streamed after "device 1" data.
You did try without the /delete call with no change
/syncdata is an endpoint that returns a block of data for an account since a given transactionId (or oldest block if you didn't provide a transactionID). This data is sorted by storage date by the server, which depends on multiple factors:
when was the device last "vpn online"
at which frequency the device is pushing data to datamailbox
when was that device packet digested by datamailbox service
You could technically have 1 year old data pushed by a device that gets connected back to vpn now, and those data would be registered in the most recent blocks.
For those reasons, the order of data block is not the order of device recording timestamp. You always have to look at the field ewons[].tags[].history[].date to known when that measure was made.
foreach (var sample in tag.history)
{
Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
Console.WriteLine(sample.value + " at " + sample.date);
Console.WriteLine(transactionId);
samplesCount++;
}
In your case, I would assume both devices are configured to push their data once a day, one pushing it's backlog, let's say, at 6AM and the other at 6PM.

how do i add a partition key using Azure.ResourceManager.CosmosDB c#

I can create a database and container without an issue on both gremlin and sql, ut I can't seem to set the partition key.
I would expect to do
///
var containerParams = new SqlContainerCreateUpdateParameters
(
new SqlContainerResource(databaseName)
{
PartitionKey = new ContainerPartitionKey()
{
Paths = new List<string>{partialKey}
}
},
new CreateUpdateOptions()
)
///
I would expect to do something like this, but the Paths field is readonly, and I can't see any other option to set it.
[Update]
i got it working with creating an object then converting to json and back to ContainerPartitionKey

Here is the syntax for creating the partition key using Cosmos DB Azure Management SDK.
SqlContainerCreateUpdateParameters sqlContainerCreateUpdateParameters = new SqlContainerCreateUpdateParameters
{
Resource = new SqlContainerResource
{
Id = containerName,
DefaultTtl = -1, //-1 = off, 0 = on no default, >0 = ttl in seconds
AnalyticalStorageTtl = -1,
PartitionKey = new ContainerPartitionKey
{
Kind = "Hash",
Paths = new List<string> { "/myPartitionKey" },
Version = 1 //version 2 for large partition key
}
}
You can find a complete SqlContainer create example here. You can also find a complete set of examples for how to use the Azure Management SDK for Cosmos DB in GitHub. Please note, it is out of date but should still work for illustrating how to use to manage Cosmos resources.

Why do I get the "reference not set to an instance of an object?

I am working with the new CosmosDB SDK v3 https://learn.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-dotnet-standard and a very simple insert, I have verified all the objects are indeed not null and have reasonable values but I still get the error message:
[1/12/2019 10:35:04] System.Private.CoreLib: Exception while executing function: HAPI_HM_Seasons. Microsoft.Azure.Cosmos.Direct: Object reference not set to an instance of an object.
I dont see why this is I must be missing something really basic here but I cant put my finger on it.
The code is as below:
List<SeasonInformation> seasonInformationList = new List<SeasonInformation>();
foreach(JObject document in listOfSeasons)
{
SeasonInformation seasonInformation = new SeasonInformation
{
id = Guid.NewGuid().ToString(),
Brand = brand,
IntegrationSource = source,
DocumentType = Enums.DocumentType.Season,
UpdatedBy = "HAPI_HM_Seasons",
UpdatedDate = DateTime.Now.ToString(),
UpdatedDateUtc = string.Format("{0:yyyy-MM-ddTHH:mm:ss.FFFZ}", DateTime.UtcNow),
OriginalData = document
};
seasonInformationList.Add(seasonInformation);
}
database = cosmosClient.GetDatabase(cosmosDBName);
container = database.GetContainer(cosmosDBCollectionNameRawData);
log.LogInformation(string.Format("HAPI_HM_Seasons BASIC setup done at {0:yyyy-MM-ddTHH:mm:ss.FFFZ}", DateTime.UtcNow));
log.LogInformation(string.Format("HAPI_HM_Seasons import {1} items BEGIN at {0:yyyy-MM-ddTHH:mm:ss.FFFZ}", DateTime.UtcNow, seasonInformationList.Count));
foreach(var season in seasonInformationList)
{
ItemResponse<SeasonInformation> response = await container.CreateItemAsync(season);
}
I have verified that the List is populated and that the season variable in the loop contains the correct data so I am a bit stuck here.
The exception happens in the last foreach loop where I try CreateItemAsync into CosmosDB

As a best practice, you need to use Async method with await in all the Cosmosdb methods just to make sure that they are getting executed and you get the response,
and modify your CreateItemAsync as follows,
ItemResponse<SeasonInformation> response = await container.CreateItemAsync(season, new PartitionKey(season.whatever));
Here is the Sample Repository

Expanding a Source object in a Stripe API call to StripeBalenceService doesn't return any customer info

I'm making a c# call to the Stripe.net API to fetch a balance history for a connected account. I'm trying to expand on the balance transaction object to see where the charge is coming from (ex. the customer who made the charge) as all the charges to connected accounts on my platform are from charge objects with a destination property to the connected account.
Here is my code and a screenshot of what the expanded source looks like, but think I should see a charge id or a customer or something refering me to the initial customer somewhere, but I don't...
var balanceService = new StripeBalanceService();
balanceService.ExpandSource = true;
var list = new List <string> () {
"data.source.source_transfer"
};
StripeList <StripeBalanceTransaction> balanceTransactions
= balanceService.List(
new StripeBalanceTransactionListOptions() {
Limit = 20,
Type = "payment",
Expand = list
},
new StripeRequestOptions() {
StripeConnectAccountId = accountId
}
);
foreach(var transaction in balanceTransactions) {
var test = transaction;
}
I feel like I should see a charge id (ex. ch_xxx) or a Customer value (which is null) all I see of any relevance is a payment id (ex. py_xxx)

It is possible to get the charge object(ch_xxx), it is just a little involved!
As you are using destination charges, the charge(ch_xxx) takes place on the platform account, and then a transfer(tr_xxx) is made to the connected account. That transfer creates a payment(py_xxx) on the connected account, which results in a balance transaction(txn_xxx).
As your code expands the source of those balance transactions, you get the payment(py_xxx). The payment is equivalent to a charge, so it has a source_transfer field. You can expand this field also! This will give you the transfer object(tr_xxx). Finally, the transfer has a source_transaction field, and this can be exapanded to give the original charge(ch_xxx)!
Putting that all together, you will want to expand on "data.source.source_transfer.source_transaction".
If you use a Stripe library in a dynamic language you can see this in action ... unfortunately, stripe-dotnet has an open issue right now which means that you can not do this directly. Instead, you will need to make the API calls manually by calling the various Retrieve functions on the IDs, instead of doing a single expansion. It would look something like this:
var paymentId = transaction.Source.Id;
var chargeService = new StripeChargeService();
var payment = chargeService.Get(
paymentId,
new StripeRequestOptions()
{
StripeConnectAccountId = accountId
}
);
var transferService = new StripeTransferService();
transferService.ExpandSourceTransaction = true;
var transfer = transferService.Get(payment.SourceTransferId);
var charge = transfer.SourceTransaction;
Console.WriteLine(charge.Id);

Changing FullName programmatically in CRM Online (2011)

I am attempting to change the "FullName" field of existing CRM system users in our Dynamics CRM 2011 Online account. I have already made the change in settings to update all future users to the format "Last, First" ... so this is for changing the existing users.
I read the best way is to do this programmatically using the CRM SDK. When I perform the actual Update command, I receive an unspecified error from the SDK: Additional information: The property IsLicensed cannot be modified.
Although I'm querying all columns for entity object SystemUsers, I'm only changing the FullName field. Has anyone else had experience with this? My code is below, I'm running this as a console app to step through each SystemUser.
static void Main(string[] args)
{
string connStr = ConfigurationManager.ConnectionStrings["CRMOnline"].ToString();
CrmConnection conn = CrmConnection.Parse(connStr);
conn.DeviceCredentials = DeviceIdManager.LoadOrRegisterDevice();
using (OrganizationService svc = new OrganizationService(conn))
{
QueryExpression qry = new QueryExpression();
qry.ColumnSet = new ColumnSet(true); // get all columns
qry.EntityName = CRMO.SystemUser.EntityLogicalName; // get entity object SystemUser
qry.Criteria.AddCondition(new ConditionExpression("calendarid", ConditionOperator.NotNull)); // but non-builtin users
EntityCollection col = svc.RetrieveMultiple(qry); // executes query
foreach (Entity ent in col.Entities)
{
Console.WriteLine();
Console.WriteLine("Current Fullname: " + ent.Attributes["fullname"].ToString());
Console.Write("Change? y/N: ");
string ans = Console.ReadLine();
if (ans.ToLower() == "y")
{
Console.Write("New Name: ");
string newname = Console.ReadLine();
if (newname != "")
{
ent.Attributes["fullname"] = newname;
svc.Update(ent); // fails here with SDK error: "Additional information: The property IsLicensed cannot be modified."
}
}
}
Console.WriteLine();
Console.WriteLine("--- Done ---");
Console.ReadLine();
}
}

Rule 28 of the Crm SDK, don't ever perform updates by performing a select, which returns back more fields than what you are planning to update. Any fields in the attribute collection of the Entity will be updated even if they haven't changed. Instead, instantiate a new entity locally, set the id and whatever attributes you want to update and update it.
On a side note, you can't update the full name of a System User. You have to update the individual pieces. So your code should really look like this:
static void Main(string[] args)
{
string connStr = ConfigurationManager.ConnectionStrings["CRMOnline"];
CrmConnection conn = CrmConnection.Parse(connStr);
conn.DeviceCredentials = DeviceIdManager.LoadOrRegisterDevice();
using (OrganizationService svc = new OrganizationService(conn))
{
QueryExpression qry = new QueryExpression();
qry.ColumnSet = new ColumnSet("firstname", "lastname", "fullname"); // get only what is needed for performance reasons
qry.EntityName = CRMO.SystemUser.EntityLogicalName; // get entity object SystemUser
qry.Criteria.AddCondition(new ConditionExpression("calendarid", ConditionOperator.NotNull)); // but non-builtin users
EntityCollection col = svc.RetrieveMultiple(qry); // executes query
foreach (Entity ent in col.Entities)
{
Console.WriteLine();
Console.WriteLine("Current Fullname: " + ent["fullname"].ToString());
Console.Write("Update? Y/N: ");
string ans = Console.ReadLine();
if (ans.ToLower() == "y")
{
// Create a new entity, setting the id and whatever attributes that need to be updated
var updateEntity = new Entity { Id = ent.Id };
updateEntity["firstname"] = ent["firstname"];
updateEntity["lastname"] = ent["lastname"];
svc.Update(updateEntity);
}
}
Console.WriteLine();
Console.WriteLine("--- Done ---");
Console.ReadLine();
}
}
Notes:
Only retrieve the columns you actually need
Create an update entity that only contains the fields you want to update
Remember that FullName is readonly
This may also be helpful

This is so others reading this can use this solution to change the FullName in CRM Online.
So in my case, where I needed to change the FullName of existing CRM users from "First Last" to "Last, First", I was able to perform regular Office 365 admin functions to complete this.
First, I changed the format in CRM Settings > System Settings to "Last Name, First Name".
Then, for each user I needed to have changed, I used the Office 365 Admin Center and edited their licenses. Un-assign the CRM license from the user and click SAVE. Wait about a minute or two for the changes to take affect. Next, go back into that same user management and re-assign the CRM license to the user, click SAVE. Wait a few minutes and you will see the FullName in CRM should be in the correct format.