We Keep Coding

datacollection program not correctly collecting

I created an data collection app for our company which collect data from our remote devices.
The data is collected from a datamailbox which is comparable with an database that works like an 10 day buffer to store the data. this is all correctly working.
The data is collected through post api requests. for example :
var url = BuildUrl("syncdata");
var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
var data = DynamicJson.Parse(response);
transactionId = data.transactionId;
I've been trying to collect multiple devices at one time but the problem is that it starts running and collect the data from the first device which works. Than our second device will start collecting the data but it only starts from where device one ended so i've been losing 12hours of data each run. For performance we use transactionId's.(each set of data has its own Id)
The workflow should be like this :
When the data is retrieved for the first time, the user specifies only
the createTransaction filter. The DataMailbox returns all the data of
all devices gateways – with historical data – of the account along a
transaction ID. For the next calls to the API, the client specifies
both createTransaction and lastTransactionId filters. The
lastTransactionId is the ID of the transaction that was returned by
the latest request. The system returns all the historical
data that has been received by the DataMailbox since the last
transaction and a new transaction ID. deviceIds is an additional
filter on the returned result. You must be cautious when using the
combination of lastTransactionId, createTransaction and deviceIds.
lastTransactionId is first used to determine what set of data — newer
than this transaction ID and from all the Device gateways — must be
returned from the DataMailbox, then deviceIds filters this set of data
to send data only from the desired device gateways. If a first request
is called with lastTransactionId, createTransaction and deviceIds, the
following request — implying a new lastTransactionId — does not
contain values history from the previous lastTransactionId of the
device gateways that were not in the deviceId from previous request.
I'm really struggling with the data collection and have no clue how to use the TransactionId and the LastTransActionId.This is the code for now
try
{
CheckLogin();
using (var webClient = new MyWebClient())
{
bool moreDataAvailable;
int samplesCount = 0;
string transactionId = Properties.Settings.Default.TransactionId;
string lastTransactionId = Properties.Settings.Default.LastTransactionId;
do
{
var url = BuildUrl("syncdata");
var response = webClient.CallApi(url, new NameValueCollection() { { "createTransaction","" }, { "lastTransactionId", transactionId } });
var data = DynamicJson.Parse(response);
transactionId = data.transactionId;
var talk2MMessage = getTalk2MMessageHeader(webClient);
if (talk2MMessage != null)
{
}
foreach (var ewon in data.ewons)
{
Directory.CreateDirectory(ewon.name);
foreach (var tag in ewon.tags)
{
try
{
Console.WriteLine(Path.Combine(ewon.name, tag.name + ""));
foreach (var sample in tag.history)
{
Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
Console.WriteLine(transactionId);
samplesCount++;
}
}
catch (RuntimeBinderException)
{ // Tag has no history. If it's in the transaction, it's most likely because it has alarm history
Console.WriteLine("Tag {0}.{1} has no history.", ewon.name, tag.name);
}
}
}
Console.WriteLine("{0} samples written to disk", samplesCount);
// Flush data received in this transaction
if (Properties.Settings.Default.DeleteData)
{
//Console.WriteLine("Flushing received data from the DataMailbox...");
url = BuildUrl("delete");
webClient.CallApi(url, new NameValueCollection() { { "transactionId", transactionId } });
Console.WriteLine("DataMailbox flushed.");
}
//save the transaction id for next run of this program
Properties.Settings.Default.LastTransactionId = lastTransactionId;
Properties.Settings.Default.Save();
// Did we receive all data?
try
{
moreDataAvailable = data.moreDataAvailable;
}
catch (RuntimeBinderException)
{ // The moreDataAvailable flag is not specified in the server response
moreDataAvailable = false;
}
if (moreDataAvailable)
Console.WriteLine("There's more data available. Let's get the next part...");
}
while (moreDataAvailable);
Here are my credentials for starting the collection like all parameters
static void CheckLogin()
{
if (string.IsNullOrEmpty(Properties.Settings.Default.Talk2MDevId))
{
Properties.Settings.Default.Talk2MDevId = Prompt("Talk2MDevId");
Properties.Settings.Default.APIToken = Prompt("API Token");
string deleteInputString = Prompt("Delete data after synchronization? (yes/no)");
Properties.Settings.Default.DeleteData = deleteInputString.ToLower().StartsWith("y");
Properties.Settings.Default.TransactionId = "";
Properties.Settings.Default.LastTransactionId = "";
Properties.Settings.Default.Save();
}
I think it's something with the transactionId and LastTransaction id but i have no clue.
More information can be found here: https://developer.ewon.biz/system/files_force/rg-0005-00-en-reference-guide-for-dmweb-api.pdf

As I understand your question, you problem is that for the first few transactionIds, you only get data from device 1 and then only data from device 2.
I'm assuming the following in my answer:
You didn't specify somewhere else in code the filter on "ewonid"
When you say you lose 12 hours of data , you are assuming it because "device 2" data are streamed after "device 1" data.
You did try without the /delete call with no change
/syncdata is an endpoint that returns a block of data for an account since a given transactionId (or oldest block if you didn't provide a transactionID). This data is sorted by storage date by the server, which depends on multiple factors:
when was the device last "vpn online"
at which frequency the device is pushing data to datamailbox
when was that device packet digested by datamailbox service
You could technically have 1 year old data pushed by a device that gets connected back to vpn now, and those data would be registered in the most recent blocks.
For those reasons, the order of data block is not the order of device recording timestamp. You always have to look at the field ewons[].tags[].history[].date to known when that measure was made.
foreach (var sample in tag.history)
{
Console.WriteLine(ewon.name + " " + tag.name + " " + tag.description);
Console.WriteLine(sample.value + " at " + sample.date);
Console.WriteLine(transactionId);
samplesCount++;
}
In your case, I would assume both devices are configured to push their data once a day, one pushing it's backlog, let's say, at 6AM and the other at 6PM.

ServiceNow - Getting all records

In ServiceNow, I am able to get only a maximum of 250 records in a SOAP request. How to get all the records?
Web Reference Url = https://*****.service-now.com/rm_story.do?WSDL
Code:
var url = "https://*****.service-now.com/rm_story.do?SOAP";
var userName = *****;
var password = *****;
var proxy = new ServiceNow_rm_story
{
Url = url,
Credentials = new NetworkCredential(userName, password)
};
try
{
var objRecord = new Namespace.WebReference.getRecords
{
// filters..
};
var recordResults = proxy.getRecords(objRecord);
}
catch (Exception ex)
{
}
In recordResults, I am getting only 250 records. How to get all the records ?

Also see this stack overflow answer which provides info.
Get ServiceNow Records Powershell - More than 250
Note that returning a large number of records can affect performance of the response and it may be more efficient to process your query in batches using offsets (i.e., get 1-100, then 101-200, ...). This can be achieved by using a sort order and offset. The ServiceNow REST Table API actually returns link headers from Get requests providing you links for the first, next and last set of records making it easy to know the url to query the next batch of records.
See: http://wiki.servicenow.com/index.php?title=Table_API#Methods
and look under 'Response Header'.

Have u tried to pass/override __limit parameter?
Google / wiki / Users manual / Release notes are always helpful

In your code snippet in line where it says //filter you should define __limit (and potentially __first_row and __last_row as explained in the example bellow)
int Skip = 0;
int Take = 250;
while (true)
{
using (var soapClient = new ServiceNowLocations.ServiceNow_cmn_location())
{
var cred = new System.Net.NetworkCredential(_user, _pass);
soapClient.Credentials = cred;
soapClient.Url = _apiUrl + "cmn_location.do?SOAP";
var getParams = new ServiceNowLocations.getRecords()
{
__first_row = Skip.ToString(),
__last_row = (Skip + Take).ToString(),
__limit = Take.ToString()
};
var records = soapClient.getRecords(getParams);
if (records != null)
{
if (records.Count() == 0)
{
break;
}
Skip += records.Count();
if (records.Count() != Take)
{
// last batch or everything in first batch
break;
}
}
else
{
// service now web service endpoint not configured correctly
break;
}
}
}

I made an library that handles interacting with ServiceNow Rest API much easier
https://emersonbottero.github.io/ServiceNow.Core/

Getting inner exception "dtd is prohibited in this xml document exchange" while reading emails from outlook

Getting error "The response received from the service didn't contain valid XML." with
inner exception "dtd is prohibited in this xml document exchange" while reading
emails from outlook(Not while reading every mail).
Can anybody please tell me what might be the issue. Below is the code where I am
getting error
FindItemsResults<Item> RetrievedItems=null ;
RetrievedItems = service.FindItems(FIds, new ItemView(4));
String[] SignatureList = ConfigurationManager.AppSettings.Get("SignatureTypes").Split(',');
if (RetrievedItems != null && RetrievedItems.Count() > 0)
{
RetrievedItems.ToList().ForEach(x =>
{
try
{
List<String> Attachments = new List<String>();
List<String> ScanFileName = new List<String>();
bool IsAvailable = true;
//Getting error while Load() - below line of code
((Microsoft.Exchange.WebServices.Data.EmailMessage)x).Load();
Vo.EmailMessage msg = new Vo.EmailMessage();
msg.MessageId = ((Microsoft.Exchange.WebServices.Data.EmailMessage)x).Id.UniqueId;
msg.From = ((Microsoft.Exchange.WebServices.Data.EmailMessage)x).From.Address;
((Microsoft.Exchange.WebServices.Data.EmailMessage)x).ToRecipients.ToList().ForEach(z => msg.To += z.Address + ",");
((Microsoft.Exchange.WebServices.Data.EmailMessage)x).ReplyTo.ToList().ForEach(y => msg.ReplyToEmailAddress += y.Address + ",");
msg.Subject = ((Microsoft.Exchange.WebServices.Data.EmailMessage)x).Subject;
msg.Body = ((Microsoft.Exchange.WebServices.Data.EmailMessage)x).Body.Text;
msg.Dated = ((Microsoft.Exchange.WebServices.Data.EmailMessage)x).DateTimeSent;

I have also seen this error. What I have found is that making the exact same call even 1 second later and it works. I've worked with Microsoft, and their "feeling" is that an intermediary device [in my case an F5] is modifying the payload before the Exchange Server gets it. I do know that if we bypass the F5 completely, the problem goes away.

Load URL in for loop in C#

I am sending a message to multiple phone numbers .
Mobile numbers are stored in an array .
string phNums = "91999999999,9199999998....";.
string[] phNos = phNums.Split(',');
But message doesn't reach all of the recipients , mostly to the numbers that are present near end of array.
The message are sent via a URL provided by SMS service provider in which the phone number and the message is embedded.
for (int i = 0; i < phNos.Length; i++)
{
url = #"http://aaa.bbb.ccc.dd/HTTPMTAPI?User=abc&Password=pqr&FromAddr=xyzSMS&DestNo=" + phNos[i] + "&msg=" + message;
Uri targetUri1 = new Uri(url);
System.Net.HttpWebRequest hwb1;
hwb1 = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(targetUri1);
hwb1.GetResponse();
}
As an alternate , I also used Webclient() but still successful message delivery is not guaranteed.
for (int i = 0; i < phNos.Length; i++)
{
WebClient cli= new WebClient();
url = #"http://aaa.bbb.ccc.dd/HTTPMTAPI?User=abc&Password=pqr&FromAddr=xyzSMS&DestNo=" + phNos[i] + "&msg=" + message;
cli.DownloadString(url);
}
How to ensure that message delivery is not skipped .
Like only if successful response is received on downloading the URL , the loop should progress to next mobile number and so on.
If there is any other possible mechanism , please do suggest. Thanks

I think this is what you want to do:
for (int i = 0; i < phNos.Length; i++)
{
url = #"http://aaa.bbb.ccc.dd/HTTPMTAPI?User=abc&Password=pqr&FromAddr=xyzSMS&DestNo=" + phNos[i] + "&msg=" + message;
Uri targetUri1 = new Uri(url);
System.Net.HttpWebRequest hwb1;
hwb1 = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(targetUri1);
System.Net.HttpWebResponse response = hwb1.GetResponse();
if (response != null)
{
int status = (int)response.StatusCode; // this changes the status
// from text response to the
// number, like 404
if (status == 404//or anything else you want to test//)
{
// put your retry logic here, make sure you add a way to break
// so you dont infinitely loop if the service is down or something
}
}
}

URLs have a length constraint. Your are likely hitting this limit and your are losing the trailing phone numbers as a result. Your best bet would be to break your requests into multiple requests of a certain size.
According to the following SO, it may be good practice to limit your request so that the url does not surpass 2000 characters.
What is the maximum length of a URL in different browsers?

Weird Error when using SOAP Web Service

For a school project, I have to create a Java Server that is able to service a .NET client that has a Service Reference to a WSDL. I have the .NET Client code:
using (var client = new MathServiceWSDLClient())
{
Console.WriteLine("Multiplying " + Num1 + " and " + Num2 + ": " + client.Multiply(Num1, Num2));
Console.WriteLine("Adding" + Num1 + " and " + Num2 + ": " + client.Add(Num1, Num2));
}
I have also written the Java Server:
Scanner sc = new Scanner(socket.getInputStream());
boolean clientExpectContinue = false;
int contentLength = -1;
String line;
while (!(line = sc.nextLine()).isEmpty()) {
System.out.println(line);
if (line.startsWith("Content-Length")) {
String[] elements = line.split(": ");
contentLength = Integer.parseInt(elements[1]);
} else if (line.startsWith("Expect")) {
clientExpectContinue = true;
}
}
int notEmpties = 0;
byte[] soapEnvelopeData = new byte[contentLength];
char[] soapChars = new char[contentLength];
for (int i = 0; i < contentLength; i++) {
soapChars[i] = (char) socket.getInputStream().read();
if (i == 0)
System.out.println("DFSDFSDf");
}
// System.out.println(socket.getInputStream().read(soapEnvelopeData));
System.out.println(soapEnvelopeData.length);
File file = new File("tempEnvelope.txt");
FileOutputStream fileOut = new FileOutputStream(file);
// fileOut.write(soapEnvelopeData);
System.out.println("Content!");
System.out.println(new String(soapChars));
fileOut.write(new String(soapChars).getBytes());
fileOut.flush();
fileOut.close();
/* Some fancy SOAP and Reflection stuff that works */
The gist of the server is as follows:
It gets the incomming request, reads through the headers and finds the content length. It parses and saves this. From here there are two versions. The first constructs a byte array the same size as the content length, and passes the array as a parameter into the socket.getInputStream().read() method. The second constructs a char array the same length of the content and then reads individual bytes from the stream and casts them to chars.
The issue comes into play when, as shown, I attempt to run the .NET client with multiple requests in one execution. The first request goes off without any sort of discernible issue. When the second one comes in, the server reads the headers, gets the content length and constructs the array. When it comes time to read from the socket, however, the program just waits. Using the char array method, I was able to learn that it waits when reading the first value. Once one minute has expired, .NET times out, and the rest of the program breaks.
If, however, I leave the server running, and only have one request per execution of the .NET client, everything is just fine; the response comes back just as it should.
I have tried some solutions already:
Creating a new MathServiceWSDLClient for every request
Putting every request in its own using() block.
Doing two of the same request at once: two Multiply() or Add() requests.
Any help is appreciated. Thank you all in advance.
~Ryno Janse van Rensburg

.net is probably keeping the socket open for subsequent requests. There may be a bug in your server-like java code related to this.
Are you able to use a framework instead for the Java server code? I would strongly recommend this, and recommend rmbedded Jetty without hesitation. This would mean you wouldn't have to work at the socket level in Java - you can let Jetty handle all of that complication.