We Keep Coding

Application sees zero/different tables from what local DynamoDB instance has

I'm building a WebAPI project. I have DynamoDB downloaded and running locally with -inMemory. After creating a couple of tables, I run this command locally: aws dynamodb list-tables --endpoint-url http://localhost:8000, which results in:
{
"TableNames": [
"Users",
"Tmp"
]
}
When I run my application, I create a client, and query for tables:
using (var client = DatabaseClientFactory.CreateClient())
{
// debugging
var temp = await client.ListTablesAsync();
return $"{temp.TableNames.Count} tables: " + string.Join(", ", temp.TableNames);
}
This returns 0 tables:
The DB client is a lightly-modified version of this starter code:
using System;
using System.Net;
using System.Net.NetworkInformation;
using Amazon;
using Amazon.DynamoDBv2;
namespace Database
{
// Adapted from https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.NET.01.html
public static class DatabaseClientFactory
{
/*-----------------------------------------------------------------------------------
* If you are creating a client for the Amazon DynamoDB service, make sure your credentials
* are set up first, as explained in:
* https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SettingUp.DynamoWebService.html,
*
* If you are creating a client for DynamoDBLocal (for testing purposes),
* DynamoDB-Local should be started first. For most simple testing, you can keep
* data in memory only, without writing anything to disk. To do this, use the
* following command line:
*
* java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -inMemory
*
* For information about DynamoDBLocal, see:
* https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html.
*-----------------------------------------------------------------------------------*/
private const int Port = 8000;
public static AmazonDynamoDBClient CreateClient()
{
// If this line throws, you need to download the AWS CLI, run `aws configure` and specify your access key.
var client = new AmazonDynamoDBClient();
var dynamoDbConfig = new AmazonDynamoDBConfig();
// First, check to see whether anyone is listening on the DynamoDB local port
// (by default, this is port 8000, so if you are using a different port, modify this accordingly)
var portAvailable = IsPortAvailable();
if (portAvailable)
{
Console.WriteLine(" -- The local instance of DynamoDB is NOT running. Using PROD.");
// TODO: this should come out of appSettings.Production.json
dynamoDbConfig.RegionEndpoint = RegionEndpoint.USEast2;
}
else
{
// Local address ignores AWS credentials
Console.WriteLine(" -- The local instance of DynamoDB is running!");
dynamoDbConfig.ServiceURL = $"http://localhost:{Port}";
}
client = new AmazonDynamoDBClient(dynamoDbConfig);
return client;
}
private static bool IsPortAvailable()
{
// Evaluate current system TCP connections. This is the same information provided
// by the netstat command line application, just in .Net strongly-typed object
// form. We will look through the list, and if our port we would like to use
// in our TcpClient is occupied, we will set isAvailable to false.
IPGlobalProperties ipGlobalProperties = IPGlobalProperties.GetIPGlobalProperties();
IPEndPoint[] tcpConnInfoArray = ipGlobalProperties.GetActiveTcpListeners();
foreach (IPEndPoint endpoint in tcpConnInfoArray)
{
if (endpoint.Port == Port)
{
return false;
}
}
return true;
}
}
}
That's odd - my two tables don't appear.
Even more strange: if I make calls like client.CreateTableAsync, they successfully create a table, and the return value of my API call becomes 1 tables: CREATED_IN_CODE
It seems like I have two versions of DynamoDB running locally on the same port - one the application accesses, and one that the CLI accesses.
Why is my app seeing a wrong/different list of tables?
Some other things I tried to no avail:
Nuking/reinstalling DynamoDB
Restarting my PC
Reverting to a previous version of my code from earlier today
Looking for the CREATED_IN_CODE table in AWS in different regions

The above code worked fine for me. Sometimes you need to be sure there is a region set, even though you are setting a ServiceUrl.

Azure Cosmos db throwing Socket Exceptions

I am using azure cosmos db with .net core 2.1 application. I am using gremlin driver with this. It's working fine but after every few days it start throwing socket exception on server and we have to recycle IIS pool. Average per day hits are 10000.
Now we are using default gateway mode. Should we have to switch to direct mode as it might be a firewall issue ?
Here is the implementation:
private DocumentClient GetDocumentClient( CosmosDbConnectionOptions configuration)
{
_documentClient = new DocumentClient(
new Uri(configuration.Endpoint),
configuration.AuthKey,
new ConnectionPolicy());
//create database if not exists
_documentClient.CreateDatabaseIfNotExistsAsync(new Database { Id = configuration.Database });
return _documentClient;
}
and in startup.cs:
services.AddSingleton(x => GetDocumentClient(cosmosDBConfig));
and here is how we are communicating with cosmos db:
private DocumentClient _documentClient;
private DocumentCollection _documentCollection;
private CosmosDbConnectionOptions _cosmosDBConfig;
public DocumentCollectionFactory(DocumentClient documentClient, CosmosDbConnectionOptions cosmosDBConfig)
{
_documentClient = documentClient;
_cosmosDBConfig = cosmosDBConfig;
}
public async Task<DocumentCollection> GetProfileCollectionAsync()
{
if (_documentCollection == null)
{
_documentCollection = await _documentClient.CreateDocumentCollectionIfNotExistsAsync(
UriFactory.CreateDatabaseUri(_cosmosDBConfig.Database),
new DocumentCollection { Id = _cosmosDBConfig.Collection },
new RequestOptions { OfferThroughput = _cosmosDBConfig.Throughput });
return _documentCollection;
}
return _documentCollection;
}
and then:
public async Task CreateProfile(Profile profile)
{
var graphCollection = await _graphCollection.GetProfileCollectionAsync();
var createQuery = GetCreateQuery(profile);
IDocumentQuery<dynamic> query = _documentClient.CreateGremlinQuery<dynamic>(graphCollection, createQuery);
if(query.HasMoreResults)
{
await query.ExecuteNextAsync();
}
}

I'm assuming that for communication with CosmosDB you are using HttpClient. The application should share a single instance of HttpClient.
Every time you make a connection after HttpClient disposal there are still a bunch of connections in the state of TIME_WAIT. This means that the connection was closed on one side ( OS ) but it is in "waiting for additional packets" state.
By default, Windows may hold this connection in this state for 240 seconds. There is a limit to how quickly OS can open new sockets. All this may lead to System.Net.Sockets.SocketException exception.
Very good article that explains in details why and how this problem appears digging into TCP diagram and explaining with more details.
UPDATED
Possible solution.
You are using the default ConnectionPolicy object. That object has a property called IdleTcpConnectionTimeout which controls the amount of idle time after which unused connections are closed. By default, idle connections are kept open indefinitely. The value must be greater than or equal to 10 minutes.
So the code could look like:
private DocumentClient GetDocumentClient( CosmosDbConnectionOptions configuration)
{
_documentClient = new DocumentClient(
new Uri(configuration.Endpoint),
configuration.AuthKey,
new ConnectionPolicy() {
IdleTcpConnectionTimeout = new TimeSpan(0,0,10,0)
});
//create database if not exists
_documentClient.CreateDatabaseIfNotExistsAsync(new Database { Id = configuration.Database });
return _documentClient;
}
Here is a link to ConnectionPolicy Class documentation

SQL Server Error while pulling hundreds of records

I am getting this error when I try to get all the username password from production copied local database, I guess it is because of not closing the connection properly, but I am not sure how . I am using the Microsoft Enterprise Library, ant thought or comment about it?
Timeout expired. The timeout period elapsed prior to
obtaining a connection from the pool. This may have occurred
because all pooled connections were in use and max pool size was reached.
this is the mothod that is getting the username and password and producing the error.
private Model.UsernameandPass GetUsernamePass(string AccountNumber)
{
Model.UsernameandPass model = null;
string myConnection = System.Configuration.ConfigurationManager.ConnectionStrings[connectionName].ToString();
SqlDatabase db = new SqlDatabase(myConnection);
using (DbCommand command = db.GetStoredProcCommand("Get_TheUsernamePassWordFromProduction"))
{
db.AddInParameter(command, "AccountNumber", DbType.String, AccountNumber);
var result = db.ExecuteReader(command);
try
{
while (result.Read())
{
model = new Model.UsernameandPass();
model.Username = result.GetString(1);
model.Password = result.GetString(2);
}
}
catch (Exception ex)
{
}
}
db = null;
return model;
}
I am getting the error in this line after program runs for a while.
var result = db.ExecuteReader(command);

You're getting that error because a connection cannot be established, not only because they aren't being closed properly. Check permissions for the user you're trying to authenticate against the database with. Also be sure to call .open()/.close() when/if you are programatically opening/closing connections.
Check this link. You may want to increase your pool size, or check for long-running queries.

Programmatically get site status from IIS, gets back COM error

I am trying to programmatically get my site status from IIS to see if it's stopped, but I kept getting the following error,
The object identifier does not represent a valid object. (Exception from HRESULT: 0x800710D8)
The application is using ServerManager Site class to access the site status. Here is the code,
//This is fine, gets back the site
var serverManager = new Microsoft.Web.Administration.ServerManager(ConfigPath);
var site = serverManager.Sites.FirstOrDefault(x => x.Id == 5);
if (site == null) return;
var appPoolName = site.Applications["/"].ApplicationPoolName;
//error!
var state = site.State;
I've test with static site to isolate the issue, making sure that the site is up and running, all configuration are valid, point to the valid application pool...etc.
Let me know if you need more details. Is it the COM thing?

I figured out where the problem is. Basically, there are two parts to the Server manager, the first part of the server manager allows you to read site details from configuration file, which is what I've been doing above. The problem with that is you will only able get the information that's in file and site state is not part of it.
The second part of the Server Manager allows you to connect to the IIS directly and it does this by interacting with the COM element. So what I should be doing is this:
ServerManager manager= ServerManager.OpenRemote("testserver");
var site = manager.Sites.First();
var status = site.State.ToString() ;

I had a similar problem but mine was caused by the delay needed to activate the changes from the call to CommitChanges on the ServerManager object. I found the answer I needed here:
ServerManager CommitChanges makes changes with a slight delay
It seems like polling is required to get consistent results. Something similar to this solved my problem (I got the exception when accessing a newly added application pool):
...
create new application pool
...
sman.CommitChanges();
int i = 0;
const int max = 10;
do
{
i++;
try
{
if (ObjectState.Stopped == pool.State)
{
write_log("Pool was stopped, starting: " + pool.Name);
pool.Start();
}
sman.CommitChanges();
break;
}
catch (System.Runtime.InteropServices.COMException e)
{
if (i < max)
{
write_log("Waiting for IIS to activate new config...");
Thread.Sleep(1000);
}
else
{
throw new Exception(
"CommitChanges timed out efter " + max + " attempts.",
e);
}
}
} while (true);
...

Some tricky quick way to validate oracle db connection

My WCF service need to check is connection available now and can we work with it. We have many remote dbs. Their connection are weird sometimes and can't be used to query data or smth else.
So, for example this is regular connection string used:
User Id=user;Password=P#ssw0rd;Data Source=NVDB1;Connection Timeout=30
Here is service method, used for getting
public List<string> GetAliveDBs(string city)
{
if (String.IsNullOrEmpty(city))
return null;
List<string> cityDbs = (from l in alldbs where !String.IsNullOrEmpty(l.Value.city) && l.Value.city.ToUpper() == city.ToUpper() select l.Value.connString).ToList();
// There is no such city databases
if (cityDbs.Count == 0)
return null;
ReaderWriterLockSlim locker = new ReaderWriterLockSlim();
Parallel.ForEach(cityDbs, p =>
{
if (!IsConnectionActive(p.connString))
{
locker.EnterWriteLock();
try
{
cityDbs.RemoveAt(cityDbs.IndexOf(p));
}
finally
{
locker.ExitWriteLock();
}
}
});
return cityDbs;
}
static public bool IsConnectionAlive(string connectionString)
{
using (OracleConnection c = new OracleConnection(connectionString))
{
try
{
c.Open();
if ((c.State == ConnectionState.Open) && (c.Ping()))
return true;
else
return false;
}
catch (Exception exc)
{
return false;
}
}
}
I use devart components to communicate with Oracle DB.
Hope for your help, guys! Thanks in advance!

Try just executing a very low cost operation that should work no matter what schema you are connected to, e.g.
SELECT 1
(that statement works on MS SQL and MySQL... should work on Oracle too but I can't confirm that).
If you get the result you expect (in this case one row, with one column, containing a "1") then the connection is valid.
At least one connection pool manager uses this strategy to validate connections periodically.
UPDATE:
Here's a SQL Server version of your method. You can probably just replace "Sql" with "Oracle".
static public bool IsConnectionAlive(string connectionString)
{
try
{
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand("SELECT 1", conn))
{
int result = (int)cmd.ExecuteScalar();
return (result == 1);
}
}
}
catch (Exception ex)
{
// You need to decide what to do here... e.g. does a malformed connection string mean the "connection isn't alive"?
// Maybe return false, maybe log the error and re-throw the exception?
throw;
}
}

If the goal is to simply determine if a server lives at the IP Address or host name then I'd recommend Ping (no 3 way handshake and has less overhead than a UDP message). You can use the System.Net.NetworkInformation.Ping class (see its documentation for an example) for this.
If you're looking to prove that there is actually something listening on the common Oracle port, I would suggest using either the System.Net.Sockets.TcpClient or System.Net.Sockets.Socket class (their documentation also provides examples) to provide this.
The simplest way to do this (by far) is to just open a connection using the Oracle API for C#. There is a very good tutorial that includes code here. It covers more than just the connection but you should be able to strip out the connection portion from the rest to fit your needs.

Oracle has products and software specifically for helping maintain high availability that can allow you to have dead connections removed from you connection pool through a setting called HA Events=true on the connection string. Your Oracle DBA will need to determine if your installation supports it.