We Keep Coding

Updating BigQuery dataset access from C#

I am working with BigQuery. I create a DataSet and I want to define access rights with C# language.
It's not clear to me how to do it.
In GOOGLE web page https://cloud.google.com/bigquery/docs/dataset-access-controls is explained how to do it with some example for Java and Pyton (see below), but no example is provided for c#.
example in pyton:
dataset = client.get_dataset(dataset_id) # Make an API request.
entry = bigquery.AccessEntry(
role="READER",
entity_type="userByEmail",
entity_id="sample.bigquery.dev#gmail.com",
)
entries = list(dataset.access_entries)
entries.append(entry)
dataset.access_entries = entries
dataset = client.update_dataset(dataset, ["access_entries"]) # Make an API request.
full_dataset_id = "{}.{}".format(dataset.project, dataset.dataset_id)
Can anybody help please?

It's probably best to use the BigQueryDataset.Patch method, from the Google.Cloud.BigQuery.V2 package:
// Fetch the existing dataset
var client = BigQueryClient.Create(projectId);
var dataset = client.GetDataset(datasetId);
var accessList = dataset.Resource.Access ?? new List<AccessData>();
accessList.Add(new AccessData
{
Role = "Reader",
UserByEmail = "sample.bigquery.dev#gmail.com"
});
var patchedResource = new Dataset { Access = accessList };
// Push the changes back up to BigQuery
dataset.Patch(patchedResource, matchETag: true);
As an alternative, you can use Update to update the replace the dataset resource completely:
// Fetch the existing dataset
var client = BigQueryClient.Create(projectId);
var dataset = client.GetDataset(datasetId);
// Modify it in memory
var resource = dataset.Resource;
if (resource.Access is null)
{
// If there's no access list yet, create one.
resource.Access = new List<AccessData>();
}
var newAccess = new AccessData
{
Role = "Reader",
UserByEmail = "sample.bigquery.dev#gmail.com"
};
resource.Access.Add(newAccess);
// Push the changes back up to BigQuery
dataset.Update();

At the and I managed to make it work.
I used the same solution suggested by Jon Skeet, using the Patch method.
I attach my code here by.
public static bool GrantDatasetAccess(string dataSetId, bool online, string role, string email,
string bigQueryJsonPath, string bigQueryScope, string projectId, clsLog log)
{
try
{
BigQueryClient client = GetBigQueryClient(online, bigQueryJsonPath, bigQueryScope, projectId);
BigQueryDataset dataset = GetDataSet(online, dataSetId, bigQueryJsonPath, bigQueryScope, projectId, log);
List<AccessData> accessList = new List<AccessData>();
var accessData = new AccessData()
{
Role = role,
GroupByEmail = null,
UserByEmail = email,
SpecialGroup = null,
IamMember = null,
Domain = null,
View = null
};
accessList.Add(accessData);
dataset.Resource.Access = accessList;
dataset.Patch(dataset.Resource, true);
}
catch (Exception e)
{
log.ManageError("Error GetDataSet: {0}\nError: {1}", dataSetId, string.Concat(e.Message, "\n", e.StackTrace));
}
return true;
}

CosmosDB Entity with the specified id does not exist

I am developing an ASP.NET Core MVC API to call resources in an Azure Cosmos DB. When I try to perform a GET for any specific ID, I receive DocumentClientException: Entity with the specified id does not exist in the system. I can confirm that the entity does exist in the system, and the connection is successful because I can successfully perform other methods and requests. The partition key is _id .
Debugging with breakpoints in Visual Studio, I can see where the correct ID is received at the API, but I can't confirm what specifically it is sending to Azure
The controller methods: (the ID field is a random string of numbers and text)
//controller is MoviesController decorated with [Route(api/[controller])]
//sample GET is to localhost:port/api/Movies/5ca6gdwndkna99
[HttpGet("{id}")]
public async Task<MoviesModel> Get(string id)
{
MoviesModel movie = await _persistence.GetMovieAsync(id);
return movie;
}
The data handling method:
public async Task<MoviesModel> GetMovieAsync(string Id)
{
string _id = Id;
RequestOptions options = new RequestOptions();
options.PartitionKey = new PartitionKey(_id);
var documentUri = UriFactory.CreateDocumentUri(_databaseId, "movies", Id);
Document result = await _client.ReadDocumentAsync(documentUri,options);
return (MoviesModel)(dynamic)result;
}
Other methods, like getting a list of all movies and returning to a table are working fine, so we can rule out network issues
public async Task<List<MoviesModel>> GetMoviesAsync()
{
var documentCollectionUri = UriFactory.CreateDocumentCollectionUri(_databaseId, "movies");
// build the query
var feedOptions = new FeedOptions() { EnableCrossPartitionQuery = true };
var query = _client.CreateDocumentQuery<MoviesModel>(documentCollectionUri, "SELECT * FROM movies", feedOptions);
var queryAll = query.AsDocumentQuery();
// combine the results
var results = new List<MoviesModel>();
while (queryAll.HasMoreResults)
{
results.AddRange(await queryAll.ExecuteNextAsync<MoviesModel>());
}
return results;
}
public async Task<List<GenresModel>> GetGenresAsync()
{
await EnsureSetupAsync();
var documentCollectionUri = UriFactory.CreateDocumentCollectionUri(_databaseId, "genres");
// build the query
var feedOptions = new FeedOptions() { EnableCrossPartitionQuery = true };
var query = _client.CreateDocumentQuery<GenresModel>(documentCollectionUri, "SELECT * FROM genres", feedOptions);
var queryAll = query.AsDocumentQuery();
// combine the results
var results = new List<GenresModel>();
while (queryAll.HasMoreResults)
{
results.AddRange(await queryAll.ExecuteNextAsync<GenresModel>());
}
return results;
}

Firstly, I would suggest to re-look at your cosmosDb design once, bcz of the following reasons...
Problems:
If your _id is random string of numbers and text, then its not good
to have the entire _id as your partition key, bcz this would create a
new partition for each entry.(although azure will range parition it
later)
Querying just by partition key is not efficient, for pin point
queries we should have both partition key and row key.
Solution:
Make the first one or two letters of your _id as your partition key. (so your partitions will be finite).
Make your _id as your row key.
If your _id = "abwed123asdf", then your query should be..
RequestOptions options = new RequestOptions();
options.PartitionKey = new PartitionKey(_id.Substring(0,1));
options.RowKey = _id;
This way, your look up will pin point to the exact required entry with the help of partition and row key. (saves lot of RUs)
Please refer docs for choosing a better partition keys for your needs https://learn.microsoft.com/en-us/azure/cosmos-db/partitioning-overview

I was able to get this to work by completely refactoring to the dotnet v3 SDK. My code for the solution is in the comments of the gitHub link:
using Microsoft.Azure.Cosmos;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using VidlyAsp.DataHandlers;
namespace VidlyAsp.DataHandlers
{
public class PersistenceNew
{
private static string _endpointUri;
private static string _primaryKey;
private CosmosClient cosmosClient;
private CosmosDatabase database;
private CosmosContainer movieContainer;
private CosmosContainer genreContainer;
private string containerId;
private string _databaseId;
public PersistenceNew(Uri endpointUri, string primaryKey)
{
_databaseId = "Vidly";
_endpointUri = endpointUri.ToString();
_primaryKey = primaryKey;
this.GetStartedAsync();
}
public async Task GetStartedAsync()
{
// Create a new instance of the Cosmos Client
this.cosmosClient = new CosmosClient(_endpointUri, _primaryKey);
database = await cosmosClient.Databases.CreateDatabaseIfNotExistsAsync(_databaseId);
CosmosContainer moviesContainer = await GetOrCreateContainerAsync(database, "movies");
CosmosContainer genresContainer = await GetOrCreateContainerAsync(database, "genres");
movieContainer = moviesContainer;
genreContainer = genresContainer;
}
public async Task<GenresModel> GetGenre(string id)
{
var sqlQueryText = ("SELECT * FROM c WHERE c._id = {0}", id).ToString();
var partitionKeyValue = id;
CosmosSqlQueryDefinition queryDefinition = new CosmosSqlQueryDefinition(sqlQueryText);
CosmosResultSetIterator<GenresModel> queryResultSetIterator = this.genreContainer.Items.CreateItemQuery<GenresModel>(queryDefinition, partitionKeyValue);
List<GenresModel> genres = new List<GenresModel>();
while (queryResultSetIterator.HasMoreResults)
{
CosmosQueryResponse<GenresModel> currentResultSet = await queryResultSetIterator.FetchNextSetAsync();
foreach (GenresModel genre in currentResultSet)
{
genres.Add(genre);
}
}
return genres.FirstOrDefault();
}
public async Task<MoviesModel> GetMovie(string id)
{
var sqlQueryText = "SELECT * FROM c WHERE c._id = '" + id + "'";
var partitionKeyValue = id;
CosmosSqlQueryDefinition queryDefinition = new CosmosSqlQueryDefinition(sqlQueryText);
CosmosResultSetIterator<MoviesModel> queryResultSetIterator = this.movieContainer.Items.CreateItemQuery<MoviesModel>(queryDefinition, partitionKeyValue);
List<MoviesModel> movies = new List<MoviesModel>();
while (queryResultSetIterator.HasMoreResults)
{
CosmosQueryResponse<MoviesModel> currentResultSet = await queryResultSetIterator.FetchNextSetAsync();
foreach (MoviesModel movie in currentResultSet)
{
movies.Add(movie);
}
}
return movies.FirstOrDefault();
}
/*
Run a query (using Azure Cosmos DB SQL syntax) against the container
*/
public async Task<List<MoviesModel>> GetAllMovies()
{
List<MoviesModel> movies = new List<MoviesModel>();
// SQL
CosmosResultSetIterator<MoviesModel> setIterator = movieContainer.Items.GetItemIterator<MoviesModel>(maxItemCount: 1);
while (setIterator.HasMoreResults)
{
foreach (MoviesModel item in await setIterator.FetchNextSetAsync())
{
movies.Add(item);
}
}
return movies;
}
public async Task<List<GenresModel>> GetAllGenres()
{
List<GenresModel> genres = new List<GenresModel>();
// SQL
CosmosResultSetIterator<GenresModel> setIterator = genreContainer.Items.GetItemIterator<GenresModel>(maxItemCount: 1);
while (setIterator.HasMoreResults)
{
foreach (GenresModel item in await setIterator.FetchNextSetAsync())
{
genres.Add(item);
}
}
return genres;
}
private static async Task<CosmosContainer> GetOrCreateContainerAsync(CosmosDatabase database, string containerId)
{
CosmosContainerSettings containerDefinition = new CosmosContainerSettings(id: containerId, partitionKeyPath: "/_id");
return await database.Containers.CreateContainerIfNotExistsAsync(
containerSettings: containerDefinition,
throughput: 400);
}
}
}

table mapping unable error in SQlite connection in windows application

When i connect my application with SQLite database. The following exception throw.
An exception of type 'SQLite.SQLiteException' occurred in
Scrap_Book.Windows.exe but was not handled in user code
Additional information: no such table: CONTENT
I created the table with the help of SqLite manager an adon of mozila firefox.
Please help me how I can solve this problem.
var dbpath = Path.Combine(Windows.Storage.ApplicationData.Current.LocalFolder.Path, "MYSCRAPBOOK.sqlite");
using (var db = new SQLite.SQLiteConnection(dbpath))
db.Insert(new CONTENT()
{
STICKERS=snodesticker
});
db.Commit();
db.Dispose();
db.Close();
var line = new MessageDialog("Records Inserted");
await line.ShowAsync();

According to message from your exception you haven't created CONTENT table.
So, I guess you need to create table before inserting data.
public static string DB_NAME = "ContactsManager.sqlite";
public static string DB_PATH = Path.Combine(Path.Combine(ApplicationData.Current.LocalFolder.Path, DB_NAME));//DataBase Name
private async Task<bool> CheckFileExistsAsync(string fileName)
{
try
{
var store = await Windows.Storage.ApplicationData.Current.LocalFolder.GetFileAsync(fileName);
return true;
}
catch
{
}
return false;
}
if (!(await CheckFileExistsAsync(DB_NAME)))
{
using (var db = new SQLiteConnection(DB_PATH))
{
db.CreateTable<CONTENT>();
}
}

Server to Client back to server synchronization

I am developing a WPF application that interacts with localdb using the Entity Framework 6.0
So the past couple of weeks I've been attempting to get synchronization setup between the following database types:
Server: SQL Server 2008 Database
Client: localdb (SQL Express)
While I can get the structure of the database from the server transferred onto the client it does not bring over the relationships.
This somehow changes the way the ADO.NET Entity Data Model is generated.
When the Entity Data Model is generated for the server structure it generates the many-to-one-to-many relationships as a collection (many-to-many), but on the localdb that is generated (with no relationships generated) it keeps the table structure. This causes issues with my application. I can't fully migrate to an offline application if the synchronization doesn't work properly.
Server to Client (Initialization of client):
using System;
using System.IO;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Data;
using System.Data.SqlClient;
using Microsoft.Synchronization;
using Microsoft.Synchronization.Data;
using Microsoft.Synchronization.Data.SqlServer;
namespace DatabaseSetup
{
class Program
{
static void Main()
{
try
{
CreateLocalDatabase();
ProvisionServer();
ProvisionClient();
Sync();
}
catch (Exception ex)
{
Console.WriteLine($"Exception thrown: {ex.Source}");
Console.WriteLine($"Exception thrown: {ex.Data}");
Console.WriteLine($"Exception thrown: {ex.Message}");
Console.ReadLine();
}
}
public static void CreateLocalDatabase()
{
var conn = new SqlConnection(LocalDb);
var command = new SqlCommand(#"CREATE DATABASE ********", conn);
try
{
conn.Open();
Console.WriteLine(command.ExecuteNonQuery() <= 0
? #"Creating '********' Database on '(localdb)\v11.0'"
: #"Database '********' already exists. Attempting to synchronize.");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if(conn.State == ConnectionState.Open)
conn.Close();
}
}
public static void ProvisionServer()
{
Console.WriteLine(#"Attempting to provision server for synchronization...");
// connect to server database
var serverConnection = new SqlConnection(Server);
// define a new scope named ProductsScope
var scopeDescription = DatabaseScope(serverConnection);
// create a server scope provisioning object based on the ProductScope
var serverProvision = new SqlSyncScopeProvisioning(serverConnection, scopeDescription);
if(!serverProvision.ScopeExists("DatabaseScope"))
{ // skipping the creation of table since table already exists on server
serverProvision.SetCreateTableDefault(DbSyncCreationOption.Skip);
// start the provisioning process
serverProvision.Apply();
Console.WriteLine(#"Provisioning complete.");
}
else
{
Console.WriteLine(#"Server already provisioned.");
}
}
public static void ProvisionClient()
{
Console.WriteLine(#"Attempting to provision client for synchronization...");
// create a connection to the SyncExpressDB database
var clientConnection = new SqlConnection(Client);
// create a connection to the SyncDB server database
var serverConnection = new SqlConnection(Server);
// get the description of ProductsScope from the SyncDB server database
var scopeDesc = DatabaseScope(serverConnection);
// create server provisioning object based on the ProductsScope
var clientProvision = new SqlSyncScopeProvisioning(clientConnection, scopeDesc);
if (!clientProvision.ScopeExists("DatabaseScope"))
{
// starts the provisioning process
clientProvision.Apply();
Console.WriteLine(#"Provisioning complete.");
}
else
{
Console.WriteLine(#"Client already provisioned.");
}
}
public static void Sync()
{
//Define conections
Console.WriteLine(#"Attempting to synchronize.");
var serverConnection = new SqlConnection(Server);
var clientConnection = new SqlConnection(Client);
//Create Sync Orchestrator
var syncOrchestrator = new SyncOrchestrator
{
Direction = SyncDirectionOrder.DownloadAndUpload,
LocalProvider = new SqlSyncProvider("DatabaseScope", clientConnection),
RemoteProvider = new SqlSyncProvider("DatabaseScope", serverConnection)
};
((SqlSyncProvider)syncOrchestrator.LocalProvider).ApplyChangeFailed += Program_ApplyChangeFailed;
var syncStats = syncOrchestrator.Synchronize();
Console.WriteLine("\n\nSynchronization complete:");
Console.WriteLine($"Start Time: {syncStats.SyncStartTime}");
Console.WriteLine($"Uploaded: {syncStats.UploadChangesTotal}");
Console.WriteLine($"Downloaded: {syncStats.DownloadChangesTotal}");
Console.WriteLine($"Time Elapsed: {syncStats.SyncEndTime}");
Console.Read();
}
private static DbSyncScopeDescription DatabaseScope(SqlConnection connection)
{
//Define scope
var scopeTables = new Collection<DbSyncTableDescription>();
foreach (var table in TableList)
{
scopeTables.Add(SqlSyncDescriptionBuilder.GetDescriptionForTable(table, connection));
}
var databaseScope = new DbSyncScopeDescription("DatabaseScope");
foreach(var table in scopeTables)
{
databaseScope.Tables.Add(table);
}
return databaseScope;
}
static void Program_ApplyChangeFailed(object sender, DbApplyChangeFailedEventArgs e)
{
// display conflict type
Console.WriteLine(e.Conflict.Type);
// display error message
Console.WriteLine(e.Error);
}
}
}
and this is the action I setup to happen when the user clicks a button on the application whenever they want to sync:
Synchronization between client and server:
using System;
using System.Collections.ObjectModel;
using System.Data.SqlClient;
using System.Windows;
using Microsoft.Synchronization;
using Microsoft.Synchronization.Data;
using Microsoft.Synchronization.Data.SqlServer;
using Outreach.Resources;
namespace WpfApplication
{
class DatabaseSynchronization
{
public static void Sync()
{
//Define conections
Console.WriteLine(#"Attempting to synchronize.");
var clientConnection = new SqlConnection(Constants.Client);
var serverConnection = new SqlConnection(Constants.Server);
//Create Sync Orchestrator
var syncOrchestrator = new SyncOrchestrator();
syncOrchestrator.LocalProvider = new SqlSyncProvider("DatabaseScope", clientConnection);
syncOrchestrator.RemoteProvider = new SqlSyncProvider("DatabaseScope", serverConnection);
syncOrchestrator.Direction = SyncDirectionOrder.UploadAndDownload;
((SqlSyncProvider)syncOrchestrator.LocalProvider).ApplyChangeFailed += Program_ApplyChangeFailed;
var syncStats = syncOrchestrator.Synchronize();
Console.WriteLine("\n\nSynchronization complete:");
Console.WriteLine($"Start Time: {syncStats.SyncStartTime}");
Console.WriteLine($"Uploaded: {syncStats.UploadChangesTotal}");
Console.WriteLine($"Downloaded: {syncStats.DownloadChangesTotal}");
Console.WriteLine($"Time Elapsed: {syncStats.SyncEndTime}");
Console.Read();
}
static void Program_ApplyChangeFailed(object sender, DbApplyChangeFailedEventArgs e)
{
// display conflict type
Console.WriteLine(e.Conflict.Type);
// display error message
Console.WriteLine(e.Error);
}
private static DbSyncScopeDescription DatabaseScope(SqlConnection connection)
{
//Define scope
var scopeTables = new Collection<DbSyncTableDescription>();
foreach(var table in Constants.MsoTableList)
{
scopeTables.Add(SqlSyncDescriptionBuilder.GetDescriptionForTable(table, connection));
}
var outreachScope = new DbSyncScopeDescription("DatabaseScope");
foreach(var table in scopeTables)
{
outreachScope.Tables.Add(table);
}
return outreachScope;
}
public static void ProvisionServer()
{
Console.WriteLine(#"Attempting to provision server for synchronization...");
// connect to server database
var serverConnection = new SqlConnection(Constants.Server);
// define a new scope named ProductsScope
var scopeDescription = DatabaseScope(serverConnection);
// create a server scope provisioning object based on the ProductScope
var serverProvision = new SqlSyncScopeProvisioning(serverConnection, scopeDescription);
if(!serverProvision.ScopeExists("DatabaseScope"))
{ // skipping the creation of table since table already exists on server
serverProvision.SetCreateTableDefault(DbSyncCreationOption.Skip);
// start the provisioning process
serverProvision.Apply();
Console.WriteLine(#"Provisioning complete.");
}
else
{
Console.WriteLine(#"Server already provisioned.");
}
}
public static void ProvisionClient()
{
Console.WriteLine(#"Attempting to provision client for synchronization...");
// create a connection to the SyncExpressDB database
var clientConnection = new SqlConnection(Constants.Client);
// create a connection to the SyncDB server database
var serverConnection = new SqlConnection(Constants.Server);
// get the description of ProductsScope from the SyncDB server database
var scopeDesc = DatabaseScope(serverConnection);
// create server provisioning object based on the ProductsScope
var clientProvision = new SqlSyncScopeProvisioning(clientConnection, scopeDesc);
if(!clientProvision.ScopeExists("DatabaseScope"))
{
// starts the provisioning process
clientProvision.Apply();
Console.WriteLine(#"Provisioning complete.");
}
else
{
Console.WriteLine(#"Client already provisioned.");
}
}
}
}
Is there something I'm doing wrong?
Are there better options than Sync Framework?

Sync Framework don't do schema syncs. it provisions the tables just so you sync data. If you want to include the full server schema, you'll have to script it yourself and execute on the client. if you're only after synching the FKs as well, you can include that as part of the db sync table description when you provision.

Create/Update method not allowed in "Activity Pointer" CRM 2011

Am having a plugin that moves the activities of lead to opportunity on qualify. I have registered the plugin on "Create" of opportunity and the following is the code
public void Execute(IServiceProvider serviceProvider)
{
IPluginExecutionContext context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
Entity entity;
if (context.InputParameters.Contains("Target") && context.InputParameters["Target"] is Entity)
{
entity = (Entity)context.InputParameters["Target"];
if (entity.Attributes.Contains("originatingleadid") == false) return;
}
else
{
return;
}
try
{
//.....
EntityReference Lead = (EntityReference)entity.Attributes["originatingleadid"];
Guid LeadGuid = Lead.Id;
MoveActivitiesFromLeadToOpportunity(service, LeadGuid, Opportunityid);
}
catch (FaultException<OrganizationServiceFault> ex)
{
throw new InvalidPluginExecutionException(
"An error occurred in the plug-in.", ex);
}
}
private void MoveActivitiesFromLeadToOpportunity(IOrganizationService service, Guid LeadID, Guid OpportunityID)
{
ConditionExpression condition = new ConditionExpression();
condition.AttributeName = "regardingobjectid";
condition.Operator = ConditionOperator.Equal;
condition.Values.Add(LeadID.ToString());
var query = new QueryExpression("activitypointer");
query.Criteria.AddCondition(condition);
//query.Conditions.Add("reagrdingobjectid", ConditionOperator.Equal, theIdOfTheRelatedRecord);
query.ColumnSet = new ColumnSet(true);
var activities = service.RetrieveMultiple(query).Entities;
foreach (var activity in activities)
{
var castedActivity = (ActivityPointer)activity;
if (castedActivity.ActivityTypeCode == "email")
{
castedActivity.Id = Guid.NewGuid();
castedActivity["regardingobjectid"] = new EntityReference("opportunity", OpportunityID);
//service.Create(castedActivity);--->Exception thrown
//service.Update(castedActivity);---->Tried this one too.Exception is thrown stating method not supported on "ActivityPointer"
}
}
Can somebody shed light on this? Am i missing something here? Thank you

You need to query for the exact entity type, because you can't update an activitypointer
private void MoveEmailsFromLeadToOpportunity(IOrganizationService service, Guid LeadID, Guid OpportunityID)
{
ConditionExpression condition = new ConditionExpression();
condition.AttributeName = "regardingobjectid";
condition.Operator = ConditionOperator.Equal;
condition.Values.Add(LeadID.ToString());
var query = new QueryExpression("email");
query.Criteria.AddCondition(condition);
query.ColumnSet = new ColumnSet(true);
var emails = service.RetrieveMultiple(query).Entities;
foreach (var email in emails)
{
email["regardingobjectid"] = new EntityReference("opportunity", OpportunityID);
service.Update(email);
}
}
you can also write a method that will retrieve first all the activities (as you already did) check the ActivityTypeCode after retrieve and update each single record depending on activity type (email, fax, ...)

Try commenting out this line:
castedActivity.Id = Guid.NewGuid();
And then just do your Update:
service.Update(castedActivity)
You are just updating the RegardingObjectId, not creating a new activity, so you shouldn't change the Id and you shouldn't use Create.