I like to write a process in Worker role, to download (sync) batch of files under a folder(directory) to local mirrored folder(directory)
Is there a timestamp(or a way to get) on the time of last folder(directory) updated?
Since folder(directory) structure unsure, but simply put is download whatever there to local, as soon as it changes. Except recursion and setup a timer to check it repeatedly, whats another smart idea do you have?
(edit) p.s. I found many solutions on sync files from local to Azure storage, but the same principle on local files cannot apply on Azure blob, I am still looking for a way that most easily to download(sync) files to local as soon as they are changed.
Eric, I believe the concept you're trying to implement isn't really that effective for your core requirement, if I understand it correctly.
Consider the following scenario:
Keep your views in the blob storage.
Implement Azure (AppFabric) Cache.
Store any view file to the cache, if it's not yet there on a web request with unlimited(or a very long) expiration time.
Enable local cache on your web role instances with a short expiration time (e.g. 5 minutes)
Create a (single, separated) worker role, outside your web roles, which scans your blobs' ETags for changes in interval. Reset the view's cache key for any blob changed
Get rid of those ugly "workers" inside of your web roles :-)
There're a few things to think about in this scenario:
Your updated views will get to the web role instances within "local cache expiration time + worker scan interval". The lower the values, the more distributed cache requests and blob storage transactions.
The Azure AppFabric Cache is the only Azure service preventing the whole platform to be truly scalable. You have to choose the best cache plan based on the overall size (in MB) of your views, the number of your instances and the number of simultaneous cache requests required per instance.
consider caching of the compiled views inside your instances (not in the AppFabric cache). Reset this local cache based on the dedicated AppFabric cache key/keys. This will raise the performance greatly for you, as rendering the output html will be as easy as injecting the model to the pre-compiled views.
of course, the cache-retrieval code in your web roles must be able to retrieve the view from the primary source (storage), if it is unable to retrieve it from the cache for whatever reason.
My suggestion is to create an abstraction on top of the blob storage, so that no one is directly writing to the blob. Then submit a message to Azure's Queue service when a new file is written. Have the file receiver poll that queue for changes. No need to scan the entire blob store recursively.
As far as the abstraction goes, use an Azure web role or worker role to authenticate and authorize your clients. Have it write to the Blob store(s). You can implement the abstraction using HTTPHandlers or WCF to directly handle the IO requests.
This abstraction will allow you to overcome the blob limitation of 5000 files you mention in the comments above, and will allow you scale out and provide additional features to your customers.
I'd be interested in seeing your code when you have a chance. Perhaps I can give you some more tips or code fixes.
Related
I made an UWP app for Microsoft Store. However, user data automatically saved in the LocalState folder will be deleted every time the app is updated. I want the data to be retained after every updating, so I'm trying to suggest the users to save their data by themselves in the Document folder or somewhere to avoid their data deleted, but I don't want to bother them. Where should I save user data?
The roaming folder will be unable to use in future and I don't want to use Azure because of its fee.
The common approach is to store the data in some remote location, like for example in the cloud. You would typically use a service of some kind to request and save the data.
If you think Azure is to expensive, you'll have to find a cheaper storage solution. The principle is the same regardless of which storage provider you use.
As mentioned in the docs, roaming data is (or at least will be) deprecated. The recommended replacement is Azure App Service.
I have a situation where i need to keep the state of 5 variables of approx. 10,000 users.
I only need to keep the state during the session. If a user close down the window the data shall be cleared of security reasons and GDPR.
The bot Will be on Facebook. There will be no authentication required for the user.
I think it will be to much to manage with the in memory storage.
Is table storage a good option here? Or any better suggestions?
For testing and prototyping purposes, you can use the Bot Builder Framework's in-memory data storage. For production bots, you can implement your own storage adapter or use one of Azure Extensions. The Azure Extensions allow you to store your bot's state data in either Table Storage, CosmosDB, or SQL.
https://learn.microsoft.com/en-us/azure/bot-service/dotnet/bot-builder-dotnet-state?view=azure-bot-service-3.0
As #Kamran said, you can use any number of storage options for the backend state storage.
Regarding the issue around session lifetime; memory storage is volatile because when the service restarts, you lose your state. Which is good for testing. But it won't really map to a users session. You could have a new 'session' but still have saved state in memory storage. You will want to look into the conversation ID, and perhaps have logic around that. That is the closest thing to session lifetime.
We have a multi-tenant system consisting of an Azure Web Role, Worker Role and Desktop/Mobile apps. Each client app allows uploading images that get routed to a tenant-specific Azure Blob Storage account.
The Azure Worker Role polls these files and processes them. We use third-party SDKs for processing that require either a file system path or a stream. Providing a stream directly from blob storage is trivial but the SDK also expects to spit out physical metadata files that our app consumes.
This is a problem since the SDK is a black box and does not provide an alternative. Is there a way to have local storage within worker roles for transient files. This storage is only required for a few seconds per worker role iteration and may be recycled/discarded if the role is recycled or shut down. In addition, the files are rather large (500MB+) so blob latency is not desired.
Searching around revealed some hacky workarounds, the best of which appears to be something that wraps blob storage to let our role access it as a file system.
Is there a way to simply have access to a file system similar to Web Role App_Data folders?
You can use RoleEnvironment.GetLocalResource() from within an Azure Worker Role to get a named handle to local file storage:
RoleEnvironment.GetLocalResource()
This will avoid hardcoding of specific file paths that may change over time, etc.
Good luck!
My company holds a dozen websites and isolated DBs (identical schemas).
every customer has its own website (different app pool) and DB.
every website has its own configuration, several connection strings, but they all have same schema for configuration.
cust1.domain.com
cust2.domain.com
cust3.domain.com
We would like to merge all websites to one (single app pool) and stay with isolated DBs for security and large amount of data reasons.
what is the best practice for designing a DAL and configuration of it?
what are the implications of it, if large amount of tenant will be on the same time? does one application pool can manage this situation or it can be managed somehow?
BTW, we are using asp-membership for users authentication.
Thanks in advance,
Eddie
Use Application_PostAuthenticate event in global.asax to load the correct database and then close the connection in Application_EndRequest
One option is to use the profile in membership and store a piece of information that will allow you to determine which of the actual db's they should be connecting to. Downside is that you will need to store this piece of information for the duration of the users session so either a cookie or session variable is likley to be needed.
The implications of one site vs many depends a lot on your environment and application, do you currently have the multiple sites on a single box or do you have a web farm? do you know the number of concurrent users for each site, the amount of traffic? Performance monitor can help you here to see how busy each site is but you may need more invasive logging to determine metrics such as concurrent users. I found this server fault question around IIS 7 performance which may be of help
You can try 'Shared DataBase With Different Schema' from multi tenant data architecture . In your DAL you can choose specific schema which perticular to current user. Simple and secure in this way
Continue reading http://msdn.microsoft.com/en-us/library/aa479086.aspx
The place where I work has 2 servers and a load balancer. The setup is horrible since I have to manually make sure both servers have the same files. I know there are ways to automate this but it has not been implemented, hopefully soon (I have no control over this). I wrote an application that collects a bunch of information from a user, then creates a folder named after the email of the user in one of the servers. The problem is that I can't control in which server the folder gets created in, so let say a user goes in.. fills his stuff and his folder gets created in server 1, user goes away for a while and goes back to the site but this time the load balancer throws the user into server 2, now the user does something that needs to be saved into his folder but since it didn't created in this server an error occurs. What can I do about this? any suggestions?
Thanks
It sounds like you could solve a few issues by implementing a cloud file service for the file writes such as Amazon S3 http://aws.amazon.com/s3/
Disk size management would no longer be a concern
Files are now written and read from S3 so load balancer concerns are solved
Benefits of a semi-edge network with AWS. (not truly edge but in my experience better than most internally hosted solutions)
Don't store your data in the file system, store it in a database.
If you really can't avoid using the file system, you could look at storing the files in a network share both servers have access to. This would be a terrible hack, however.
It sounds like you may be having a session state issue. It sounds odd the way you describe it, but have a look at this article. It's old, but covers the basics. If it doesn't try googling "asp.net session state web farm"
http://ondotnet.com/pub/a/dotnet/2003/03/24/sessionstate.html
Use NAS or SAN to centralize storage. That same network-accessible storage can store the shared configuration that IIS can be setup to use.
Web Deploy v2 just released from Microsoft, I would encourage the powers that be to investigate that, along with Application Request Routing and the greater Web Farm Framework.
This is a normal infrastructure setup. Below are the two commonly used solutions for the situation you are in.
If you have network attached storage available (e.g. Netapps), you can use this storage to centrally store all of your user files that need to be available across all servers in your web farm.
Redesign your application to store all user specific data in a database.