New tools for Azure Data at Build 2021

If you’re building cloud-native applications you need a reliable, efficient data platform. Reliable microservices need a way to store state, whether in NoSQL key/value systems or massively scalable SQL databases. It’s no different in Azure, and Microsoft has been building out its cloud data offering during the past few years to give developers a mix of its own proprietary and open source data platforms.

At its Build 2021 developer event, Microsoft is unveiling some major changes to that data platform, aiming to make it more attractive to developers and offer features that will help build a new generation of applications.

Azure SQL adds an immutable ledger

One of the more fascinating items, the launch of a ledger feature for Azure SQL, makes more sense of the announcement that Microsoft is closing Azure’s blockchain-as-a-service platform. Much of enterprise blockchain development has focused on its role as an immutable source of transaction data, where systems and processes need a trustworthy record of what has been done and by whom.

This is where modern ledgers come in, as a way of creating that blockchain-like verification model. Here, however, the ledger is just another table in a familiar database that can provide that point of trust without requiring a complete redesign and redevelopment of your application. There’s no point in replacing an existing database with a complex, relatively slow blockchain if all you need to do is add a new cryptographically secure ledger table to an existing database to manage that data.

There’s no need to learn new skills or implement new tools, as this is all part of the familiar SQL Server running on Azure. Existing applications can be updated to add ledgers without needing new code. It can all be managed inside the database with database developers and administrators using existing database management tools.

Using Azure SQL ledgers in your applications

Having a ledger like this in an Azure SQL database allows you to ensure that data in a database hasn’t been altered; this tool can help you comply with regulations (especially if audits are required) and reduce the risk of malicious altering of data to cover up fraud or other financial crimes. The ledger table contains a cryptographic hash of the transactions which itself can be stored outside your database in Azure’s existing tamper-proof storage services for additional protection.

Azure SQL’s ledger tables support both updateable ledgers and append-only ledgers. Updateable ledgers show how data in a database has been modified; for example, they can monitor changes to a bank account or a product inventory. The updateable ledger will show the delete and insert operations used to update a managed table along with the associated transaction IDs. An append-only ledger blocks update and delete operations, supporting insert-only patterns. This reduces the complexity of the ledger structure but does limit the use cases to blockchain-like behavior.

Using an append-only ledger with SQL Azure should let you build the type of blockchain application that the Azure blockchain service offered, working with trusted partners to manage immutable data structures that captured Electronic Data Interchange (EDI) or similar business transactions. However, by removing the overhead associated with distributed data structures you should get much better performance while still getting the security benefits of blockchain (and also reducing the energy requirements that come with proof-of-work blockchains).

Azure SQL’s ledger is a similar concept to the Linux Foundation Hyperledger project, but with a focus on providing better transaction integrity and security in a pure SQL environment. With a SQL Azure ledger, it should be possible to step back through every transaction, helping pinpoint errors and providing a level of trust not currently available.

Working with Azure SQL ledgers at scale

Some issues remain with the Azure SQL ledger model as it’s focused on working with monolithic stores. Although you can use it with a database that scales in a single region using SQL Hyperscale, if you’re elastically sharding your database geographically, each shard will need its own ledger to support local transactions. It’s not currently designed to work with distributed transactions, and if you’re thinking about building a system that uses that approach, you’re more likely to want to use something like Cosmos DB and build your own ledger system as part of your application.

Having multiple separate ledgers shouldn’t be a problem for distributed applications, as data associated with a specific shard will be in a specific ledger, and as new shards are deployed, new ledgers can be added. It will be necessary to keep track of your ledgers, although that should be possible through both the Azure SQL tools and the Azure portal.

Securing open source databases on Azure

A trusted platform also needs security, and Azure has been rolling out more features for its active cloud workload protection tool Azure Defender. Build sees it adding support for more databases than the current Azure SQL supports. The latest release adds a public preview for Azure’s PostgreSQL and MySQL services. Running outside your applications, Azure Defender provides a feed of security data to the Azure Security Center dashboard where you can monitor your applications for early signs of attacks.

Azure Defender builds on Microsoft’s Security Graph platform not only to identify known attacks but also to highlight anomalies that may indicate in-progress attacks. Adding additional security to databases reduces the risk of breaches and data leaks, showing you where databases are vulnerable and advising you on appropriate mitigations.

Updating Cosmos DB for secure data

Microsoft isn’t ignoring its foundational services in this round of Azure updates. Its Cosmos distributed database is gaining support for always-encrypted data, allowing you to encrypt data before it’s stored in JSON documents, locking down confidential data. Combined with general availability for Cosmos DB’s role-based access control tool, you are now able to ensure that the right people have access to your data, and even if intruders or malware do get unauthorized access, that data is unusable without the right encryption keys.

Outside of its new security, Cosmos DB is getting support for an integrated in-memory cache that can reduce calls to the underlying operational Cosmos DB instance. This will behave much like Redis Cache and should help keep costs down, as it’s billed at a fixed hourly rate.

In addition to secure access to data, Azure’s Synapse analytics tool is getting a no-code link to the Dynamics 365 and the Power Platform’s Dataverse. Intended to host operational data for line-of-business systems (and to link to third-party Open Data-compliant platforms like SAP and Adobe’s Marketing Cloud), Dataverse integration with Synapse will open up more opportunities for business analysts and developers. Cloud and business data sources can be brought together and used with Azure’s developer platform, including providing large, labeled data sets for training new machine learning systems and customizing prebuilt Azure’s Cognitive Services models.

The combination of data platform features that Microsoft is adding to Azure at Build 2021 should help developers deliver trusted applications at cloud scale. There’s a lot to look forward to here, especially in conjunction with other new Azure features, such as improved API management, additional machine learning services, new lower-cost and free tiers for many services, as well as more application services that should make the whole platform more attractive for developing and running enterprise-critical applications.