Using Cloud storage platforms with MicroStream

When the Java instances are converted to their binary representation within the channels, they are written out to the storage targets. Internally, MicroStream has the concept of the Abstract File System (AFS) to be able to support multiple types of storage.

An implementation is available that uses a directory on disk, another one uses SQL connection to read or store the data in databases, but there is also the possibility to use cloud storage.

These cloud storage connectors were part of the enterprise solution but are now made available in the open-source version as of 7.1

In this blog, we have a closer look at how you can use the Azure Storage for storing the binary representation of the data.

Cloud Storage

MicroStream abstracts the access to the storage away bind the Abstract File System (AFS). So a channel can just use this abstraction to read and write data from *somewhere* without knowing where that actually might be.

The default implementation of this AFS system is using the Non-Blocking IO (NIO) of Java to access a directory. But other implementations are available, including a BlobStoreFileSystem that can connect to cloud storage.

When we start the StorageManager, we can provide it with a properly configured blob storage so that our application will use the Azure Storage facilities. It is a cloud storage solution where you can store data objects in a highly available, scalable, durable, and secure way.

The other storage connectors like AWS S3, Redis, and Kafka for example, work in a very similar way as the Azure one.

Configure for Azure

The configuration of Azure Storage as storage for MicroStream is very straightforward.

From the project dependency views, you need 2 additional dependencies

– The Abstract File System implementation for Azure Storage: <artifactId>microstream-afs-azure-storage</artifactId>
– The Azure SDK to access the Azure Cloud endpoints to store and retrieve blobs from the platform

<dependency>
   <groupId>com.azure</groupId>
   <artifactId>azure-storage-blob</artifactId>
   <version>12.19.0</version>
</dependency>

Within the code, we need to get hold of a BlobServiceClient from the Azure SDK which is a client for your Azure storage account.

With this client, you can create, list, and delete containers that hold your files (you can see them as disks) and from this instance, you can create or access BlockBlobClient instances that represent your data, the files in the disk analogy.

This snippet shows how to configure the Storage Manager to make use of the Azure Storage facility.

BlobServiceClient client = createClient();
BlobStoreFileSystem fileSystem = BlobStoreFileSystem.New(AzureStorageConnector.Caching(client));

StorageManager storageManager = EmbeddedStorage.start(root, fileSystem.ensureDirectoryPath("microstream-storage"));

The microstream-storage is the name of an existing blob container within your storage account. You can create it through the Azure CLI, or using the Azure Storage Explorer application.

The creation of the BlobServiceClient and how the credentials can be loaded is very well described in the Azure SDK documentation.

You don’t need to use the programmatic way to configure MicroStream, you can also use configuration files or use framework integration, like the Spring Boot one.

In the case of a properties file, don’t forget to add the microstream-storage-embedded-configuration dependency to your project. A properties file for Azure storage could look like this

storage-filesystem.azure.storage.credentials.type=shared-key
storage-filesystem.azure.storage.credentials.account-name=rubusms
storage-filesystem.azure.storage.credentials.account-key=<shared-key>
storage-filesystem.azure.storage.endpoint=https://rubusms.blob.core.windows.net
storage-directory=microstream-storage

And this file can be used using the standard way of using external configuration for MicroStream:

StorageManager storageManager = EmbeddedStorageConfiguration.load(
"storage.properties"
)
.createEmbeddedStorageFoundation()
.createEmbeddedStorageManager(root)
.start();

But also the Spring Boot integration, and also the other integrations can make use of the Azure Storage connector.

The usage is identical to the scenario where you define the configuration externally in a properties file. When the configuration entries, prefixed with ‘one.microstream.’ are placed in the application.properties file or any other source that is processed as part of the Spring Boot application configuration.

Code examples can be found in the azure directory of this GitHub repository.

Conclusion

The Cloud Storage connectors are now part of the open-source version since version 7.1 which was released recently ( see blog )
By adding the Abstract File System implementation for the platform you want to use, the specific java client code, and the appropriate configuration your application can use the Cloud Storage in an unaltered version.

And from that point on, you make use of the highly available, scalable, durable, platform in a secure way.