Skip to content
  • Products
    • Persistence for Java / JVM
    • Persistence for Android
    • MicroStream Cluster
    • Serializer
  • Community
    • GitHub
    • Discussion
    • Contribute
    • Issues
    • StackOverflow
  • Resources
    • Get Started
    • Docs
    • Examples
    • Videos
    • Blog
    • Release & Update Plan
  • Services
    • Support
    • Training
  • Company
    • About us
    • Contact
  • Subscribe to our Newsletter
  • User icon Sign in

Cookie

We use cookies to make it easier to use and to further improve our service.

If you have given us your consent, you can revoke it at any time in the data protection declaration.

Use all cookies Manage cookies Reject cookies

Cookie

?
?
?
Use all cookiesSave settingsReject cookiesView Privacy Policy

Necessary

Necessary cookies help make a website usable by enabling basic functions such as page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

Name Anbieter Zweck Ablauf Typ
c3kie_removeCookieWindow microstream.one Saves the consent status of the user whether the cookie window should be displayed. 1 Jahr HTML Local Storage
c3kie_googleAnalytics microstream.one Saves the consent status of the user as to whether Google Analytics is allowed to run. 1 Jahr HTML Local Storage
c3kie_googleAnalytics microstream.one Saves the consent status of the user as to whether Google Analytics is allowed to run. 1 Jahr HTML Local Storage
c3kie_tagManager microstream.one Saves the consent status of the user as to whether Google Tag Manager is allowed to run. 1 Jahr HTML Local Storage
c3kie_facebook microstream.one Saves the consent status of the user as to whether Facebook is allowed to run. 1 Jahr HTML Local Storage
c3kie_matomo microstream.one Saves the consent status of the user as to whether Matomo is allowed to run. 1 Jahr HTML Local Storage
c3kie_youtube microstream.one Saves the consent status of the user as to whether YouTube is allowed to run. 1 Jahr HTML Local Storage

Statistics

Statistics cookies help website owners understand how visitors interact with websites by collecting and reporting information anonymously.

Name Anbieter Zweck Ablauf Typ
_dc_gtm_ Google Used by Google Analytics to limit the request rate. 1 Jahr HTTP Cookie
_gid_ Google Registers a unique ID that is used to generate statistical data on how the visitor uses the website. 2 Jahre HTTP Cookie
_gcl_au Google Used to send data to Google Analytics about the device and visitor behavior. Captures the visitor across devices and marketing channels. Session Pixel Tracker
_gat_ Google Used to store a unique user ID. 1 Tag HTTP Cookie
_gat_gtag_UA_ Google Used to store a unique user ID. 1 Tag HTTP Cookie
yt-player-headers-readable YouTube Used to determine the optimal video quality based on the visitor's device and network settings. Persistent HTML Local Storage

Marketing

Marketing cookies are used to track visitors across websites. The intent is to show ads that are relevant and engaging to the individual user, making them more valuable to publishers and third party advertising providers.

Name Anbieter Zweck Ablauf Typ
VISITOR_INFO1_LIVE YouTube Tries to estimate the range of users on pages with built-in YouTube videos. 179 Tage HTTP Cookie
YSC YouTube Registers a unique ID to keep statistics on which videos from YouTube the user has seen. Session HTTP Cookie
yt.innertube::nextId YouTube Registers a unique ID to keep statistics on which videos from YouTube the user has seen. Persistent HTML Local Storage
yt.innertube::requests YouTube Registers a unique ID to keep statistics on which videos from YouTube the user has seen. Persistent HTML Local Storage
ytidb::LAST_RESULT_ENTRY_KEY YouTube Saves the user's video player settings with embedded YouTube video. Persistent HTML Local Storage
yt-remote-cast-available YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-cast-installed YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-connected-devices YouTube Saves the user's video player settings with embedded YouTube video. Persistent HTML Local Storage
yt-remote-device-id YouTube Saves the user's video player settings with embedded YouTube video. Persistent HTML Local Storage
yt-remote-fast-check-period YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-session-app YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-session-name YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage

Not all Data are Equally

Feb172023
Blog

All enterprise applications are using data at their core. There are maybe a few applications that interact with several services and aggregate/ process them, but the vast majority are maintaining data. That can be from the shopping basket on a webshop, production planning data in a factory, or data events for automotive or health applications.

But not all data that your application is using, are the same. Not every request makes use of all your data. So what does this mean for your implementation?

Different types of data

Your application can have a lot of data, gigabytes or even terabytes of data. But all of them are not needed every time, all the time. And due to the different usage frequencies, you might need to apply a different strategy for how you store and access them. This will assure the optimal solution, especially in terms of performance and user experience, especially when you have a lot of data.

When you only have a small dataset, there is no problem to keep them all of them in memory, all the time, with MicroStream.

So let’s discuss the different types more in detail. You can roughly identify 3 types of data.

Core data

The core data or hot data are needed almost every time. For example, in a web shop, these are the products that you sell. They are needed when people search for what they need in our example. So, they should be easily and fast available.

In traditional environments, these data are often cached so that they don’t need to be loaded from the external system, the database, or the noSQL solution, every time.
When you are using MicroStream, these data should not be lazily loaded so that they are available in the heap.

Although this seems similar to using a cache, there are important differences when using MicroStream. Using a cache is a problem to solve the slow access of your data using databases and noSQL solutions. The main reason for this slowness is the latency (remote storage) and conversion to the specific format of the system. With MicroStream, the storage of your data as plain porous within the JVM heap is the default operational modus. Once loaded, data is available together with your program statements and thus accessible extremely fast as no loading nor conversion is needed.

Depending on your situation, this data corresponds with 10 to 30% of your total data set.

Request Data

Besides the core data, requests need some specific data to fulfill the requests. These can be the account data related to the shopping basket handling in our webshop example. The account data of all our shop users are not needed all the time. Only the data specific to the user making the purchase is needed at the time. And most of the time, we don’t need the data at all.

This type of data is ideal to lazily load with MicroStream. Once we have identified what data we need to load into memory, this can be through various _indexing_ mechanisms like a simple `HashMap` or by using Apache Lucene if we want to search for the required data. And once loaded, it can stay there for the time needed when the user finishes his interaction with your application. Lazily loaded data within MicroStream are removed after a certain time when not accessed or can be marked for removal at the end of the user session.

This lazy loading functionality of MicroStream is still more efficient than regular database or noSQL solutions.

This data corresponds with 20 to 50% of your total data set, depending on your situation.

Historical data

The last category of data is historical data which is only required in rare cases. Access to this type can be initiated by the user, like requesting his order history, or by background jobs like monthly calculations on your platform. These data don’t need to be highly available and it is fine that it take ‘a bit of time’ to retrieve them. The Lazy loading feature of MicroStream can still be a way of loading the historical data. But even when you are using MicroStream for the Core and request data, you might decide to use an alternative solution. Storing these historical data in a database or data warehouse solution makes sense. Since this kind of data is used for various types of purposes and not only online access, these environments make sense.

Again depending on the specific case, the amount might vary between 30 to 70% of your total data set.

Conclusion

Not all your data is used for the same purpose or scenario. Some data are essential for every request and others are only needed for batch processing with low-performance requirements. So it is only natural that you make your data available in different ways, corresponding to the accessibility requirements of the data. MicroStream with its two types of data access, in memory available or lazily loaded on demand, is ideal to have a single system that can provide your data using different characteristics.
But the MicroStream focus on easy and data available with high performance does not exclude that you explore other solutions, like storing the historical data. A database or data warehouse is a logical choice for that type of data.

Category: BlogFebruary 17, 2023Leave a comment
Tags: Data Persistence

Author: Rudy De Busscher

Post navigation

PreviousPrevious post:A peek into upcoming version 8.0NextNext post:MicroStream Cloud is now online!

Related Posts

The optimal Lazy List size
March 10, 2023
A peek into upcoming version 8.0
January 26, 2023
MicroStream as Spring Cache provider
January 5, 2023
MicroStream joins the Eclipse Foundation
MicroStream is now an Eclipse Foundation Member
December 21, 2022
Some Data Model considerations for searching and indexing with MicroStream
December 12, 2022
MicroStream with Helidon MP
November 17, 2022

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Post comment

Search
Categories
  • Blog(42)
  • CloudNotification(1)
  • Conferences(13)
  • General(3)
  • News(18)
  • Performance(1)
  • Releases(12)
  • Training(3)
Related Posts
  • May 16 – 17:00 – 21:00 CEST MicroStream Advanced Course Part 2
    March 13, 2023
  • May 9 – 17:00 – 21:00 CEST MicroStream Advanced Course Part 1
    March 13, 2023
  • April 26 – 17:00 – 21:00 CEST MicroStream Fundamentals Course
    March 13, 2023
  • The optimal Lazy List size
    March 10, 2023
  • MicroStream Cloud is now online!
    March 8, 2023
  • Not all Data are Equally
    February 17, 2023
  • A peek into upcoming version 8.0
    January 26, 2023
  • MicroStream as Spring Cache provider
    January 5, 2023
  • MicroStream joins the Eclipse Foundation
    MicroStream is now an Eclipse Foundation Member
    December 21, 2022
  • Some Data Model considerations for searching and indexing with MicroStream
    December 12, 2022
Tags
Android AWS Cache Cloud connector Cluster Communication Conference Data Persistence developers event feature framework free fundamentals GitHub Hackathon Heldion How to InfoQ Integration Java Java 17 JAVAPRO JCON logging LTS Micronaut MicroProfile Microservices MicroStream MicroStream 2 MicroStream 4 MicroStream 6 New Features online Open Liberty Open Source Oracle Performance public Red Hat Summit release Serialization Spring Boot Training
MicroStream

Store Java Object Graphs natively, relieved of heavy-weight DBMS Dependencies. Create ultra-fast In- Memory Database Applications & Microservices with Pure Java. The Pure Java Paradigm Shift in Database Development.

Upcoming Event

April 19, 2023 | 17:00 – 21:00 CEST
Read more

Platforms

  • MicroStream for Java / JVM
  • MicroStream for Android

Community

  • GitHub
  • Discussion
  • Contribute
  • Issues
  • StackOverflow
  • MeetUp

Resources

  • Get Started
  • Docs
  • Examples
  • Videos
  • Blog
  • Release plan

Services

  • Support
  • Training

Company

  • About us
  • Contact

Stay Connected

  • Twitter
  • LinkedIn
  • YouTube
  • GitHub
  • StackOverflow
  • MeetUp

Get the latest MicroStream news:

Subscribe

© 2023 MicroStream Software. All rights reserved.
  • Terms of use
  • Privacy
  • Legal notice

Cookie

We use cookies to make it easier to use and to further improve our service.

If you have given us your consent, you can revoke it at any time in the data protection declaration.

Use all cookies Manage cookies Reject cookies

Cookie

?
?
?
Use all cookiesSave settingsReject cookiesView Privacy Policy

Necessary

Necessary cookies help make a website usable by enabling basic functions such as page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

Name Anbieter Zweck Ablauf Typ
c3kie_removeCookieWindow microstream.one Saves the consent status of the user whether the cookie window should be displayed. 1 Jahr HTML Local Storage
c3kie_googleAnalytics microstream.one Saves the consent status of the user as to whether Google Analytics is allowed to run. 1 Jahr HTML Local Storage
c3kie_googleAnalytics microstream.one Saves the consent status of the user as to whether Google Analytics is allowed to run. 1 Jahr HTML Local Storage
c3kie_tagManager microstream.one Saves the consent status of the user as to whether Google Tag Manager is allowed to run. 1 Jahr HTML Local Storage
c3kie_facebook microstream.one Saves the consent status of the user as to whether Facebook is allowed to run. 1 Jahr HTML Local Storage
c3kie_matomo microstream.one Saves the consent status of the user as to whether Matomo is allowed to run. 1 Jahr HTML Local Storage
c3kie_youtube microstream.one Saves the consent status of the user as to whether YouTube is allowed to run. 1 Jahr HTML Local Storage

Statistics

Statistics cookies help website owners understand how visitors interact with websites by collecting and reporting information anonymously.

Name Anbieter Zweck Ablauf Typ
_dc_gtm_ Google Used by Google Analytics to limit the request rate. 1 Jahr HTTP Cookie
_gid_ Google Registers a unique ID that is used to generate statistical data on how the visitor uses the website. 2 Jahre HTTP Cookie
_gcl_au Google Used to send data to Google Analytics about the device and visitor behavior. Captures the visitor across devices and marketing channels. Session Pixel Tracker
_gat_ Google Used to store a unique user ID. 1 Tag HTTP Cookie
_gat_gtag_UA_ Google Used to store a unique user ID. 1 Tag HTTP Cookie
yt-player-headers-readable YouTube Used to determine the optimal video quality based on the visitor's device and network settings. Persistent HTML Local Storage

Marketing

Marketing cookies are used to track visitors across websites. The intent is to show ads that are relevant and engaging to the individual user, making them more valuable to publishers and third party advertising providers.

Name Anbieter Zweck Ablauf Typ
VISITOR_INFO1_LIVE YouTube Tries to estimate the range of users on pages with built-in YouTube videos. 179 Tage HTTP Cookie
YSC YouTube Registers a unique ID to keep statistics on which videos from YouTube the user has seen. Session HTTP Cookie
yt.innertube::nextId YouTube Registers a unique ID to keep statistics on which videos from YouTube the user has seen. Persistent HTML Local Storage
yt.innertube::requests YouTube Registers a unique ID to keep statistics on which videos from YouTube the user has seen. Persistent HTML Local Storage
ytidb::LAST_RESULT_ENTRY_KEY YouTube Saves the user's video player settings with embedded YouTube video. Persistent HTML Local Storage
yt-remote-cast-available YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-cast-installed YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-connected-devices YouTube Saves the user's video player settings with embedded YouTube video. Persistent HTML Local Storage
yt-remote-device-id YouTube Saves the user's video player settings with embedded YouTube video. Persistent HTML Local Storage
yt-remote-fast-check-period YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-session-app YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage
yt-remote-session-name YouTube Saves the user's video player settings with embedded YouTube video. Session HTML Local Storage