Store State of a Background process with MicroStream

With the MicroStream framework, you can treat an object graph within the JVM heap as your database. This means you can perform queries by executing any java method against those objects. The Stream API is very well-suited to be used here.
The Java instances are serialised in a binary format to an external storage so that the state of the object graph can be restored when your process starts up the next time.

We have developed a new serialisation engine from scratch, as we want to overcome a few shortcomings of the standard Java one.

Serialise any object

One of the requirements we had, was that any kind of Object could be part of the Object graph that consists of your database. And thus we should be able to serialise any object, not only those that implement Serializable.

Therefore we developed a new engine that can just store any object. There is no need to have a certain (marker) interface implemented, has some annotations, or a schema that generates code. Any Java instance can be stored and loaded again and we don’t even use the constructor or any other method for this, like the readObject() method. This way, we make sure that no vulnerability exploit can be made.

Any object?

In theory, yes, any object can be handled by MicroStream. We have excluded a few classes as serialising them doesn’t make much sense or are related to some OS resources so a serialisation would not work. An example is the Thread class.

When you try to serialise a Thread instance, you will receive a PersistenceExceptionTypeNotPersistable exception.

Other classes that will throw this Exception are the InputStream and OutputStream ones, Socket, and Iterator and Enumeration instances. Of course, the collections themselves can be used, but references to an Iterator that might be stored in a field, are not.

Also, lambdas cannot be serialised due to the special format they are stored in by the compiler. So when this field f is part of the object graph that you want to store, you will receive an Exception.

Function<Integer, String> f = x -> String.format("Function result for %s", x);

Of course, the above lambda doesn’t make sense to store, since it doesn’t hold any data. If you want to store a ‘lambda‘, define it as an actual class, an anonymous inner class is also supported. This doesn’t change how you can make use of it in your code.

  Function<Integer, String> f = new Function<>() {

      @Override
      public String apply(Integer x) {
          return String.format("Anonymous inner class result for %s", x);
      }
  };

Background process with state

Since we have indicated that a Thread cannot be stored, you cannot store for example a Thread that runs in the background of your application and execute every period some tasks and keep the last run time to filter the next time, for example.

Also, a TimerTask cannot be stored since it has a state field that gets set once the task is scheduled by a Timer instance so that the same task cannot be started again later on. And since this state field is also serialised by MicroStream, a deserialised instance cannot be used anymore.
But a Runnable instance can be serialised, so with the help of ScheduledExecutorService, you can achieve your requirement.

For example, the Runnable implementation can look like this. Here we just keep the last run date as an example.

public class ProcessRunnable implements Runnable {

    transient Persister persister;

    private Date lastRun;

    public ProcessRunnable(Persister storageManager) {
        this.persister = storageManager;
    }

    @Override
    public void run() {
        System.out.println("Last run was at " + lastRun);
        lastRun = new Date();
        System.out.println("Now running at " + lastRun);
        persister.store(this);

    }

}

So every time the Runnable is executed, it stores its state in the external storage.

And a simple Java SE program to demonstrate how it can be used looks like

   public static void main(String[] args) {
        BackgroundRoot root = new BackgroundRoot();

        StorageManager storageManager = EmbeddedStorage.start(root, Paths.get("data/background"));

        ScheduledExecutorService ses = Executors.newScheduledThreadPool(1);

        if (root.getRunnableTask() == null) {
            root.setRunnableTask(new ProcessRunnable(storageManager));
            storageManager.store(root);
        }

        ses.scheduleAtFixedRate(root.getRunnableTask(), 0, 10, TimeUnit.SECONDS);


    }

    public static class BackgroundRoot {
        private ProcessRunnable runnableTask;

        public ProcessRunnable getRunnableTask() {
            return runnableTask;
        }

        public void setRunnableTask(ProcessRunnable runnableTask) {
            this.runnableTask = runnableTask;
        }
    }

When there is no ProcessRunnable yet, first it creates one. Otherwise, there is already an instance defined with the data from the last run and we just start it through ScheduledExecutorService instance.

Background process with state

The MicroStream framework has no limitation on which object can be stored. There is no interface required like serialisable, annotations, or any other special construct. However, some classes are excluded like Thread, InputStream, and OutputStream instances.

Since also a TimerTask cannot be reused due to the restriction within the JDK implementation (related to the state field).

But you can implement your requirements for a background task that keeps state using a Runnable and the ScheduledExecutorService. Be aware that you cannot define the Runnable as a lambda when you want to make use of MicroStream but since you also need fields for your state, that is not a problem here.