VMS Tech Demo 4

BDB JE 7.5 should run on VMS Itanium with Java 8. BDB JE 5.0 runs on VMS Alpha with Java 5.

There is not actually any VMS specific Java code in this article. The VMS aspect is that it tries to relate BDB JE to VMS index-sequential files. And there are VMS scripts to build and run the code.

Besides Java examples there will also be a few Jython (JVM Python) examples to show that BDB JE works fine for other JVM languages as well.

Hypothetical context:

Let us assume that we have an older VMS application written in native language (Pascal/Cobol/Basic) using index-sequential files and it has been decided to migrate to a JVM language (Java/Kotlin/Groovy/Scala) to get to a newer language but wanting to stay on VMS.

Relational database is the obvious choice. I have already written about that choice numerous times:

But maybe there are reasons for not to go that route and sticking to the Key Value Store / ISAM model.

Index-sequential files can be used from JVM languages. I have also written about that:

So if one decide to go for another NoSQL Key Value Store database for Java, then BDB JE is a pretty obvious choice.

Be sure to check whether the license for the BDB JE version you use is compatible with your usage. Old versions are under AGPL. Newer versions are under Apache license. AGPL may be a problem for you.

BLOB in BLOB out:

BDB JE expose a low level API where both keys and values are just byte arrays. One can store a chunk of bytes (a BLOB) and later retrieve the same chunk of bytes (BLOB).

This is somewhat equivalent of the RMS API where one just pass address of data and length of data.

Note that this approach is easier in a native language than in Java as you in a native language can just take the address of the original data structure and pass while you in Java have to serialize an object and pass the result of that.

Working with defined records:

For the VMS people that is a bit like how some VMS languages has support for mapping records to VMS index-sequential files.

import com.sleepycat.persist.model.Entity;
import com.sleepycat.persist.model.PrimaryKey;

@Entity
public class DataO {
    @PrimaryKey
    private String key;
    private int iv;
    private double xv;
    private String sv;
    public DataO() {
        this("", 0, 0.0, "");
    }
    public DataO(String key, int iv, double xv, String sv) {
        this.key = key;
        this.iv = iv;
        this.xv = xv;
        this.sv = sv;
    }
    public String getKey() {
        return key;
    }
    public void setKey(String key) {
        this.key = key;
    }
    public int getIv() {
        return iv;
    }
    public void setIv(int iv) {
        this.iv = iv;
    }
    public double getXv() {
        return xv;
    }
    public void setXv(double xv) {
        this.xv = xv;
    }
    public String getSv() {
        return sv;
    }
    public void setSv(String sv) {
        this.sv = sv;
    }
    @Override
    public String toString() {
        return String.format("{iv: %d, xv: %f, sv: %s}", iv, xv, sv);
    }
}

import java.io.File;
import java.io.IOException;
import java.util.Iterator;

import com.sleepycat.je.Environment;
import com.sleepycat.je.EnvironmentConfig;
import com.sleepycat.persist.EntityCursor;
import com.sleepycat.persist.EntityStore;
import com.sleepycat.persist.PrimaryIndex;
import com.sleepycat.persist.StoreConfig;

public class TestO {
    private static void dump(PrimaryIndex<String, DataO> pk, String key) throws ClassNotFoundException, IOException {
        System.out.printf("Key=%s\n", key);
        if(pk.contains(key)) {
            DataO o = pk.get(key);
            System.out.println(o);
        } else {
            System.out.println("Not found");
        }
    }
    private static final int NREC = 1000;
    public static void main(String[] args) throws IOException, ClassNotFoundException {
        // open database
        EnvironmentConfig envcfg = new EnvironmentConfig();
        envcfg.setAllowCreateVoid(true);
        Environment env = new Environment(new File("data/"), envcfg);
        StoreConfig stcfg = new StoreConfig();
        stcfg.setAllowCreate(true);
        EntityStore est = new EntityStore(env, "testdb", stcfg);
        PrimaryIndex<String, DataO> pk = est.getPrimaryIndex(String.class, DataO.class);
        // put data
        for(int i = 0; i < NREC; i++) {
            String key = "Key#" + (1000 + i + 1);
            DataO o = new DataO();
            o.setKey(key);
            o.setIv(i + 1);
            o.setXv(i + 1.0);
            o.setSv(String.format("This is value %d", i + 1));
            pk.put(o);
        }
        //
        String key;
        key = "Key#" + 1077;
        // get
        dump(pk, key);
        // delete
        pk.delete(key);
        // get non existing
        dump(pk, key);
        //
        key = "Key#" + 1088;
        // update and get
        dump(pk, key);
        DataO o = pk.get(key);
        o.setIv(o.getIv() + 1);
        o.setXv(o.getXv() + 0.1);
        o.setSv(o.getSv() + " updated");
        pk.put(o);
        dump(pk, key);
        // list all
        EntityCursor<DataO> c = pk.entities();
        Iterator<DataO> it = c.iterator();
        int n = 0;
        while(it.hasNext()) {
            DataO ito = it.next();
            if(!ito.getKey().startsWith("Key#")) {
                System.out.println("Unexpected key: " + ito.getKey());
            }
            if(ito.getIv() < 1 || NREC < ito.getIv()) {
                System.out.println("Unexpected value :" + ito);
            }
            n++;
        }
        c.close();
        System.out.println(n);
        // list keys where "Key#1075" <= key < "Key#1085"
        EntityCursor<DataO> c2 = pk.entities("Key#1075", true, "Key#1085", false);
        Iterator<DataO> it2 = c2.iterator();
        int n2 = 0;
        while(it2.hasNext()) {
            @SuppressWarnings("unused")
            DataO ito2 = it2.next();
            n2++;
        }
        c2.close();
        System.out.println(n2);
        // close database
        est.close();
        env.close();
    }
}

Build and run:

$ javac -classpath /javalib/je-5_0_73.jar TestO.java DataO.java
$ define/nolog java$filename_controls 8
$ define/nolog decc$efs_charset true
$ define/nolog java$filename_match_list "*.jdb=shr=get,put"
$ java -classpath .:/javalib/je-5_0_73.jar "TestO"

import com.sleepycat.persist.model.Entity;
import com.sleepycat.persist.model.PrimaryKey;

@Entity
public class DataO {
    @PrimaryKey
    private String key;
    private int iv;
    private double xv;
    private String sv;
    public DataO() {
        this("", 0, 0.0, "");
    }
    public DataO(String key, int iv, double xv, String sv) {
        this.key = key;
        this.iv = iv;
        this.xv = xv;
        this.sv = sv;
    }
    public String getKey() {
        return key;
    }
    public void setKey(String key) {
        this.key = key;
    }
    public int getIv() {
        return iv;
    }
    public void setIv(int iv) {
        this.iv = iv;
    }
    public double getXv() {
        return xv;
    }
    public void setXv(double xv) {
        this.xv = xv;
    }
    public String getSv() {
        return sv;
    }
    public void setSv(String sv) {
        this.sv = sv;
    }
    @Override
    public String toString() {
        return String.format("{iv: %d, xv: %f, sv: %s}", iv, xv, sv);
    }
}

from java.lang import String
from java.io import File

from com.sleepycat.je import Environment
from com.sleepycat.je import EnvironmentConfig
from com.sleepycat.persist import EntityStore
from com.sleepycat.persist import PrimaryIndex
from com.sleepycat.persist import StoreConfig

import DataO

def dump(pk, key):
    print('Key=%s' % (key))
    if pk.contains(key):
        o = pk.get(key)
        print(o)
    else:
        print('Not found')

NREC = 1000

# open database
envcfg = EnvironmentConfig()
envcfg.setAllowCreateVoid(True)
env = Environment(File("data/"), envcfg)
stcfg = StoreConfig()
stcfg.setAllowCreate(True)
est = EntityStore(env, "testdb", stcfg)
pk = est.getPrimaryIndex(String, DataO);
# put data
for i in range(NREC):
    o = DataO('Key#' + str(1000 + i + 1), i + 1, i + 1.0, 'This is value %d' % (i + 1))
    pk.put(o)
#
key = "Key#" + str(1077)
# get
dump(pk, key)
# delete
pk.delete(key)
# get non existing
dump(pk, key)
#
key = "Key#" + str(1088)
# update and get
dump(pk, key)
o = pk.get(key)
o.iv = o.iv + 1
o.xv = o.xv + 0.1
o.sv = o.sv + ' updated'
pk.put(o)
dump(pk, key)
# list all
c = pk.entities()
n = 0
for o in c.iterator():
    if not o.key.startswith('Key#'):
        print('Unexpected key: ' + o.id)
    if o.iv < 1 or NREC < o.iv:
        print('Unexpected value: ' + d)
    n = n + 1
c.close()
print(n)
# list keys where "Key#1075" <= key < "Key#1085"
c2 = pk.entities('Key#1075', True, 'Key#1085', False)
n2 = 0
for o in c2.iterator():
    n2 = n2 + 1
c2.close()
print(n2)
# close database
est.close()
env.close()

Run:

$ javac -classpath /javalib/je-5_0_73.jar DataO.java
$ define/nolog java$filename_controls 8
$ define/nolog decc$efs_charset true
$ define/nolog java$filename_match_list "*.jdb=shr=get,put"
$ define/nolog jython_libs "/javalib/je-5_0_73.jar"
$ jython testo.py

Multi key:

An important capability of VMS index-sequential files is the support for multiple keys.

import com.sleepycat.persist.model.Entity;
import com.sleepycat.persist.model.PrimaryKey;
import com.sleepycat.persist.model.Relationship;
import com.sleepycat.persist.model.SecondaryKey;

@Entity
public class BaseData {
    @PrimaryKey
    private int id;
    @SecondaryKey(relate=Relationship.MANY_TO_ONE)
    private String name;
    @SecondaryKey(relate=Relationship.MANY_TO_ONE)
    private String address;
    private int value;
    public BaseData() {
        this(0, "", "", 0);
    }
    public BaseData(int id, String name, String address, int value) {
        super();
        this.id = id;
        this.name = name;
        this.address = address;
        this.value = value;
    }
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getName() {
        return name;
    }
    public void setName(String name) {
        this.name = name;
    }
    public String getAddress() {
        return address;
    }
    public void setAddress(String address) {
        this.address = address;
    }
    public int getValue() {
        return value;
    }
    public void setValue(int value) {
        this.value = value;
    }
}

import java.io.File;

import com.sleepycat.je.Environment;
import com.sleepycat.je.EnvironmentConfig;
import com.sleepycat.persist.EntityStore;
import com.sleepycat.persist.PrimaryIndex;
import com.sleepycat.persist.SecondaryIndex;
import com.sleepycat.persist.StoreConfig;

public class MultiKey {
    public static void main(String[] args) {
        // open database
        EnvironmentConfig envcfg = new EnvironmentConfig();
        envcfg.setAllowCreateVoid(true);
        envcfg.setTransactional(true);
        Environment env = new Environment(new File("data/"), envcfg);
        StoreConfig stcfg = new StoreConfig();
        stcfg.setAllowCreate(true);
        stcfg.setTransactional(true);
        EntityStore est = new EntityStore(env, "mulkeydb", stcfg);
        PrimaryIndex<Integer, BaseData> idix = est.getPrimaryIndex(Integer.class, BaseData.class);
        SecondaryIndex<String, Integer, BaseData> namix = est.getSecondaryIndex(idix, String.class, "name");
        SecondaryIndex<String, Integer, BaseData> addrix = est.getSecondaryIndex(idix, String.class, "address");
        // put 3 records
        idix.put(new BaseData(1, "A", "A A A", 123));
        idix.put(new BaseData(2, "B", "B B B", 456));
        idix.put(new BaseData(3, "C", "C C C", 789));
        // lookup by id
        System.out.println(idix.get(2).getValue());
        System.out.println(idix.get(3).getValue());
        // lookup by name
        System.out.println(namix.get("B").getValue());
        System.out.println(namix.get("C").getValue());
        // lookup by address
        System.out.println(addrix.get("B B B").getValue());
        System.out.println(addrix.get("C C C").getValue());
        // close database
        est.close();
        env.close();
    }
}

Build and run:

$ javac -classpath /javalib/je-5_0_73.jar MultiKey.java BaseData.java
$ define/nolog java$filename_controls 8
$ define/nolog decc$efs_charset true
$ define/nolog java$filename_match_list "*.jdb=shr=get,put"
$ java -classpath .:/javalib/je-5_0_73.jar "MultiKey"

import com.sleepycat.persist.model.Entity;
import com.sleepycat.persist.model.PrimaryKey;
import com.sleepycat.persist.model.Relationship;
import com.sleepycat.persist.model.SecondaryKey;

@Entity
public class BaseData {
    @PrimaryKey
    private int id;
    @SecondaryKey(relate=Relationship.MANY_TO_ONE)
    private String name;
    @SecondaryKey(relate=Relationship.MANY_TO_ONE)
    private String address;
    private int value;
    public BaseData() {
        this(0, "", "", 0);
    }
    public BaseData(int id, String name, String address, int value) {
        super();
        this.id = id;
        this.name = name;
        this.address = address;
        this.value = value;
    }
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getName() {
        return name;
    }
    public void setName(String name) {
        this.name = name;
    }
    public String getAddress() {
        return address;
    }
    public void setAddress(String address) {
        this.address = address;
    }
    public int getValue() {
        return value;
    }
    public void setValue(int value) {
        this.value = value;
    }
}

from java.lang import Integer
from java.lang import String
from java.io import File

from com.sleepycat.je import Environment
from com.sleepycat.je import EnvironmentConfig
from com.sleepycat.persist import EntityStore
from com.sleepycat.persist import PrimaryIndex
from com.sleepycat.persist import SecondaryIndex
from com.sleepycat.persist import StoreConfig

import BaseData

# open database
envcfg = EnvironmentConfig()
envcfg.setAllowCreateVoid(True)
env = Environment(File("data/"), envcfg)
stcfg = StoreConfig()
stcfg.setAllowCreate(True)
est = EntityStore(env, "mulkeydb", stcfg)
idix = est.getPrimaryIndex(Integer, BaseData)
namix = est.getSecondaryIndex(idix, String, "name")
addrix = est.getSecondaryIndex(idix, String, "address")
# put 3 records
idix.put(BaseData(1, "A", "A A A", 123))
idix.put(BaseData(2, "B", "B B B", 456))
idix.put(BaseData(3, "C", "C C C", 789))
# lookup by id
print(idix.get(2).value)
print(idix.get(3).value)
# lookup by name
print(namix.get("B").value)
print(namix.get("C").value)
# lookup by address
print(addrix.get("B B B").value)
print(addrix.get("C C C").value)
# close database
est.close()
env.close()

Run:

$ javac -classpath /javalib/je-5_0_73.jar BaseData.java
$ define/nolog java$filename_controls 8
$ define/nolog decc$efs_charset true
$ define/nolog java$filename_match_list "*.jdb=shr=get,put"
$ define/nolog jython_libs "/javalib/je-5_0_73.jar"
$ jython multikey.py

relate=Relationship.MANY_TO_ONE means that duplicates are allowed for the secondary key.

relate=Relationship.ONE_TO_ONE would enforce unique values for the secondary key.

Big records:

Transactions:

For more info about traditional transactions and transaction isolation level see here.

But to really illustrate the power we need to see the example with some failures between the two updates to show that we end up in a consistent state.

Conclusion:

BDB JE is a very capable NOSQL Key Value Store database. And it works fine on VMS.

But I doubt that it will be used much on VMS. JVM languages has never been widely used on VMS.

VMS Tech Demo 4 - BDB JE

Content:

Introduction: