VMS Tech Demo 21 - data export with Spring Batch

Content:

Introduction
Spring Batch
Setup
From RDBMS
1. To XML
2. To JSON
3. To CSV
From Index-Sequential files
1. To XML
2. To JSON
3. To CSV
Conclusion

Introduction:

This is a repeat of VMS Tech Demo 18 - getting data out of VMS for the XML/JSON/CSV part using Spring Batch framework instead of custom code - using XML config instead of Groovy/Jython code.

Some skills in Java and Spring are required. But if you want to use Spring Batch, then you hopefully already have those.

The only VMS specific part is the use of VMS index-sequential files.

Spring Batch:

Despite the names then Spring Batch and VMS batch has really nothing in common.

So:

VMS batch get a job running on some system at some point in time - it does not care about what the job actually does
Spring Batch is a framework for writing data processing jobs - it does not care about how the job get started

For more info about Spring Batch see article here.

Setup:

We will demo:

RDBMS -> XML
RDBMS -> JSON
RDBMS -> CSV
Index-Sequential files -> XML
Index-Sequential files -> JSON
Index-Sequential files -> CSV

Spring configuration will be done using old style XML config files. That seems most appropriate for this kind of batch processing to me.

But let me be clear: that XML config file is de facto "programming in XML". The XML config file is the logic in the batch job.

All examples will use the same main program that basically read the XML config file and execute the job defined in that XML.

Data classes:

Spring Batch can use mapped data classes.

Data classes plain (no annotations):

Orders.java:

import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;

public class Orders {
    private int id;
    private String customer;
    private String status;
    private List<OrderLines> orderLines = new ArrayList<OrderLines>();
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    public List<OrderLines> getOrderLines() {
        return orderLines;
    }
    public void setOrderLines(List<OrderLines> lines) {
        this.orderLines = lines;
    }
}

OrderLines.java:

import java.math.BigDecimal;

public class OrderLines {
    private int id;
    private String item;
    private int qty;
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

OrderLinesX.java:

import java.math.BigDecimal;

public class OrderLinesX {
    private int id;
    private int orderId;
    private String item;
    private int qty;
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public int getOrderId() {
        return orderId;
    }
    public void setOrderId(int orderId) {
        this.orderId = orderId;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

Data classes with JPA annotations:

OrdersJPA.java:

import java.util.Set;
import java.util.TreeSet;

import javax.persistence.CascadeType;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.FetchType;
import javax.persistence.Id;
import javax.persistence.OneToMany;
import javax.persistence.OrderBy;
import javax.persistence.Table;

@Entity
@Table(name="orders")
public class OrdersJPA {
    private int id;
    private String customer;
    private String status;
    private Set<OrderLinesJPA> orderLines = new TreeSet<OrderLinesJPA>();
    @Id
    @Column(name="id")
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    @Column(name="customer")
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    @Column(name="status")
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    @OneToMany(mappedBy = "orderId", fetch = FetchType.EAGER, cascade = CascadeType.ALL)
    @OrderBy("id")
    public Set<OrderLinesJPA> getOrderLines() {
        return orderLines;
    }
    public void setOrderLines(Set<OrderLinesJPA> lines) {
        this.orderLines = lines;
    }
    // convenience method
    public OrdersJPA add(OrderLinesJPA ol) {
        orderLines.add(ol);
        return this;
    }
}

OrderLinesJPA.java:

import java.math.BigDecimal;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Table;

@Entity
@Table(name="orderlines")
public class OrderLinesJPA implements Comparable<OrderLinesJPA> {
    private int id;
    private int orderId;
    private String item;
    private int qty;
    private BigDecimal price;
    @Id
    @Column(name="id")
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    @Column(name="orderid")
    public int getOrderId() {
        return orderId;
    }
    public void setOrderId(int orderId) {
        this.orderId = orderId;
    }
    @Column(name="item")
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    @Column(name="qty")
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    @Column(name="price")
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
    @Override
    public int compareTo(OrderLinesJPA o) {
        return id - o.id;
    }
}

Data classes with ISAM annotations:

OrdersISAM.java:

import java.util.ArrayList;
import java.util.List;

import dk.vajhoej.isam.KeyField;
import dk.vajhoej.record.ArrayField;
import dk.vajhoej.record.FieldType;
import dk.vajhoej.record.Struct;
import dk.vajhoej.record.StructField;
import dk.vajhoej.record.TransientField;

@Struct
public class OrdersISAM {
    @KeyField(n=0)
    @StructField(n=0, type=FieldType.INT4)
    private int id;
    @StructField(n=1, type=FieldType.FIXSTR, length=32, pad=true, padchar=' ')
    private String customer;
    @StructField(n=2, type=FieldType.FIXSTR, length=16, pad=true, padchar=' ')
    private String status;
    @TransientField
    private List<OrderLinesISAM> orderLinesT = new ArrayList<OrderLinesISAM>();
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    public List<OrderLinesISAM> getOrderLinesT() {
        return orderLinesT;
    }
    public void setOrderLinesT(List<OrderLinesISAM> orderLinesT) {
        this.orderLinesT = orderLinesT;
    }
}

OrderLinesISAM.java:

import java.math.BigDecimal;

import dk.vajhoej.isam.KeyField;
import dk.vajhoej.record.FieldType;
import dk.vajhoej.record.Struct;
import dk.vajhoej.record.StructField;

@Struct
public class OrderLinesISAM {
    @KeyField(n=0)
    @StructField(n=0, type=FieldType.INT4)
    private int id;
    @KeyField(n=1)
    @StructField(n=1, type=FieldType.INT4)
    private int orderId;
    @StructField(n=2, type=FieldType.FIXSTR, length=32, pad=true, padchar=' ')
    private String item;
    @StructField(n=3, type=FieldType.INT4)
    private int qty;
    @StructField(n=4, type=FieldType.PACKEDBCD, length=6, decimals=2)
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public int getOrderId() {
        return orderId;
    }
    public void setOrderId(int orderId) {
        this.orderId = orderId;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

Data classes with JAXB annotations:

OrdersJAXB.java:

import java.util.List;
import java.util.ArrayList;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElements;
import javax.xml.bind.annotation.XmlType;
import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement(name="order")
@XmlType(propOrder={"id", "customer", "status", "orderLines"})
public class OrdersJAXB {
    private int id;
    private String customer;
    private String status;
    private OrderLinesCollectionJAXB orderLines = new OrderLinesCollectionJAXB();
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    @XmlElement(name="order-lines")
    public OrderLinesCollectionJAXB getOrderLines() {
        return orderLines;
    }
    public void setOrderLines(OrderLinesCollectionJAXB orderLines) {
        this.orderLines = orderLines;
    }
}

OrderLinesJAXB.java:

import java.math.BigDecimal;

import javax.xml.bind.annotation.XmlType;

@XmlType(propOrder={"id", "item", "qty", "price"})
public class OrderLinesJAXB {
    private int id;
    private String item;
    private int qty;
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

OrderLinesCollectionJAXB.java:

import java.util.ArrayList;
import java.util.List;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElements;

public class OrderLinesCollectionJAXB {
    private List<OrderLinesJAXB> list = new ArrayList<OrderLinesJAXB>();
    @XmlElements(@XmlElement(name="order-line"))
    public List<OrderLinesJAXB> getList() {
        return list;
    }
    public void setList(List<OrderLinesJAXB> list) {
        this.list = list;
    }
    // convenience method
    public void add(OrderLinesJAXB ol) {
        list.add(ol);
    }
}

Data conversion classes:

Instead of direct source to target conversion data classes will be normalized to the plain data classes.

FromJPA.groovy:

import org.springframework.batch.item.*

class FromJPA implements ItemProcessor<OrdersJPA,Orders> {
    static mo = new AutoMap(OrdersJPA.class, Orders.class, true)
    static mol = new AutoMap(OrderLinesJPA.class, OrderLines.class, true)
    @Override
    Orders process(OrdersJPA ojpa) {
        Orders o = new Orders()
        mo.convert(ojpa, o)
        for(oljpa in ojpa.orderLines) {
            OrderLines ol = new OrderLines()
            mol.convert(oljpa, ol)
            o.orderLines.add(ol)
        }
        return o
    }
}

FromISAM.groovy:

import org.springframework.batch.item.*

class FromISAM implements ItemProcessor<OrdersISAM,Orders> {
    static mo = new AutoMap(OrdersISAM.class, Orders.class, true)
    static mol = new AutoMap(OrderLinesISAM.class, OrderLines.class, true)
    @Override
    Orders process(OrdersISAM oisam) {
        Orders o = new Orders()
        mo.convert(oisam, o)
        for(olisam in oisam.orderLinesT) {
            OrderLines ol = new OrderLines()
            mol.convert(olisam, ol)
            o.orderLines.add(ol)
        }
        return o
    }
}

ToJAXB.groovy:

import org.springframework.batch.item.*

class ToJAXB implements ItemProcessor<Orders,OrdersJAXB> {
    static mo = new AutoMap(Orders.class, OrdersJAXB.class, true)
    static mol = new AutoMap(OrderLines.class, OrderLinesJAXB.class, true)
    @Override
    OrdersJAXB process(Orders o) {
        OrdersJAXB ojaxb = new OrdersJAXB()
        mo.convert(o, ojaxb)
        for(ol in o.orderLines) {
            OrderLinesJAXB oljaxb = new OrderLinesJAXB()
            mol.convert(ol, oljaxb)
            ojaxb.orderLines.add(oljaxb)
        }
        return ojaxb
    }
}

Main:

The main class has no specific logic - it just load Spring XML config file and start processing based on that.

main.groovy:

import org.springframework.batch.core.*
import org.springframework.context.*
import org.springframework.context.support.*

ctx = new FileSystemXmlApplicationContext(args[0])
jobLauncher = ctx.getBean("jobLauncher")
job = ctx.getBean(args[1])
exe = jobLauncher.run(job, new JobParameters())
while(exe.isRunning()) {
    print("*")
    Thread.sleep(10)
}
printf("Done, status = %s %s\n", exe.getExitStatus().getExitCode(), exe.getExitStatus().getExitDescription())
for(ex in exe.getAllFailureExceptions()) {
    ex.printStackTrace()
}

$ groovy_cp = "springbatch/spring-context-5_3_31.jar:springbatch/spring-batch-core-4_3_10.jar:springbatch/spring-batch-infrastructure-4_3_10.jar"
$ groovyc """main.groovy"""
$ java -cp .:springbatch/*:/disk0/net/groovy/groovy-4.0.12/lib/* "main" <xml-config-file> <job-name>

springbatch/* contains:

antlr-2_7_7.jar;1
aspectjrt-1_9_9_1.jar;1
aspectjweaver-1_9_9_1.jar;1
byte-buddy-1_12_7.jar;1
classmate-1_5_1.jar;1
commons-logging-1_1.jar;1
hibernate-commons-annotations-5_1_2_Final.jar;1
hibernate-core-5_6_5_Final.jar;1
isam-vms.jar;1
isam.jar;1
jackson-annotations-2_9_9.jar;1
jackson-core-2_9_9.jar;1
jackson-databind-2_9_9.jar;1
jandex-2_4_2_Final.jar;1
javax_batch-api-1_0.jar;1
javax_persistence-api-2_2.jar;1
jboss-logging-3_4_3_Final.jar;1
jboss-transaction-api_1_2_spec-1_1_1_Final.jar;1
micrometer-core-1_9_17.jar;1
mysql-connector-j-8_0_33.jar;1
record.jar;1
spring-aop-5_3_31.jar;1
spring-batch-core-4_3_10.jar;1
spring-batch-infrastructure-4_3_10.jar;1
spring-beans-5_3_31.jar;1
spring-context-5_3_31.jar;1
spring-core-5_3_31.jar;1
spring-expression-5_3_31.jar;1
spring-jdbc-5_3_31.jar;1
spring-orm-5_3_31.jar;1
spring-oxm-5_3_31.jar;1
spring-retry-1_3_4.jar;1
spring-tx-5_3_31.jar;1
xstream-1_4_20.jar;1

From RDBMS:

JPA will be used for RDBMS data access.

persistence.xml:

<persistence xmlns="http://java.sun.com/xml/ns/persistence"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
             version="2.0">
   <persistence-unit name="orders">
      <provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
      <class>OrdersJPA</class>
      <class>OrderLinesJPA</class>
      <exclude-unlisted-classes/>
      <properties>
          <property name="hibernate.show_sql" value="false"/>
          <property name="hibernate.connection.driver_class" value="com.mysql.cj.jdbc.Driver"/>
          <property name="hibernate.connection.url" value="jdbc:mysql://arnepc5/Test"/>
          <property name="hibernate.connection.username" value="arne"/>
          <property name="hibernate.connection.password" value="hemmeligt"/>
          <property name="hibernate.connection.pool_size" value="5"/>
      </properties>
   </persistence-unit>
</persistence>

To XML:

Flow:

db2xml.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="dbTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitName" value="orders"/>
    </bean>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>OrdersJAXB</value>
                <value>OrderLinesCollectionJAXB</value>
                <value>OrderLinesJAXB</value>
            </list>
        </property>
    </bean>
    <!-- step beans -->
    <bean id="ordersJPAReader" class="org.springframework.batch.item.database.JpaPagingItemReader">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="queryString" value="SELECT DISTINCT o FROM OrdersJPA AS o JOIN FETCH o.orderLines ol"/>
    </bean>
    <bean id="copyJPA2JAXBProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromJPA"/>
                <bean class="ToJAXB"/>
            </list>
        </property>
    </bean>
    <bean id="ordersXmlWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter">
        <property name="resource" value="orders1.xml"/>
        <property name="marshaller" ref="jaxbMarshaller"/>                                                         
        <property name="rootTagName" value="orders"/>
        <property name="overwriteOutput" value="true"/>
    </bean>
    <!-- job -->
    <batch:job id="db2xml">
        <batch:step id="export">
            <batch:tasklet transaction-manager="dbTransactionManager">
                <batch:chunk reader="ordersJPAReader" writer="ordersXmlWriter" processor="copyJPA2JAXBProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

To JSON:

Flow:

db2json.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="dbTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitName" value="orders"/>
    </bean>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <!-- step beans -->
    <bean id="ordersJPAReader" class="org.springframework.batch.item.database.JpaPagingItemReader">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="queryString" value="SELECT DISTINCT o FROM OrdersJPA AS o JOIN FETCH o.orderLines ol"/>
    </bean>
    <bean id="copyJPA2PlainProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromJPA"/>
            </list>
        </property>
    </bean>
    <bean id="ordersJsonWriter" class="org.springframework.batch.item.json.JsonFileItemWriter">
        <constructor-arg value="orders1.json"/>
        <constructor-arg>
            <bean class="org.springframework.batch.item.json.JacksonJsonObjectMarshaller"/>
        </constructor-arg>
    </bean>
    <!-- job -->
    <batch:job id="db2json">
        <batch:step id="export">
            <batch:tasklet transaction-manager="dbTransactionManager">
                <batch:chunk reader="ordersJPAReader" writer="ordersJsonWriter" processor="copyJPA2PlainProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

To CSV:

Flow:

There is a little complication for CSV, because a single record from the database must end up as one line in orders1.csv and multiple lines in orderlines1.csv.

To handle this we use a setup like:

                                     |-->ordersCsvWriter (FlatFileItemWriter)-->orders1.csv
bothCsvWriter (CompositeItemWriter)--|
                                     |-->expandingWriter (custom doing loop)-->orderLinesCsvWriter (FlatFileItemWriter)-->orderlines1.csv

Expanding.groovy:

import org.springframework.batch.item.*

class Expanding implements ItemWriter<Orders> {
    static mol = new AutoMap(OrderLines.class, OrderLinesX.class, true)
    ItemWriter<OrderLinesX> delegate
    @Override
    void write(List<Orders> co) {
        for(Orders o : co) {
            for(OrderLines ol in o.orderLines) {
                OrderLinesX olx = new OrderLinesX()
                mol.convert(ol, olx)
                olx.orderId = o.id
                delegate.write(Arrays.asList(olx))
            }
        }
    }
}

db2csv.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="dbTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitName" value="orders"/>
    </bean>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <!-- step beans -->
    <bean id="ordersJPAReader" class="org.springframework.batch.item.database.JpaPagingItemReader">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="queryString" value="SELECT DISTINCT o FROM OrdersJPA AS o JOIN FETCH o.orderLines ol"/>
    </bean>
    <bean id="copyJPA2PlainProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromJPA"/>
            </list>
        </property>
    </bean>
    <bean id="ordersCsvWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
        <property name="resource" value="orders1.csv"/>
        <property name="lineAggregator">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                <property name="delimiter" value=","/>
                <property name="fieldExtractor">
                    <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                        <property name="names" value="id,customer,status"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
    <bean id="orderLinesCsvWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
        <property name="resource" value="orderlines1.csv"/>
        <property name="lineAggregator">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                <property name="delimiter" value=","/>
                <property name="fieldExtractor">
                    <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                        <property name="names" value="id,orderId,item,qty,price"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
    <bean id="expandingWriter" class="Expanding">
        <property name="delegate" ref="orderLinesCsvWriter"/>
    </bean>
    <bean id="bothCsvWriter" class="org.springframework.batch.item.support.CompositeItemWriter">
        <property name="delegates">
            <list>
                <ref bean="ordersCsvWriter"/>
                <ref bean="expandingWriter"/>
            </list>
        </property>
    </bean>
    <!-- job -->
    <batch:job id="db2csv">
        <batch:step id="export">
            <batch:tasklet transaction-manager="dbTransactionManager">
                <batch:chunk reader="ordersJPAReader" writer="bothCsvWriter" processor="copyJPA2PlainProcessor" commit-interval="1">
                    <batch:streams>
                        <batch:stream ref="orderLinesCsvWriter"/>
                    </batch:streams>
                </batch:chunk>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

From Index-Sequential files:

Spring Batch does obviously not come with builtin support for VMS index-sequential files, but I have an ISAM library for such access and I have written Spring Batch ItemReader's based on that.

SingleIsamItemReader.java:

import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.NonTransientResourceException;
import org.springframework.batch.item.ParseException;
import org.springframework.batch.item.UnexpectedInputException;
import org.springframework.batch.item.support.AbstractItemStreamItemReader;

import dk.vajhoej.isam.IsamException;
import dk.vajhoej.isam.IsamResult;
import dk.vajhoej.isam.IsamSource;
import dk.vajhoej.record.RecordException;

public class SingleIsamItemReader<T> extends AbstractItemStreamItemReader<T> {
    private IsamSource source;
    private Class<T> type;
    private IsamResult<T> result;
    public IsamSource getSource() {
        return source;
    }
    public void setSource(IsamSource source) {
        this.source = source;
    }
    public Class<T> getType() {
        return type;
    }
    public void setType(Class<T> type) {
        this.type = type;
    }
    @Override
    public void open(ExecutionContext ec) {
        try {
            result = source.readStart(type);
        } catch (IsamException e) {
            throw new RuntimeException(e);
        } catch (RecordException e) {
            throw new RuntimeException(e);
        }
    }
    @Override
    public T read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
        if(result.read()) {
            return result.current();
        } else {
            return null;
        }
    }
    @Override
    public void close() {
        try {
            result.close();
            source.close();
        } catch (IsamException e) {
            throw new RuntimeException(e);
        }
    }
}

ReaderSecondarySource.java:

import java.lang.reflect.Method;

import dk.vajhoej.isam.IsamSource;

public class ReaderSecondarySource<T> {
    private String primaryKey; // name of key in primary source used to lookup in secondary source
    private String primaryList; // name of list in primary source containing records from secondary source 
    private Class<T> type; // type of records in secondary source
    private int index; // index of key in secondary source  used to lookup in secondary source
    private IsamSource source; // secondary source
    private Method primaryKeyGetter;
    private Method primaryListGetter;
    public String getPrimaryKey() {
        return primaryKey;
    }
    public void setPrimaryKey(String primaryKey) {
        this.primaryKey = primaryKey;
    }
    public String getPrimaryList() {
        return primaryList;
    }
    public void setPrimaryList(String primaryList) {
        this.primaryList = primaryList;
    }
    public Class<T> getType() {
        return type;
    }
    public void setType(Class<T> type) {
        this.type = type;
    }
    public int getIndex() {
        return index;
    }
    public void setIndex(int index) {
        this.index = index;
    }
    public IsamSource getSource() {
        return source;
    }
    public void setSource(IsamSource source) {
        this.source = source;
    }
    public Method getPrimaryKeyGetter() {
        return primaryKeyGetter;
    }
    public void setPrimaryKeyGetter(Method primaryKeyGetter) {
        this.primaryKeyGetter = primaryKeyGetter;
    }
    public Method getPrimaryListGetter() {
        return primaryListGetter;
    }
    public void setPrimaryListGetter(Method primaryListGetter) {
        this.primaryListGetter = primaryListGetter;
    }
}

MultiIsamItemReader.java:

import java.lang.reflect.Method;
import java.util.List;

import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.NonTransientResourceException;
import org.springframework.batch.item.ParseException;
import org.springframework.batch.item.UnexpectedInputException;
import org.springframework.batch.item.support.AbstractItemStreamItemReader;

import dk.vajhoej.isam.IsamException;
import dk.vajhoej.isam.IsamResult;
import dk.vajhoej.isam.IsamSource;
import dk.vajhoej.isam.Key;
import dk.vajhoej.isam.KeyInfoArrayCache;
import dk.vajhoej.record.RecordException;

public class MultiIsamItemReader<T> extends AbstractItemStreamItemReader<T> {
    private IsamSource primarySource;
    private Class<T> type;
    private List<ReaderSecondarySource> secondarySources;
    private IsamResult<T> result;
    public IsamSource getPrimarySource() {
        return primarySource;
    }
    public void setPrimarySource(IsamSource primarySource) {
        this.primarySource = primarySource;
    }
    public Class<T> getType() {
        return type;
    }
    public void setType(Class<T> type) {
        this.type = type;
    }
    public List<ReaderSecondarySource> getSecondarySources() {
        return secondarySources;
    }
    public void setSecondarySources(List<ReaderSecondarySource> secondarySources) {
        this.secondarySources = secondarySources;
    }
    private Method getGetter(String fldnam) {
        try {
            return type.getMethod("get" + fldnam.substring(0, 1).toUpperCase() + fldnam.substring(1));
        } catch (NoSuchMethodException e) {
            throw new RuntimeException(e);
        } catch (SecurityException e) {
            throw new RuntimeException(e);
        }
    }
    @Override
    public void open(ExecutionContext ec) {
        try {
            result = primarySource.readStart(type);
            for(ReaderSecondarySource<?> src2 : secondarySources) {
                src2.setPrimaryKeyGetter(getGetter(src2.getPrimaryKey()));
                src2.setPrimaryListGetter(getGetter(src2.getPrimaryList()));
            }
        } catch (IsamException e) {
            throw new RuntimeException(e);
        } catch (RecordException e) {
            throw new RuntimeException(e);
        }
    }
    @Override
    public T read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
        if(result.read()) {
            T o = result.current();
            for(ReaderSecondarySource src2 : secondarySources) {
                int keyix = src2.getIndex();
                Comparable keyval = (Comparable)src2.getPrimaryKeyGetter().invoke(o);
                List lst2 = (List)src2.getPrimaryListGetter().invoke(o);
                Class t = src2.getType();
                IsamSource is = src2.getSource();
                IsamResult ir = is.readGE(t, new Key(keyix, keyval));
                while(ir.read()) {
                    Object o2 = ir.current();
                    Comparable keyval2 = (Comparable)KeyInfoArrayCache.analyze(t)[src2.getIndex()].getField().get(o2);
                    if(keyval.compareTo(keyval2) == 0) {
                        lst2.add(o2);
                    } else {
                        break;
                    }
                }
            }
            return o;
        } else {
            return null;
        }
    }
    @Override
    public void close() {
        try {
            result.close();
            primarySource.close();
            for(ReaderSecondarySource src2 : secondarySources) {
                src2.getSource().close();
            }
        } catch (IsamException e) {
            throw new RuntimeException(e);
        }
    }
}

Note that these readers are not batching/chunking. For two reasons:

I wanted to keep them simple
I don't think there are any performance benefits of batching/chunking - it will end up with one SYS$GET call per record no matter what

The ISAM library can be downloaded here.

To XML:

Flow:

isq2xml.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>OrdersJAXB</value>
                <value>OrderLinesCollectionJAXB</value>
                <value>OrderLinesJAXB</value>
            </list>
        </property>
    </bean>
    <!-- step beans -->
    <bean id="orderLinesISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orderlines.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="true"/>
    </bean>
    <bean id="orderLines" class="ReaderSecondarySource">
        <property name="primaryKey" value="id"/>
        <property name="primaryList" value="orderLinesT"/>
        <property name="type" value="OrderLinesISAM"/>
        <property name="index" value="1"/>
        <property name="source" ref="orderLinesISAMSource"/>
    </bean>
    <bean id="ordersISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orders.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="true"/>
    </bean>
    <bean id="ordersISAMReader" class="MultiIsamItemReader">
        <property name="primarySource" ref="ordersISAMSource" />
        <property name="type" value="OrdersISAM"/>
        <property name="secondarySources">
            <list>
                <ref bean="orderLines"/>
            </list>
        </property>
    </bean>
    <bean id="copyISAM2JAXBProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromISAM"/>
                <bean class="ToJAXB"/>
            </list>
        </property>
    </bean>
    <bean id="ordersXmlWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter">
        <property name="resource" value="orders2.xml"/>
        <property name="marshaller" ref="jaxbMarshaller"/>                                                         
        <property name="rootTagName" value="orders"/>
        <property name="overwriteOutput" value="true"/>
    </bean>
    <!-- job -->
    <batch:job id="isq2xml">
        <batch:step id="export">
            <batch:tasklet transaction-manager="repoTransactionManager">
                <batch:chunk reader="ordersISAMReader" writer="ordersXmlWriter" processor="copyISAM2JAXBProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

To JSON:

Flow:

isq2json.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <!-- step beans -->
    <bean id="orderLinesISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orderlines.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="true"/>
    </bean>
    <bean id="orderLines" class="ReaderSecondarySource">
        <property name="primaryKey" value="id"/>
        <property name="primaryList" value="orderLinesT"/>
        <property name="type" value="OrderLinesISAM"/>
        <property name="index" value="1"/>
        <property name="source" ref="orderLinesISAMSource"/>
    </bean>
    <bean id="ordersISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orders.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="true"/>
    </bean>
    <bean id="ordersISAMReader" class="MultiIsamItemReader">
        <property name="primarySource" ref="ordersISAMSource" />
        <property name="type" value="OrdersISAM"/>
        <property name="secondarySources">
            <list>
                <ref bean="orderLines"/>
            </list>
        </property>
    </bean>
    <bean id="copyISAM2PlainProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromISAM"/>
            </list>
        </property>
    </bean>
    <bean id="ordersJsonWriter" class="org.springframework.batch.item.json.JsonFileItemWriter">
        <constructor-arg value="orders2.json"/>
        <constructor-arg>
            <bean class="org.springframework.batch.item.json.JacksonJsonObjectMarshaller"/>
        </constructor-arg>
    </bean>
    <!-- job -->
    <batch:job id="isq2json">
        <batch:step id="export">
            <batch:tasklet transaction-manager="repoTransactionManager">
                <batch:chunk reader="ordersISAMReader" writer="ordersJsonWriter" processor="copyISAM2PlainProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

To CSV:

Flow:

Same issue for CSV as for database export:

                                     |-->ordersCsvWriter (FlatFileItemWriter)-->orders2.csv
bothCsvWriter (CompositeItemWriter)--|
                                     |-->expandingWriter (custom doing loop)-->orderLinesCsvWriter (FlatFileItemWriter)-->orderlines2.csv

isq2csv.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <!-- step beans -->
    <bean id="orderLinesISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orderlines.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="true"/>
    </bean>
    <bean id="orderLines" class="ReaderSecondarySource">
        <property name="primaryKey" value="id"/>
        <property name="primaryList" value="orderLinesT"/>
        <property name="type" value="OrderLinesISAM"/>
        <property name="index" value="1"/>
        <property name="source" ref="orderLinesISAMSource"/>
    </bean>
    <bean id="ordersISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orders.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="true"/>
    </bean>
    <bean id="ordersISAMReader" class="MultiIsamItemReader">
        <property name="primarySource" ref="ordersISAMSource" />
        <property name="type" value="OrdersISAM"/>
        <property name="secondarySources">
            <list>
                <ref bean="orderLines"/>
            </list>
        </property>
    </bean>
    <bean id="copyISAM2PlainProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromISAM"/>
            </list>
        </property>
    </bean>
    <bean id="ordersCsvWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
        <property name="resource" value="orders2.csv"/>
        <property name="lineAggregator">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                <property name="delimiter" value=","/>
                <property name="fieldExtractor">
                    <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                        <property name="names" value="id,customer,status"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
    <bean id="orderLinesCsvWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
        <property name="resource" value="orderlines2.csv"/>
        <property name="lineAggregator">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                <property name="delimiter" value=","/>
                <property name="fieldExtractor">
                    <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                        <property name="names" value="id,orderId,item,qty,price"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
    <bean id="expandingWriter" class="Expanding">
        <property name="delegate" ref="orderLinesCsvWriter"/>
    </bean>
    <bean id="bothCsvWriter" class="org.springframework.batch.item.support.CompositeItemWriter">
        <property name="delegates">
            <list>
                <ref bean="ordersCsvWriter"/>
                <ref bean="expandingWriter"/>
            </list>
        </property>
    </bean>
    <!-- job -->
    <batch:job id="isq2csv">
        <batch:step id="export">
            <batch:tasklet transaction-manager="repoTransactionManager">
                <batch:chunk reader="ordersISAMReader" writer="bothCsvWriter" processor="copyISAM2PlainProcessor" commit-interval="1">
                    <batch:streams>
                        <batch:stream ref="orderLinesCsvWriter"/>
                    </batch:streams>
                </batch:chunk>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

Conclusion:

Does it make sense to use Spring Batch instead of custom code?

First we note that it does not save code - in fact this example replaced 30-60 lines of Groovy/Jython with 40-100 lines of XML.

But it does add some robustness and flexibility to the solution.

And on top of that it may have some benefits for some enterprises:

Spring Batch may be the corporate standard and there are resources available with Spring Batch skills
it may be easier to get a XML "config change" approved than a Groovy/Jython/whatever "code change" (*)

*) Technically that is total BS, but sometimes rules for such things does not make any sense.

So it may be worth considering.

Article history:

Version	Date	Description
1.0	March 30th 2025	Initial version

Comments:

Please send comments to Arne Vajhøj

VMS Tech Demo 21 - data export with Spring Batch

Content:

Introduction:

Spring Batch:

Setup:

Data classes:

Data conversion classes:

Main:

From RDBMS:

To XML:

To JSON:

To CSV:

From Index-Sequential files:

To XML:

To JSON:

To CSV:

Conclusion:

Article history:

Other articles:

Comments: