VMS Tech Demo 22 - data import with Spring Batch

Content:

Introduction
Spring Batch
Setup
To RDBMS
To Index-Sequential files
Conclusion

Introduction:

This is a repeat of VMS Tech Demo 18 - getting data out of VMS for the XML/JSON/CSV part using Spring Batch framework instead of custom code - using XML config instead of Groovy/Jython code.

Some skills in Java and Spring are required. But if you want to use Spring Batch, then you hopefully already have those.

The only VMS specific part is the use of VMS index-sequential files.

Spring Batch:

Despite the names then Spring Batch and VMS batch has really nothing in common.

So:

VMS batch get a job running on some system at some point in time - it does not care about what the job actually does
Spring Batch is a framework for writing data processing jobs - it does not care about how the job get started

For more info about Spring Batch see article here.

Setup:

We will demo:

XML -> RDBMS
JSON -> RDBMS
CSV -> RDBMS
XML -> Index-Sequential files
JSON -> Index-Sequential files
CSV -> Index-Sequential files

Spring configuration will be done using old style XML config files. That seems most appropriate for this kind of batch processing to me.

But let me be clear: that XML config file is de facto "programming in XML". The XML config file is the logic in the batch job.

All examples will use the same main program that basically read the XML config file and execute the job defined in that XML.

Data classes:

Spring Batch can use mapped data classes.

Data classes plain (no annotations):

Orders.java:

import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;

public class Orders {
    private int id;
    private String customer;
    private String status;
    private List<OrderLines> orderLines = new ArrayList<OrderLines>();
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    public List<OrderLines> getOrderLines() {
        return orderLines;
    }
    public void setOrderLines(List<OrderLines> lines) {
        this.orderLines = lines;
    }
}

OrderLines.java:

import java.math.BigDecimal;

public class OrderLines {
    private int id;
    private String item;
    private int qty;
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

OrderLinesX.java:

import java.math.BigDecimal;

public class OrderLinesX {
    private int id;
    private int orderId;
    private String item;
    private int qty;
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public int getOrderId() {
        return orderId;
    }
    public void setOrderId(int orderId) {
        this.orderId = orderId;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

Data classes with JPA annotations:

OrdersJPA.java:

import java.util.Set;
import java.util.TreeSet;

import javax.persistence.CascadeType;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.FetchType;
import javax.persistence.Id;
import javax.persistence.OneToMany;
import javax.persistence.OrderBy;
import javax.persistence.Table;

@Entity
@Table(name="orders")
public class OrdersJPA {
    private int id;
    private String customer;
    private String status;
    private Set<OrderLinesJPA> orderLines = new TreeSet<OrderLinesJPA>();
    @Id
    @Column(name="id")
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    @Column(name="customer")
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    @Column(name="status")
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    @OneToMany(mappedBy = "orderId", fetch = FetchType.EAGER, cascade = CascadeType.ALL)
    @OrderBy("id")
    public Set<OrderLinesJPA> getOrderLines() {
        return orderLines;
    }
    public void setOrderLines(Set<OrderLinesJPA> lines) {
        this.orderLines = lines;
    }
    // convenience method
    public OrdersJPA add(OrderLinesJPA ol) {
        orderLines.add(ol);
        return this;
    }
}

OrderLinesJPA.java:

import java.math.BigDecimal;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Table;

@Entity
@Table(name="orderlines")
public class OrderLinesJPA implements Comparable<OrderLinesJPA> {
    private int id;
    private int orderId;
    private String item;
    private int qty;
    private BigDecimal price;
    @Id
    @Column(name="id")
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    @Column(name="orderid")
    public int getOrderId() {
        return orderId;
    }
    public void setOrderId(int orderId) {
        this.orderId = orderId;
    }
    @Column(name="item")
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    @Column(name="qty")
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    @Column(name="price")
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
    @Override
    public int compareTo(OrderLinesJPA o) {
        return id - o.id;
    }
}

Data classes with ISAM annotations:

OrdersISAM.java:

import java.util.ArrayList;
import java.util.List;

import dk.vajhoej.isam.KeyField;
import dk.vajhoej.record.ArrayField;
import dk.vajhoej.record.FieldType;
import dk.vajhoej.record.Struct;
import dk.vajhoej.record.StructField;
import dk.vajhoej.record.TransientField;

@Struct
public class OrdersISAM {
    @KeyField(n=0)
    @StructField(n=0, type=FieldType.INT4)
    private int id;
    @StructField(n=1, type=FieldType.FIXSTR, length=32, pad=true, padchar=' ')
    private String customer;
    @StructField(n=2, type=FieldType.FIXSTR, length=16, pad=true, padchar=' ')
    private String status;
    @TransientField
    private List<OrderLinesISAM> orderLinesT = new ArrayList<OrderLinesISAM>();
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    public List<OrderLinesISAM> getOrderLinesT() {
        return orderLinesT;
    }
    public void setOrderLinesT(List<OrderLinesISAM> orderLinesT) {
        this.orderLinesT = orderLinesT;
    }
}

OrderLinesISAM.java:

import java.math.BigDecimal;

import dk.vajhoej.isam.KeyField;
import dk.vajhoej.record.FieldType;
import dk.vajhoej.record.Struct;
import dk.vajhoej.record.StructField;

@Struct
public class OrderLinesISAM {
    @KeyField(n=0)
    @StructField(n=0, type=FieldType.INT4)
    private int id;
    @KeyField(n=1)
    @StructField(n=1, type=FieldType.INT4)
    private int orderId;
    @StructField(n=2, type=FieldType.FIXSTR, length=32, pad=true, padchar=' ')
    private String item;
    @StructField(n=3, type=FieldType.INT4)
    private int qty;
    @StructField(n=4, type=FieldType.PACKEDBCD, length=6, decimals=2)
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public int getOrderId() {
        return orderId;
    }
    public void setOrderId(int orderId) {
        this.orderId = orderId;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

Data classes with JAXB annotations:

OrdersJAXB.java:

import java.util.List;
import java.util.ArrayList;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElements;
import javax.xml.bind.annotation.XmlType;
import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement(name="order")
@XmlType(propOrder={"id", "customer", "status", "orderLines"})
public class OrdersJAXB {
    private int id;
    private String customer;
    private String status;
    private OrderLinesCollectionJAXB orderLines = new OrderLinesCollectionJAXB();
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getCustomer() {
        return customer;
    }
    public void setCustomer(String customer) {
        this.customer = customer;
    }
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    @XmlElement(name="order-lines")
    public OrderLinesCollectionJAXB getOrderLines() {
        return orderLines;
    }
    public void setOrderLines(OrderLinesCollectionJAXB orderLines) {
        this.orderLines = orderLines;
    }
}

OrderLinesJAXB.java:

import java.math.BigDecimal;

import javax.xml.bind.annotation.XmlType;

@XmlType(propOrder={"id", "item", "qty", "price"})
public class OrderLinesJAXB {
    private int id;
    private String item;
    private int qty;
    private BigDecimal price;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getItem() {
        return item;
    }
    public void setItem(String item) {
        this.item = item;
    }
    public int getQty() {
        return qty;
    }
    public void setQty(int qty) {
        this.qty = qty;
    }
    public BigDecimal getPrice() {
        return price;
    }
    public void setPrice(BigDecimal price) {
        this.price = price;
    }
}

OrderLinesCollectionJAXB.java:

import java.util.ArrayList;
import java.util.List;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElements;

public class OrderLinesCollectionJAXB {
    private List<OrderLinesJAXB> list = new ArrayList<OrderLinesJAXB>();
    @XmlElements(@XmlElement(name="order-line"))
    public List<OrderLinesJAXB> getList() {
        return list;
    }
    public void setList(List<OrderLinesJAXB> list) {
        this.list = list;
    }
    // convenience method
    public void add(OrderLinesJAXB ol) {
        list.add(ol);
    }
}

Data conversion classes:

Instead of direct source to target conversion data classes will be normalized to the plain data classes.

FromJAXB.groovy:

import org.springframework.batch.item.*

class FromJAXB implements ItemProcessor<OrdersJAXB,Orders> {
    static mo = new AutoMap(OrdersJAXB.class, Orders.class, true)
    static mol = new AutoMap(OrderLinesJAXB.class, OrderLines.class, true)
    @Override
    Orders process(OrdersJAXB ojaxb) {
        Orders o = new Orders()
        mo.convert(ojaxb, o)
        for(oljaxb in ojaxb.orderLines.list) {
            OrderLines ol = new OrderLines()
            mol.convert(oljaxb, ol)
            o.orderLines.add(ol)
        }
        return o
    }
}

ToJPA.groovy:

import org.springframework.batch.item.*

class ToJPA implements ItemProcessor<Orders,OrdersJPA> {
    static mo = new AutoMap(Orders.class, OrdersJPA.class, true)
    static mol = new AutoMap(OrderLines.class, OrderLinesJPA.class, true)
    @Override
    OrdersJPA process(Orders o) {
        OrdersJPA ojpa = new OrdersJPA()
        mo.convert(o, ojpa)
        for(ol in o.orderLines) {
            OrderLinesJPA oljpa = new OrderLinesJPA()
            mol.convert(ol, oljpa)
            oljpa.orderId = ojpa.id
            ojpa.orderLines.add(oljpa)
        }
        return ojpa
    }
}

ToISAM.groovy:

import org.springframework.batch.item.*

class ToISAM implements ItemProcessor<Orders,OrdersISAM> {
    static mo = new AutoMap(Orders.class, OrdersISAM.class, true)
    static mol = new AutoMap(OrderLines.class, OrderLinesISAM.class, true)
    @Override
    OrdersISAM process(Orders o) {
        OrdersISAM oisam = new OrdersISAM()
        mo.convert(o, oisam)
        for(ol in o.orderLines) {
            OrderLinesISAM olisam = new OrderLinesISAM()
            mol.convert(ol, olisam)
            olisam.orderId = oisam.id
            oisam.orderLinesT.add(olisam)
        }
        return oisam
    }
}

Main:

The main class has no specific logic - it just load Spring XML config file and start processing based on that.

main.groovy:

import org.springframework.batch.core.*
import org.springframework.context.*
import org.springframework.context.support.*

ctx = new FileSystemXmlApplicationContext(args[0])
jobLauncher = ctx.getBean("jobLauncher")
job = ctx.getBean(args[1])
exe = jobLauncher.run(job, new JobParameters())
while(exe.isRunning()) {
    print("*")
    Thread.sleep(10)
}
printf("Done, status = %s %s\n", exe.getExitStatus().getExitCode(), exe.getExitStatus().getExitDescription())
for(ex in exe.getAllFailureExceptions()) {
    ex.printStackTrace()
}

$ groovy_cp = "springbatch/spring-context-5_3_31.jar:springbatch/spring-batch-core-4_3_10.jar:springbatch/spring-batch-infrastructure-4_3_10.jar"
$ groovyc """main.groovy"""
$ java -cp .:springbatch/*:/disk0/net/groovy/groovy-4.0.12/lib/* "main" <xml-config-file> <job-name>

springbatch/* contains:

antlr-2_7_7.jar;1
aspectjrt-1_9_9_1.jar;1
aspectjweaver-1_9_9_1.jar;1
byte-buddy-1_12_7.jar;1
classmate-1_5_1.jar;1
commons-logging-1_1.jar;1
hibernate-commons-annotations-5_1_2_Final.jar;1
hibernate-core-5_6_5_Final.jar;1
isam-vms.jar;1
isam.jar;1
jackson-annotations-2_9_9.jar;1
jackson-core-2_9_9.jar;1
jackson-databind-2_9_9.jar;1
jandex-2_4_2_Final.jar;1
javax_batch-api-1_0.jar;1
javax_persistence-api-2_2.jar;1
jboss-logging-3_4_3_Final.jar;1
jboss-transaction-api_1_2_spec-1_1_1_Final.jar;1
micrometer-core-1_9_17.jar;1
mysql-connector-j-8_0_33.jar;1
record.jar;1
spring-aop-5_3_31.jar;1
spring-batch-core-4_3_10.jar;1
spring-batch-infrastructure-4_3_10.jar;1
spring-beans-5_3_31.jar;1
spring-context-5_3_31.jar;1
spring-core-5_3_31.jar;1
spring-expression-5_3_31.jar;1
spring-jdbc-5_3_31.jar;1
spring-orm-5_3_31.jar;1
spring-oxm-5_3_31.jar;1
spring-retry-1_3_4.jar;1
spring-tx-5_3_31.jar;1
xstream-1_4_20.jar;1

To RDBMS:

JPA will be used for RDBMS data access.

persistence.xml:

<persistence xmlns="http://java.sun.com/xml/ns/persistence"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
             version="2.0">
   <persistence-unit name="orders">
      <provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
      <class>OrdersJPA</class>
      <class>OrderLinesJPA</class>
      <exclude-unlisted-classes/>
      <properties>
          <property name="hibernate.show_sql" value="false"/>
          <property name="hibernate.connection.driver_class" value="com.mysql.cj.jdbc.Driver"/>
          <property name="hibernate.connection.url" value="jdbc:mysql://arnepc5/Test"/>
          <property name="hibernate.connection.username" value="arne"/>
          <property name="hibernate.connection.password" value="hemmeligt"/>
          <property name="hibernate.connection.pool_size" value="5"/>
      </properties>
   </persistence-unit>
</persistence>

From XML:

Flow:

xml2db.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="dbTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitName" value="orders"/>
    </bean>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>OrdersJAXB</value>
                <value>OrderLinesCollectionJAXB</value>
                <value>OrderLinesJAXB</value>
            </list>
        </property>
    </bean>
    <!-- step beans -->
    <bean id="ordersXmlReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
        <property name="fragmentRootElementName" value="order"/>
        <property name="resource" value="orders1.xml"/>
        <property name="unmarshaller" ref="jaxbMarshaller"/>                                                         
    </bean>
    <bean id="copyJAXB2JPAProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromJAXB"/>
                <bean class="ToJPA"/>
            </list>
        </property>
    </bean>
    <bean id="ordersJPAWriter" class="org.springframework.batch.item.database.JpaItemWriter">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <!-- job -->
    <batch:job id="xml2db">
        <batch:step id="import">
            <batch:tasklet transaction-manager="dbTransactionManager">
                <batch:chunk reader="ordersXmlReader" writer="ordersJPAWriter" processor="copyJAXB2JPAProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

From JSON:

Flow:

json2db.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="dbTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitName" value="orders"/>
    </bean>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <!-- step beans -->
    <bean id="ordersJsonReader" class="org.springframework.batch.item.json.JsonItemReader">
        <property name="resource" value="orders1.json"/>
        <property name="jsonObjectReader">
            <bean class="org.springframework.batch.item.json.JacksonJsonObjectReader">
                <constructor-arg value="Orders"/>
            </bean>
        </property>
    </bean>
    <bean id="copyPlain2JPAProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="ToJPA"/>
            </list>
        </property>
    </bean>
    <bean id="ordersJPAWriter" class="org.springframework.batch.item.database.JpaItemWriter">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <!-- job -->
    <batch:job id="json2db">
        <batch:step id="import">
            <batch:tasklet transaction-manager="dbTransactionManager">
                <batch:chunk reader="ordersJsonReader" writer="ordersJPAWriter" processor="copyPlain2JPAProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

From CSV:

Flow:

There is a little complication for CSV, because we have two CSV files orders1.csv and orderlines1.csv which must be combined.

To handle this we use a setup like:

                                   |-->ordersReader (FlatFileItemReader)-->orders1.csv
bothCsvReader (custom loop merge)--|
                                   |-->orderlinesReader (FlatFileItemReader)-->orderlines1.csv

Combining.groovy:

import org.springframework.batch.item.*

class Combining implements ItemReader<Orders> {
    static mol = new AutoMap(OrderLinesX.class, OrderLines.class, true)
    ItemReader<Orders> ordersReader
    ItemReader<OrderLinesX> orderLinesReader
    OrderLinesX leftover
    @Override
    Orders read() {
        Orders o = ordersReader.read()
        if(o != null) {
            if(leftover != null && leftover.orderId == o.id) {
                OrderLines ol = new OrderLines()
                mol.convert(leftover, ol)
                o.orderLines.add(ol)
                leftover = null
            }
            for(;;) {
                OrderLinesX olx = orderLinesReader.read();
                if(olx == null) {
                    break
                } else if(olx.orderId != o.id) {
                    leftover = olx
                    break
                } else {
                    OrderLines ol = new OrderLines()
                    mol.convert(olx, ol)
                    o.orderLines.add(ol)
                }
            }
        }
        return o
    }
}

csv2db.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="dbTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitName" value="orders"/>
    </bean>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <!-- step beans -->
    <bean id="ordersBean" class="Orders" scope="prototype"/>
    <bean id="ordersCsvReader" class="org.springframework.batch.item.file.FlatFileItemReader">
        <property name="resource" value="orders1.csv"/>
        <property name="lineMapper">
            <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
                <property name="lineTokenizer">
                    <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                        <property name="delimiter" value=","/>
                        <property name="names" value="id,customer,status"/>
                    </bean>
                </property>
                <property name="fieldSetMapper">
                    <bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
                        <property name="prototypeBeanName" value="ordersBean"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
    <bean id="orderLinesBean" class="OrderLinesX" scope="prototype"/>
    <bean id="orderLinesCsvReader" class="org.springframework.batch.item.file.FlatFileItemReader">
        <property name="resource" value="orderlines1.csv"/>
        <property name="lineMapper">
            <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
                <property name="lineTokenizer">
                    <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                        <property name="delimiter" value=","/>
                        <property name="names" value="id,orderId,item,qty,price"/>
                    </bean>
                </property>
                <property name="fieldSetMapper">
                    <bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
                        <property name="prototypeBeanName" value="orderLinesBean"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
    <bean id="bothCsvReader" class="Combining">
        <property name="ordersReader" ref="ordersCsvReader"/>
        <property name="orderLinesReader" ref="orderLinesCsvReader"/>
    </bean>
    <bean id="copyPlain2JPAProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="ToJPA"/>
            </list>
        </property>
    </bean>
    <bean id="ordersJPAWriter" class="org.springframework.batch.item.database.JpaItemWriter">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    </bean>
    <!-- job -->
    <batch:job id="csv2db">
        <batch:step id="import">
            <batch:tasklet transaction-manager="dbTransactionManager">
                <batch:chunk reader="bothCsvReader" writer="ordersJPAWriter" processor="copyPlain2JPAProcessor" commit-interval="1">
                    <batch:streams>
                        <batch:stream ref="ordersCsvReader"/>
                        <batch:stream ref="orderLinesCsvReader"/>
                    </batch:streams>
                </batch:chunk>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

To Index-Sequential files:

Spring Batch does obviously not come with builtin support for VMS index-sequential files, but I have an ISAM library for such access and I have written Spring Batch ItemWriter's based on that.

SingleIsamItemWriter.java:

import java.util.List;

import org.springframework.batch.item.support.AbstractItemStreamItemWriter;

import dk.vajhoej.isam.IsamSource;

public class SingleIsamItemWriter<T> extends AbstractItemStreamItemWriter<T> {
    private IsamSource source;
    public IsamSource getSource() {
        return source;
    }
    public void setSource(IsamSource source) {
        this.source = source;
    }
    @Override
    public void write(List<? extends T> items) throws Exception {
        for(T item : items) {
            source.create(item);
        }
    }
}

WriterSecondarySource.java:

import java.lang.reflect.Method;

import dk.vajhoej.isam.IsamSource;

public class WriterSecondarySource<T> {
    private String primaryList; // name of list in primary source containing records from secondary source 
    private IsamSource source; // secondary source
    private Method primaryListGetter;
    public String getPrimaryList() {
        return primaryList;
    }
    public void setPrimaryList(String primaryList) {
        this.primaryList = primaryList;
    }
    public IsamSource getSource() {
        return source;
    }
    public void setSource(IsamSource source) {
        this.source = source;
    }
    public Method getPrimaryListGetter() {
        return primaryListGetter;
    }
    public void setPrimaryListGetter(Method primaryListGetter) {
        this.primaryListGetter = primaryListGetter;
    }
}

MultiIsamItemWriter.java:

import java.lang.reflect.Method;
import java.util.List;

import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.support.AbstractItemStreamItemWriter;

import dk.vajhoej.isam.IsamException;
import dk.vajhoej.isam.IsamSource;

public class MultiIsamItemWriter<T> extends AbstractItemStreamItemWriter<T> {
    private IsamSource primarySource;
    private Class<T> type;
    private List<WriterSecondarySource> secondarySources;
    public IsamSource getPrimarySource() {
        return primarySource;
    }
    public void setPrimarySource(IsamSource primarySource) {
        this.primarySource = primarySource;
    }
    public Class<T> getType() {
        return type;
    }
    public void setType(Class<T> type) {
        this.type = type;
    }
    public List<WriterSecondarySource> getSecondarySources() {
        return secondarySources;
    }
    public void setSecondarySources(List<WriterSecondarySource> secondarySources) {
        this.secondarySources = secondarySources;
    }
    private Method getGetter(String fldnam) {
        try {
            return type.getMethod("get" + fldnam.substring(0, 1).toUpperCase() + fldnam.substring(1));
        } catch (NoSuchMethodException e) {
            throw new RuntimeException(e);
        } catch (SecurityException e) {
            throw new RuntimeException(e);
        }
    }
    @Override
    public void open(ExecutionContext ec) {
        for(WriterSecondarySource<?> src2 : secondarySources) {
            src2.setPrimaryListGetter(getGetter(src2.getPrimaryList()));
        }
    }
    @Override
    public void write(List<? extends T> items) throws Exception {
        for(T item : items) {
            primarySource.create(item);
            for(WriterSecondarySource<?> src2 : secondarySources) {
                for(Object o : (List<?>)src2.getPrimaryListGetter().invoke(item) ) {
                    src2.getSource().create(o);
                }
            }
        }
    }
    @Override
    public void close() {
        try {
            primarySource.close();
            for(WriterSecondarySource src2 : secondarySources) {
                src2.getSource().close();
            }
        } catch (IsamException e) {
            throw new RuntimeException(e);
        }
    }
}

Note that these writers are not batching/chunking. For two reasons:

I wanted to keep them simple
I don't think there are any performance benefits of batching/chunking - it will end up with one SYS$PUT call per record no matter what

The ISAM library can be downloaded here.

From XML:

Flow:

xml2isq.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>OrdersJAXB</value>
                <value>OrderLinesCollectionJAXB</value>
                <value>OrderLinesJAXB</value>
            </list>
        </property>
    </bean>
    <!-- step beans -->
    <bean id="ordersXmlReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
        <property name="fragmentRootElementName" value="order"/>
        <property name="resource" value="orders1.xml"/>
        <property name="unmarshaller" ref="jaxbMarshaller"/>                                                         
    </bean>
    <bean id="copyJAXB2ISAMProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="FromJAXB"/>
                <bean class="ToISAM"/>
            </list>
        </property>
    </bean>
    <bean id="orderLinesISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orderlines.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="false"/>
    </bean>
    <bean id="orderLines" class="WriterSecondarySource">
        <property name="primaryList" value="orderLinesT"/>
        <property name="source" ref="orderLinesISAMSource"/>
    </bean>
    <bean id="ordersISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orders.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="false"/>
    </bean>
    <bean id="ordersISAMWriter" class="MultiIsamItemWriter">
        <property name="primarySource" ref="ordersISAMSource" />
        <property name="type" value="OrdersISAM"/>
        <property name="secondarySources">
            <list>
                <ref bean="orderLines"/>
            </list>
        </property>
    </bean>
    <!-- job -->
    <batch:job id="xml2isq">
        <batch:step id="import">
            <batch:tasklet transaction-manager="repoTransactionManager">
                <batch:chunk reader="ordersXmlReader" writer="ordersISAMWriter" processor="copyJAXB2ISAMProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

From JSON:

Flow:

json2isq.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <!-- step beans -->
    <bean id="ordersJsonReader" class="org.springframework.batch.item.json.JsonItemReader">
        <property name="resource" value="orders1.json"/>
        <property name="jsonObjectReader">
            <bean class="org.springframework.batch.item.json.JacksonJsonObjectReader">
                <constructor-arg value="Orders"/>
            </bean>
        </property>
    </bean>
    <bean id="copyPlain2ISAMProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="ToISAM"/>
            </list>
        </property>
    </bean>
    <bean id="orderLinesISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orderlines.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="false"/>
    </bean>
    <bean id="orderLines" class="WriterSecondarySource">
        <property name="primaryList" value="orderLinesT"/>
        <property name="source" ref="orderLinesISAMSource"/>
    </bean>
    <bean id="ordersISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orders.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="false"/>
    </bean>
    <bean id="ordersISAMWriter" class="MultiIsamItemWriter">
        <property name="primarySource" ref="ordersISAMSource" />
        <property name="type" value="OrdersISAM"/>
        <property name="secondarySources">
            <list>
                <ref bean="orderLines"/>
            </list>
        </property>
    </bean>
    <!-- job -->
    <batch:job id="json2isq">
        <batch:step id="import">
            <batch:tasklet transaction-manager="repoTransactionManager">
                <batch:chunk reader="ordersJsonReader" writer="ordersISAMWriter" processor="copyPlain2ISAMProcessor" commit-interval="1"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

From CSV:

Flow:

Same issue for CSV as for database export:

                                   |-->ordersReader (FlatFileItemReader)-->orders2.csv
bothCsvReader (custom loop merge)--|
                                   |-->orderlinesReader (FlatFileItemReader)-->orderlines2.csv

csv2isq.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:batch="http://www.springframework.org/schema/batch" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">
    <!-- support beans -->
    <bean id="repoTransactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
        <property name="transactionManager" ref="repoTransactionManager"/>
    </bean>     
    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository"/>
    </bean>
    <bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>OrdersJAXB</value>
                <value>OrderLinesCollectionJAXB</value>
                <value>OrderLinesJAXB</value>
            </list>
        </property>
    </bean>
    <!-- step beans -->
    <bean id="ordersBean" class="Orders" scope="prototype"/>
    <bean id="ordersCsvReader" class="org.springframework.batch.item.file.FlatFileItemReader">
        <property name="resource" value="orders1.csv"/>
        <property name="lineMapper">
            <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
                <property name="lineTokenizer">
                    <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                        <property name="delimiter" value=","/>
                        <property name="names" value="id,customer,status"/>
                    </bean>
                </property>
                <property name="fieldSetMapper">
                    <bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
                        <property name="prototypeBeanName" value="ordersBean"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
    <bean id="orderLinesBean" class="OrderLinesX" scope="prototype"/>
    <bean id="orderLinesCsvReader" class="org.springframework.batch.item.file.FlatFileItemReader">
        <property name="resource" value="orderlines1.csv"/>
        <property name="lineMapper">
            <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
                <property name="lineTokenizer">
                    <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                        <property name="delimiter" value=","/>
                        <property name="names" value="id,orderId,item,qty,price"/>
                    </bean>
                </property>
                <property name="fieldSetMapper">
                    <bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
                        <property name="prototypeBeanName" value="orderLinesBean"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean> 
    <bean id="bothCsvReader" class="Combining">
        <property name="ordersReader" ref="ordersCsvReader"/>
        <property name="orderLinesReader" ref="orderLinesCsvReader"/>
    </bean>
    <bean id="copyPlain2ISAMProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
        <property name="delegates">
            <list>
                <bean class="ToISAM"/>
            </list>
        </property>
    </bean>
    <bean id="orderLinesISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orderlines.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="false"/>
    </bean>
    <bean id="orderLines" class="WriterSecondarySource">
        <property name="primaryList" value="orderLinesT"/>
        <property name="source" ref="orderLinesISAMSource"/>
    </bean>
    <bean id="ordersISAMSource" class="dk.vajhoej.isam.local.LocalIsamSource">
        <constructor-arg value="orders.isq"/>
        <constructor-arg value="dk.vajhoej.vms.rms.IndexSequential"/>
        <constructor-arg value="false"/>
    </bean>
    <bean id="ordersISAMWriter" class="MultiIsamItemWriter">
        <property name="primarySource" ref="ordersISAMSource" />
        <property name="type" value="OrdersISAM"/>
        <property name="secondarySources">
            <list>
                <ref bean="orderLines"/>
            </list>
        </property>
    </bean>
    <!-- job -->
    <batch:job id="csv2isq">
        <batch:step id="import">
            <batch:tasklet transaction-manager="repoTransactionManager">
                <batch:chunk reader="bothCsvReader" writer="ordersISAMWriter" processor="copyPlain2ISAMProcessor" commit-interval="1">
                    <batch:streams>
                        <batch:stream ref="ordersCsvReader"/>
                        <batch:stream ref="orderLinesCsvReader"/>
                    </batch:streams>
                </batch:chunk>
            </batch:tasklet>
        </batch:step>
    </batch:job>
</beans>

Conclusion:

Does it make sense to use Spring Batch instead of custom code?

First we note that it does not save code - in fact this example replaced 30-60 lines of Groovy/Jython with 40-100 lines of XML.

But it does add some robustness and flexibility to the solution.

And on top of that it may have some benefits for some enterprises:

Spring Batch may be the corporate standard and there are resources available with Spring Batch skills
it may be easier to get a XML "config change" approved than a Groovy/Jython/whatever "code change" (*)

*) Technically that is total BS, but sometimes rules for such things does not make any sense.

So it may be worth considering.

Article history:

Version	Date	Description
1.0	March 30th 2025	Initial version

Comments:

Please send comments to Arne Vajhøj

VMS Tech Demo 22 - data import with Spring Batch

Content:

Introduction:

Spring Batch:

Setup:

Data classes:

Data conversion classes:

Main:

To RDBMS:

From XML:

From JSON:

From CSV:

To Index-Sequential files:

From XML:

From JSON:

From CSV:

Conclusion:

Article history:

Other articles:

Comments: