Getting data into databases may not be great computer science, but it is a very common task. Also for data to be delivered to VMS.
There are 2 fundamental approaches to getting data out:
This article is a companion to VMS Tech Demo 18 - getting data out of VMS.
Note that they share both design principles and actual code. But they can be read independently.
RDBMS:
CREATE TABLE orders (
id INTEGER NOT NULL,
customer VARCHAR(32),
status VARCHAR(16),
PRIMARY KEY(id)
);
CREATE TABLE orderlines (
id INTEGER NOT NULL,
orderid INTEGER NOT NULL,
item VARCHAR(32),
qty INTEGER,
price DECIMAL(10, 2),
PRIMARY KEY(id)
);
Index-sequential file:
$ create/fdl=sys$input orders.isq
FILE
ORGANIZATION indexed
RECORD
FORMAT fixed
SIZE 52
KEY 0
SEG0_LENGTH 4
SEG0_POSITION 0
TYPE int4
$
$ create/fdl=sys$input orderlines.isq
FILE
ORGANIZATION indexed
RECORD
FORMAT fixed
SIZE 50
KEY 0
SEG0_LENGTH 4
SEG0_POSITION 0
TYPE int4
KEY 1
SEG0_LENGTH 4
SEG0_POSITION 4
TYPE int4
DUPLICATES yes
$
XML:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<orders>
<order>
<id>1</id>
<customer>A</customer>
<status>Delivered</status>
<order-lines>
<order-line>
<id>1</id>
<item>X</item>
<qty>1</qty>
<price>10.00</price>
</order-line>
</order-lines>
</order>
<order>
<id>2</id>
<customer>B</customer>
<status>Delivered</status>
<order-lines>
<order-line>
<id>2</id>
<item>Y</item>
<qty>2</qty>
<price>20.00</price>
</order-line>
</order-lines>
</order>
<order>
<id>3</id>
<customer>C</customer>
<status>Delivered</status>
<order-lines>
<order-line>
<id>3</id>
<item>Z</item>
<qty>1</qty>
<price>30.00</price>
</order-line>
<order-line>
<id>4</id>
<item>W</item>
<qty>1</qty>
<price>5.00</price>
</order-line>
</order-lines>
</order>
</orders>
JSON:
[
{
"id": 1,
"customer": "A",
"status": "Delivered",
"orderLines": [
{
"id": 1,
"item": "X",
"qty": 1,
"price": 10.00
}
]
},
{
"id": 2,
"customer": "B",
"status": "Delivered",
"orderLines": [
{
"id": 2,
"item": "Y",
"qty": 2,
"price": 20.00
}
]
},
{
"id": 3,
"customer": "C",
"status": "Delivered",
"orderLines": [
{
"id": 3,
"item": "Z",
"qty": 1,
"price": 30.00
},
{
"id": 4,
"item": "W",
"qty": 1,
"price": 5.00
}
]
}
]
CSV:
"1","A","Delivered"
"2","B","Delivered"
"3","C","Delivered"
"1","1","X","1","10.00"
"2","2","Y","2","20.00"
"3","3","Z","1","30.00"
"4","3","W","1","5.00"
We will demo by examples.
Databases:
Languages:
Import formats:
Data paths:
It will be mappings based:
That means very little actual logic code.
No copying fields from result set to data object, no writing CSV quote and delimiters, no writing XML tags, no writing JSON properties.
You read a list of data objects, convert and write a list of data objects.
The conversion/auto mapping step could be omitted if the same data classes supports both input and output format. But that is not a good software design.
For writing RDBMS we will use JPA API with Hibernate as implementation. Very standard solution.
For writing index-sequential files we will use my ISAM library.
For reading CSV files we will use the widely used OpenCSV library.
For reading XML we will use JAXB.
For reading JSON we will use Google Gson.
The files on disk can be transferred to VMS system using HTTP(S), (S)FTP, SMTP, NFS mounted disks, SMB mounted disk or whatever.
For accessing message queue we will use JMS API.
With JMS API it is easy to change message queue. One only need to change the connection factory creation.
Message Queue | Groovy | Python |
---|---|---|
ActiveMQ | cf = new ActiveMQConnectionFactory("tcp://localhost:61616") | cf = ActiveMQConnectionFactory('tcp://localhost:61616') |
RabbitMQ | cf = new RMQConnectionFactory() cf.host = "localhost" cf.port = 5672 |
cf = RMQConnectionFactory() cf.host = 'localhost' cf.port = 5672 |
HornetMQ | props["host"] = "localhost" props["port"] = 5445 cf = new HornetQJMSConnectionFactory(false, new TransportConfiguration("org.hornetq.core.remoting.impl.netty.NettyConnectorFactory"), props)) |
props['host'] = 'localhost' props['port'] = 5445 cf = HornetQJMSConnectionFactory(false, TransportConfiguration('org.hornetq.core.remoting.impl.netty.NettyConnectorFactory', props)) |
Note that connection factory is different if support for XA transactions are needed. XA transaction are way out of scope for this article. If interested check out Transactions - Atomicity.
csv2db.groovy:
import javax.persistence.*
import com.opencsv.bean.*
def process(ois, olis) {
// read from CSV
ocsvdata = new CsvToBeanBuilder(new InputStreamReader(ois)).withType(OrdersCSV.class).build().parse();
olcsvdata = new CsvToBeanBuilder(new InputStreamReader(olis)).withType(OrderLinesCSV.class).build().parse();
// convert
mo = new AutoMap(OrdersCSV.class, OrdersJPA.class, true)
mol = new AutoMap(OrderLinesCSV.class, OrderLinesJPA.class, true)
jpadata = new ArrayList()
xref = new HashMap()
for(ocsv in ocsvdata) {
ojpa = new OrdersJPA()
mo.convert(ocsv, ojpa)
jpadata.add(ojpa)
xref[ojpa.id] = ojpa
}
for(olcsv in olcsvdata) {
oljpa = new OrderLinesJPA()
mol.convert(olcsv, oljpa)
xref[oljpa.orderId].orderLines.add(oljpa)
}
// write to database
emf = Persistence.createEntityManagerFactory("orders")
em = emf.createEntityManager()
for(ojpa in jpadata) {
em.getTransaction().begin()
em.save(ojpa)
em.getTransaction().commit()
}
em.close()
emf.close()
//
println("csv2db done")
}
def process_file(ofnm, olfnm) {
ois = new FileInputStream(ofnm)
olis = new FileInputStream(olfnm)
process(ois, olis)
ois.close()
olis.close()
}
process_file("orders.csv", "orderlines.csv")
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac -cp /javalib/opencsv-5_9.jar OrdersCSV.java OrderLinesCSV.java
$ javac AutoMap.java
$ groovy_cp = ".:/javalib/opencsv-5_9.jar:/javalib/commons-collections4-4_5_0-M2.jar:/javalib/commons-lang3-3_17_0.jar:/javalib/commons-beanutils-1_8_0.jar:/javalib/commons-logging-1_1.jar:''hibpath'"
$ groovy "csv2db.groovy"
csv2db.py:
from java.io import FileInputStream, InputStreamReader
from java.util import ArrayList, HashMap
from javax.persistence import Persistence
from com.opencsv.bean import CsvToBeanBuilder
import OrdersCSV, OrderLinesCSV
import OrdersJPA, OrderLinesJPA
import AutoMap
def process(ois, olis):
# read from CSV
ocsvdata = CsvToBeanBuilder(InputStreamReader(ois)).withType(OrdersCSV).build().parse()
olcsvdata = CsvToBeanBuilder(InputStreamReader(olis)).withType(OrderLinesCSV).build().parse()
# convert
mo = AutoMap(OrdersCSV, OrdersJPA, True)
mol = AutoMap(OrderLinesCSV, OrderLinesJPA, True)
jpadata = ArrayList()
xref = HashMap()
for ocsv in ocsvdata:
ojpa = OrdersJPA()
mo.convert(ocsv, ojpa)
jpadata.add(ojpa)
xref[ojpa.id] = ojpa
for olcsv in olcsvdata:
oljpa = OrderLinesJPA()
mol.convert(olcsv, oljpa)
xref[oljpa.orderId].orderLines.add(oljpa)
# write to database
emf = Persistence.createEntityManagerFactory('orders')
em = emf.createEntityManager()
for ojpa in jpadata:
em.getTransaction().begin()
em.save(ojpa)
em.getTransaction().commit()
em.close()
emf.close()
#
print('csv2db done')
def process_file(ofnm, olfnm):
ois = FileInputStream(ofnm)
olis = FileInputStream(olfnm)
process(ois, olis)
ois.close()
olis.close()
process_file('orders.csv', 'orderlines.csv')
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac -cp /javalib/opencsv-5_9.jar OrdersCSV.java OrderLinesCSV.java
$ javac AutoMap.java
$ jython_libs_prefix = hibpath + ":"
$ define/nolog jython_libs "/javalib/opencsv-5_9.jar:/javalib/commons-collections4-4_5_0-M2.jar:/javalib/commons-lang3-3_17_0.jar:/javalib/commons-beanutils-1_8_0.jar:/javalib/commons-logging-1_1.jar"
$ jython "csv2db.py"
csv2isq.groovy:
import java.io.*
import com.opencsv.bean.*
import dk.vajhoej.isam.map.*
def process(ois, olis) {
// read from CSV
ocsvdata = new CsvToBeanBuilder(new InputStreamReader(ois)).withType(OrdersCSV.class).build().parse();
olcsvdata = new CsvToBeanBuilder(new InputStreamReader(olis)).withType(OrderLinesCSV.class).build().parse();
// convert
mo = new AutoMap(OrdersCSV.class, OrdersISAM.class, true)
mol = new AutoMap(OrderLinesCSV.class, OrderLinesISAM.class, true)
isamdata = new ArrayList()
xref = new HashMap()
for(ocsv in ocsvdata) {
oisam = new OrdersISAM()
mo.convert(ocsv, oisam)
isamdata.add(oisam)
xref[oisam.id] = oisam
}
for(olcsv in olcsvdata) {
olisam = new OrderLinesISAM()
mol.convert(olcsv, olisam)
xref[olisam.orderId].orderLinesT.add(olisam)
}
// write to index-sequential file
oisqmp = IsamMap.createIsamMapRMS("orders.isq", OrdersISAM.class)
olisqmp = IsamMap.createIsamMapRMS("orderlines.isq", OrderLinesISAM.class)
for(oisam in isamdata) {
oisqmp.put(oisam.id, oisam)
for(olisam in oisam.orderLinesT) {
olisqmp.put(olisam.id, olisam)
}
}
//
println("csv2isq done")
}
def process_file(ofnm, olfnm) {
ois = new FileInputStream(ofnm)
olis = new FileInputStream(olfnm)
process(ois, olis)
ois.close()
olis.close()
}
process_file("orders.csv", "orderlines.csv")
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac -cp /javalib/opencsv-5_9.jar OrdersCSV.java OrderLinesCSV.java
$ javac AutoMap.java
$ groovy_cp = ".:/javalib/opencsv-5_9.jar:/javalib/commons-collections4-4_5_0-M2.jar:/javalib/commons-lang3-3_17_0.jar:/javalib/commons-beanutils-1_8_0.jar:/javalib/commons-logging-1_1.jar:record.jar:isam.jar:isam-vms.jar"
$ groovy "csv2isq.groovy"
csv2isq.py:
from java.io import FileInputStream, InputStreamReader
from java.util import ArrayList, HashMap
from com.opencsv.bean import CsvToBeanBuilder
from dk.vajhoej.isam.map import IsamMap
import OrdersCSV, OrderLinesCSV
import OrdersISAM, OrderLinesISAM
import AutoMap
def process(ois, olis):
# read from CSV
ocsvdata = CsvToBeanBuilder(InputStreamReader(ois)).withType(OrdersCSV).build().parse()
olcsvdata = CsvToBeanBuilder(InputStreamReader(olis)).withType(OrderLinesCSV).build().parse()
# convert
mo = AutoMap(OrdersCSV, OrdersISAM, True)
mol = AutoMap(OrderLinesCSV, OrderLinesISAM, True)
isamdata = ArrayList()
xref = HashMap()
for ocsv in ocsvdata:
oisam = OrdersISAM()
mo.convert(ocsv, oisam)
isamdata.add(oisam)
xref[oisam.id] = oisam
for olcsv in olcsvdata:
olisam = OrderLinesISAM()
mol.convert(olcsv, olisam)
xref[olisam.orderId].orderLinesT.add(olisam)
# write to index-sequential file
oisqmp = IsamMap.createIsamMapRMS('orders.isq', OrdersISAM)
olisqmp = IsamMap.createIsamMapRMS('orderlines.isq', OrderLinesISAM)
for oisam in isamdata:
oisqmp.put(oisam.id, oisam)
for olisam in oisam.orderLinesT:
olisqmp.put(olisam.id, olisam)
#
print('csv2isq done')
def process_file(ofnm, olfnm):
ois = FileInputStream(ofnm)
olis = FileInputStream(olfnm)
process(ois, olis)
ois.close()
olis.close()
process_file('orders.csv', 'orderlines.csv')
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac -cp /javalib/opencsv-5_9.jar OrdersCSV.java OrderLinesCSV.java
$ javac AutoMap.java
$ define/nolog jython_libs "/javalib/opencsv-5_9.jar:/javalib/commons-collections4-4_5_0-M2.jar:/javalib/commons-lang3-3_17_0.jar:/javalib/commons-beanutils-1_8_0.jar:/javalib/commons-logging-1_1.jar:record.jar:isam.jar:isam-vms.jar"
$ jython "csv2isq.py"
xml2db.groovy:
import javax.jms.*
import javax.persistence.*
import javax.xml.bind.*
import org.apache.activemq.*
def process(is) {
// read from XML
jxbctx = JAXBContext.newInstance(OrdersCollectionJAXB.class, OrdersJAXB.class, OrderLinesCollectionJAXB.class, OrderLinesJAXB.class)
um = jxbctx.createUnmarshaller()
jaxbdata = um.unmarshal(is)
// convert
mo = new AutoMap(OrdersJAXB.class, OrdersJPA.class, true)
mol = new AutoMap(OrderLinesJAXB.class, OrderLinesJPA.class, true)
jpadata = new ArrayList()
for(ojaxb in jaxbdata.list) {
ojpa = new OrdersJPA()
mo.convert(ojaxb, ojpa)
for(oljaxb in ojaxb.orderLines.list) {
oljpa = new OrderLinesJPA()
mol.convert(oljaxb, oljpa)
oljpa.orderId = ojpa.id
ojpa.orderLines.add(oljpa)
}
jpadata.add(ojpa)
}
// write to database
emf = Persistence.createEntityManagerFactory("orders")
em = emf.createEntityManager()
for(ojpa in jpadata) {
em.getTransaction().begin()
em.save(ojpa)
em.getTransaction().commit()
}
em.close()
emf.close()
//
println("xml2db done")
}
def process_file(fnm) {
is = new FileInputStream(fnm)
process(is)
is.close()
}
if(args[0] == "file") {
process_file(args[1])
} else if(args[0] == "queue") {
qcf = new ActiveMQConnectionFactory(args[1])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(false, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue("XmlQ")
receiver = ses.createReceiver(q)
while(true) {
msg = receiver.receive(1000)
if(msg == null) break
bais = new ByteArrayInputStream(msg.text.bytes)
process(bais)
bais.close()
}
receiver.close()
ses.close()
con.close()
}
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ groovy_cp = ".:''hibpath':/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar"
$ groovy "xml2db.groovy" "file" "orders.xml"
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ groovy_cp = ".:''hibpath':/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar"
$ groovy "xml2db.groovy" "queue" "tcp://localhost:61616"
xml2db.py:
from sys import argv
from java.io import FileInputStream, ByteArrayInputStream
from java.util import ArrayList
from javax.jms import Session
from javax.persistence import Persistence
from javax.xml.bind import JAXBContext
from org.apache.activemq import ActiveMQConnectionFactory
import OrdersCollectionJAXB, OrdersJAXB, OrderLinesCollectionJAXB, OrderLinesJAXB
import OrdersJPA, OrderLinesJPA
import AutoMap
def process(xis):
# read from XML
jxbctx = JAXBContext.newInstance(OrdersCollectionJAXB, OrdersJAXB, OrderLinesCollectionJAXB, OrderLinesJAXB)
um = jxbctx.createUnmarshaller()
jaxbdata = um.unmarshal(xis)
# convert
mo = AutoMap(OrdersJAXB, OrdersJPA, True)
mol = AutoMap(OrderLinesJAXB, OrderLinesJPA, True)
jpadata = ArrayList()
for ojaxb in jaxbdata.list:
ojpa = OrdersJPA()
mo.convert(ojaxb, ojpa)
for oljaxb in ojaxb.orderLines.list:
oljpa = OrderLinesJPA()
mol.convert(oljaxb, oljpa)
oljpa.orderId = ojpa.id
ojpa.orderLines.add(oljpa)
jpadata.add(ojpa)
# write to database
emf = Persistence.createEntityManagerFactory('orders')
em = emf.createEntityManager()
for ojpa in jpadata:
em.getTransaction().begin()
em.save(ojpa)
em.getTransaction().commit()
em.close()
emf.close()
#
print('xml2db done')
def process_file(fnm):
xis = FileInputStream(fnm)
process(xis)
xis.close()
if argv[1] == 'file':
process_file(argv[2])
elif argv[1] == 'queue':
qcf = ActiveMQConnectionFactory(argv[2])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(False, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue('XmlQ')
receiver = ses.createReceiver(q)
while True:
msg = receiver.receive(1000)
if msg == None: break
bais = ByteArrayInputStream(msg.text.encode('utf-8'))
process(bais)
bais.close()
receiver.close()
ses.close()
con.close()
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ jython_libs_prefix = hibpath + ":"
$ define/nolog jython_libs "activemq-all-5_4_3.jar"
$ jython "xml2db.py" "file" "orders.xml"
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ jython_libs_prefix = hibpath + ":"
$ define/nolog jython_libs "activemq-all-5_4_3.jar"
$ jython "xml2db.py" "queue" "tcp://localhost:61616"
xml2isq.groovy:
import java.io.*
import javax.jms.*
import javax.xml.bind.*
import org.apache.activemq.*
import dk.vajhoej.isam.map.*
def process(is) {
// read from XML
jxbctx = JAXBContext.newInstance(OrdersCollectionJAXB.class, OrdersJAXB.class, OrderLinesCollectionJAXB.class, OrderLinesJAXB.class)
um = jxbctx.createUnmarshaller()
jaxbdata = um.unmarshal(is)
// convert
mo = new AutoMap(OrdersJAXB.class, OrdersISAM.class, true)
mol = new AutoMap(OrderLinesJAXB.class, OrderLinesISAM.class, true)
isamdata = new ArrayList()
for(ojaxb in jaxbdata.list) {
oisam = new OrdersISAM()
mo.convert(ojaxb, oisam)
for(oljaxb in ojaxb.orderLines.list) {
olisam = new OrderLinesISAM()
mol.convert(oljaxb, olisam)
olisam.orderId = oisam.id
oisam.orderLinesT.add(olisam)
}
isamdata.add(oisam)
}
// write to index-sequential file
oisqmp = IsamMap.createIsamMapRMS("orders.isq", OrdersISAM.class)
olisqmp = IsamMap.createIsamMapRMS("orderlines.isq", OrderLinesISAM.class)
for(oisam in isamdata) {
oisqmp.put(oisam.id, oisam)
for(olisam in oisam.orderLinesT) {
olisqmp.put(olisam.id, olisam)
}
}
//
println("xml2isq done")
}
def process_file(fnm) {
is = new FileInputStream(fnm)
process(is)
is.close()
}
if(args[0] == "file") {
process_file(args[1])
} else if(args[0] == "queue") {
qcf = new ActiveMQConnectionFactory(args[1])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(false, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue("XmlQ")
receiver = ses.createReceiver(q)
while(true) {
msg = receiver.receive(1000)
if(msg == null) break
bais = new ByteArrayInputStream(msg.text.bytes)
process(bais)
bais.close()
}
receiver.close()
ses.close()
con.close()
}
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ groovy_cp = ".:record.jar:isam.jar:isam-vms.jar:/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar"
$ groovy "xml2isq.groovy" "file" "orders.xml"
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ groovy_cp = ".:record.jar:isam.jar:isam-vms.jar:/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar"
$ groovy "xml2isq.groovy" "queue" "tcp://localhost:61616"
xml2isq.py:
from sys import argv
from java.io import FileInputStream, ByteArrayInputStream
from java.util import ArrayList
from javax.jms import Session
from javax.xml.bind import JAXBContext
from org.apache.activemq import ActiveMQConnectionFactory
from dk.vajhoej.isam.map import IsamMap
import OrdersCollectionJAXB, OrdersJAXB, OrderLinesCollectionJAXB, OrderLinesJAXB
import OrdersISAM, OrderLinesISAM
import AutoMap
def process(xis):
# read from XML
jxbctx = JAXBContext.newInstance(OrdersCollectionJAXB, OrdersJAXB, OrderLinesCollectionJAXB, OrderLinesJAXB)
um = jxbctx.createUnmarshaller()
jaxbdata = um.unmarshal(xis)
# convert
mo = AutoMap(OrdersJAXB, OrdersISAM, True)
mol = AutoMap(OrderLinesJAXB, OrderLinesISAM, True)
isamdata = ArrayList()
for ojaxb in jaxbdata.list:
oisam = OrdersISAM()
mo.convert(ojaxb, oisam)
for oljaxb in ojaxb.orderLines.list:
olisam = OrderLinesISAM()
mol.convert(oljaxb, olisam)
olisam.orderId = oisam.id
oisam.orderLinesT.add(olisam)
isamdata.add(oisam)
# write to index-sequential file
oisqmp = IsamMap.createIsamMapRMS('orders.isq', OrdersISAM)
olisqmp = IsamMap.createIsamMapRMS('orderlines.isq', OrderLinesISAM)
for oisam in isamdata:
oisqmp.put(oisam.id, oisam)
for olisam in oisam.orderLinesT:
olisqmp.put(olisam.id, olisam)
#
print('xml2isq done')
def process_file(fnm):
xis = FileInputStream(fnm)
process(xis)
xis.close()
if argv[1] == 'file':
process_file(argv[2])
elif argv[1] == 'queue':
qcf = ActiveMQConnectionFactory(argv[2])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(False, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue('XmlQ')
receiver = ses.createReceiver(q)
while True:
msg = receiver.receive(1000)
if msg == None: break
bais = ByteArrayInputStream(msg.text.encode('utf-8'))
process(bais)
bais.close()
receiver.close()
ses.close()
con.close()
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ define/nolog jython_libs "record.jar:isam.jar:isam-vms.jar:activemq-all-5_4_3.jar"
$ jython "xml2isq.py" "file" "orders.xml"
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac OrdersCollectionJAXB.java OrdersJAXB.java OrderLinesCollectionJAXB.java OrderLinesJAXB.java
$ javac AutoMap.java
$ define/nolog jython_libs "record.jar:isam.jar:isam-vms.jar:activemq-all-5_4_3.jar"
$ jython "xml2isq.py" "queue" "tcp://localhost:61616"
json2db.groovy:
import javax.persistence.*
import javax.jms.*
import org.apache.activemq.*
import com.google.gson.*
def process(is) {
// read from JSON
gson = new Gson()
data = gson.fromJson(new InputStreamReader(is), Orders[].class)
// convert
mo = new AutoMap(Orders.class, OrdersJPA.class, true)
mol = new AutoMap(OrderLines.class, OrderLinesJPA.class, true)
jpadata = new ArrayList()
for(o in data) {
ojpa = new OrdersJPA()
mo.convert(o, ojpa)
for(ol in o.orderLines) {
oljpa = new OrderLinesJPA()
mol.convert(ol, oljpa)
oljpa.orderId = ojpa.id
ojpa.orderLines.add(oljpa)
}
jpadata.add(ojpa)
}
// write to database
emf = Persistence.createEntityManagerFactory("orders")
em = emf.createEntityManager()
for(ojpa in jpadata) {
em.getTransaction().begin()
em.save(ojpa)
em.getTransaction().commit()
}
em.close()
emf.close()
//
println("json2db done")
}
def process_file(fnm) {
is = new FileInputStream(fnm)
process(is)
is.close()
}
if(args[0] == "file") {
process_file(args[1])
} else if(args[0] == "queue") {
qcf = new ActiveMQConnectionFactory(args[1])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(false, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue("JsonQ")
receiver = ses.createReceiver(q)
while(true) {
msg = receiver.receive(1000)
if(msg == null) break
bais = new ByteArrayInputStream(msg.text.bytes)
process(bais)
bais.close()
}
receiver.close()
ses.close()
con.close()
}
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ groovy_cp = ".:''hibpath':/javalib/gson-2_2_4.jar:/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar:/javalib/gson-2_2_4.jar"
$ groovy "json2db.groovy" "file" "orders.json"
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ groovy_cp = ".:''hibpath':/javalib/gson-2_2_4.jar:/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar:/javalib/gson-2_2_4.jar"
$ groovy "json2db.groovy" "queue" "tcp://localhost:61616"
json2db.py:
from sys import argv
from java.io import FileInputStream, InputStreamReader, ByteArrayInputStream
from java.lang import Class
from java.util import ArrayList
from javax.persistence import Persistence
from javax.jms import Session
from org.apache.activemq import ActiveMQConnectionFactory
from com.google.gson import Gson
import Orders, OrderLines
import OrdersJPA, OrderLinesJPA
import AutoMap
def process(xis):
# read from JSON
gson = Gson()
data = gson.fromJson(InputStreamReader(xis), Class.forName("[LOrders;"))
# convert
mo = AutoMap(Orders, OrdersJPA, True)
mol = AutoMap(OrderLines, OrderLinesJPA, True)
jpadata = ArrayList()
for o in data:
ojpa = OrdersJPA()
mo.convert(o, ojpa)
for ol in o.orderLines:
oljpa = OrderLinesJPA()
mol.convert(ol, oljpa)
oljpa.orderId = ojpa.id
ojpa.orderLines.add(oljpa)
jpadata.add(ojpa)
# write to database
emf = Persistence.createEntityManagerFactory('orders')
em = emf.createEntityManager()
for ojpa in jpadata:
em.getTransaction().begin()
em.save(ojpa)
em.getTransaction().commit()
em.close()
emf.close()
#
print('json2db done')
def process_file(fnm):
xis = FileInputStream(fnm)
process(xis)
xis.close()
if argv[1] == 'file':
process_file(argv[2])
elif argv[1] == 'queue':
qcf = ActiveMQConnectionFactory(argv[2])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(False, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue('JsonQ')
receiver = ses.createReceiver(q)
while(True):
msg = receiver.receive(1000)
if\ msg == None: break
bais = ByteArrayInputStream(msg.text.encode('utf-8'))
process(bais)
bais.close()
receiver.close()
ses.close()
con.close()
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ jython_libs_prefix = hibpath + ":"
$ define/nolog jython_libs "/javalib/gson-2_2_4.jar:activemq-all-5_4_3.jar"
$ jython "json2db.py" "file" "orders.json"
$ hibpath = "/javalib/javax_persistence-api-2_2.jar:/javalib/hibernate-core-5_6_5_Final.jar:/javalib/hibernate-commons-annotations-5_1_2_Final.jar:/javalib/javax_activation-api-1_2_0.jar:/javalib/jboss-transaction-api_1_2_spec-1_1_1_Final.jar:/javalib/istack-commons-runtime-3_0_7.jar:/javalib/stax-ex-1_8.jar:/javalib/txw2-2_3_1.jar:/javalib/jboss-logging-3_4_3_Final.jar:/javalib/antlr-2_7_7.jar:/javalib/byte-buddy-1_12_7.jar:/javalib/classmate-1_5_1.jar:/javalib/jandex-2_4_2_Final.jar:/javalib/mysql-connector-j-8_0_33.jar"
$ javac -cp /javalib/javax_persistence-api-2_2.jar OrdersJPA.java OrderLinesJPA.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ jython_libs_prefix = hibpath + ":"
$ define/nolog jython_libs "/javalib/gson-2_2_4.jar:activemq-all-5_4_3.jar"
$ jython "json2db.py" "queue" "tcp://localhost:61616"
json2isq.groovy:
import java.io.*
import javax.jms.*
import org.apache.activemq.*
import com.google.gson.*
import dk.vajhoej.isam.map.*
def process(is) {
// read from JSON
gson = new Gson()
data = gson.fromJson(new InputStreamReader(is), Orders[].class)
// convert
mo = new AutoMap(Orders.class, OrdersISAM.class, true)
mol = new AutoMap(OrderLines.class, OrderLinesISAM.class, true)
isamdata = new ArrayList()
for(o in data) {
oisam = new OrdersISAM()
mo.convert(o, oisam)
for(ol in o.orderLines) {
olisam = new OrderLinesISAM()
mol.convert(ol, olisam)
olisam.orderId = oisam.id
oisam.orderLinesT.add(olisam)
}
isamdata.add(oisam)
}
// write to index-sequential file
oisqmp = IsamMap.createIsamMapRMS("orders.isq", OrdersISAM.class)
olisqmp = IsamMap.createIsamMapRMS("orderlines.isq", OrderLinesISAM.class)
for(oisam in isamdata) {
oisqmp.put(oisam.id, oisam)
for(olisam in oisam.orderLinesT) {
olisqmp.put(olisam.id, olisam)
}
}
//
println("json2isq done")
}
def process_file(fnm) {
is = new FileInputStream(fnm)
process(is)
is.close()
}
if(args[0] == "file") {
process_file(args[1])
} else if(args[0] == "queue") {
qcf = new ActiveMQConnectionFactory(args[1])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(false, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue("JsonQ")
receiver = ses.createReceiver(q)
while(true) {
msg = receiver.receive(1000)
if(msg == null) break
bais = new ByteArrayInputStream(msg.text.bytes)
process(bais)
bais.close()
}
receiver.close()
ses.close()
con.close()
}
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ groovy_cp = ".:record.jar:isam.jar:isam-vms.jar:/javalib/gson-2_2_4.jar:/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar"
$ groovy "json2isq.groovy" "file" "orders.json"
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ groovy_cp = ".:record.jar:isam.jar:isam-vms.jar:/javalib/gson-2_2_4.jar:/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar"
$ groovy "json2isq.groovy" "queue" "tcp://localhost:61616"
json2isq.py:
from sys import argv
import jarray
from java.io import FileInputStream, InputStreamReader, ByteArrayInputStream
from java.lang import Class
from java.util import ArrayList
from javax.jms import Session
from org.apache.activemq import ActiveMQConnectionFactory
from com.google.gson import Gson
from dk.vajhoej.isam.map import IsamMap
import Orders, OrderLines
import OrdersISAM, OrderLinesISAM
import AutoMap
def process(xis):
# read from JSON
gson = Gson()
data = gson.fromJson(InputStreamReader(xis), Class.forName("[LOrders;"))
# convert
mo = AutoMap(Orders, OrdersISAM, True)
mol = AutoMap(OrderLines, OrderLinesISAM, True)
isamdata = ArrayList()
for o in data:
oisam = OrdersISAM()
mo.convert(o, oisam)
for ol in o.orderLines:
olisam = OrderLinesISAM()
mol.convert(ol, olisam)
olisam.orderId = oisam.id
oisam.orderLinesT.add(olisam)
isamdata.add(oisam)
# write to index-sequential file
oisqmp = IsamMap.createIsamMapRMS('orders.isq', OrdersISAM)
olisqmp = IsamMap.createIsamMapRMS('orderlines.isq', OrderLinesISAM)
for oisam in isamdata:
oisqmp.put(oisam.id, oisam)
for olisam in oisam.orderLinesT:
olisqmp.put(olisam.id, olisam)
#
print('json2isq done')
def process_file(fnm):
xis = FileInputStream(fnm)
process(xis)
xis.close()
if argv[1] == 'file':
process_file(argv[2])
elif argv[1] == 'queue':
qcf = ActiveMQConnectionFactory(argv[2])
con = qcf.createQueueConnection()
con.start()
ses = con.createQueueSession(False, Session.AUTO_ACKNOWLEDGE)
q = ses.createQueue('JsonQ')
receiver = ses.createReceiver(q)
while(True):
msg = receiver.receive(1000)
if msg == None: break
bais = ByteArrayInputStream(msg.text.encode('utf-8'))
process(bais)
bais.close()
receiver.close()
ses.close()
con.close()
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ define/nolog jython_libs "record.jar:isam.jar:isam-vms.jar:/javalib/gson-2_2_4.jar:activemq-all-5_4_3.jar"
$ jython "json2isq.py" "file" "orders.json"
$ javac -cp record.jar:isam.jar:isam-vms.jar OrdersISAM.java OrderLinesISAM.java
$ javac Orders.java OrderLines.java
$ javac AutoMap.java
$ groovy_cp = ".:record.jar:isam.jar:isam-vms.jar:/javalib/gson-2_2_4.jar:/activemq$root/lib/activemq-client-5.16.7.jar:/activemq$root/lib/geronimo-jms_1.1_spec-1.1.1.jar:/activemq$root/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/activemq$root/lib/hawtbuf-1.11.jar"
$ define/nolog jython_libs "record.jar:isam.jar:isam-vms.jar:/javalib/gson-2_2_4.jar:activemq-all-5_4_3.jar"
$ jython "json2isq.py" "queue" "tcp://localhost:61616"
Version | Date | Description |
---|---|---|
1.0 | October 25th 2024 | Initial version |
See list of all articles here
Please send comments to Arne Vajhøj