Some times it is necessary to load the records line by line from a large file and insert into the database table. There can be several ways to do this:

• Use the database provided tool to load the data into table. It can be a special sql command or a special stored procedure. This method is quite database specific and not portable.
• Write the program to load the data into the database table (Using JDBC if the program is written in Java).  This method can be applied to different databases if the insert SQL statement conforms to a level of standard.
• Use JPA to persist the object one by one. This allows for changing database more easily (under the condition that the JPA provider supports the database.

We used the third options to load the records into the database as we want to support multiple databases without  low level tweaking. However, the first implementation seemed very slow compared with the second option. And it became slower and slower when process of persisting records  progressed.

After a few checking, we realized that we didn’t call the method “clear” of the class EntityManager each time we persisted the object.  The EntityManager kept managing these objects so the resource was not released immediately, making it become slower and slower.

Once we add the “clear” method call, the performance became acceptable. So lesson learned about the use of JPA.

Change the default storage engine

By default, OpenJPA uses innodb as the storage engine. While my application does not need to support transactions and other fancy db features, i’d like to use more plain and “traditional” myisam storage engine. To do so, I just need to set the property “openjpa.jdbc.DBDictionary” with the value “TableType=myisam”. This property object is the one passed to the method “Persistence.createEntityManagerFactory(String, Properties)”.

Allow to store bigger objects (blobs)
By default, OpenJPA maps the java object to mysql data type “blob”. However, as you may be aware of, blob type in mysql only can store up to 65536(2¹⁶) bytes of data (check http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html), which is quite smaller than the blob in JavaDB or Derby (2G).

Mysql does provide other data types which supports larger storage. mediumblob supports up to 16M while longblob up to 4G. So if I want to store objects other than 64K, I need to change the mapping in OpenJPA. To change the mapping, just set the same property “openjpa.jdbc.DBDictionary”, but with value “blobTypeName=mediumblob”.

If I want both myisam as storage engine and mediumblob as mapped blob type, the property should be set as: “TableType=myisam,blobTypeName=mediumblob”.

Avoid the error “Packet too large”
Now i can save objects up to 16M in size to the database. I run my program, only to find another exception popped up. It says something like “Packet too large”.

This requires the change of mysql server setting as indicated in mysql documentation. The setting “max_allowed_packet“, default to 1M bytes, is too small in my case.

This problem happened to me when i tried the apache derby database (now it’s also called java db, part of jdk 6).
Trying the following command in ij (the bundled simple console) just gave me an error like this “ERROR: Unable to establish connection”:

ij> connect “jdbc:derby://localhost/firstdb;create=true”;

It’s annoying, because i didn’t know what’s wrong from the error message.
Finally, i found out the reason, which is quite trivial. That is: i should not use the double quote (”) to quote the database url. Instead, i need to use single quote (’).

Maybe it’s documented somewhere in the derby documents, which , unfortunately, i missed.