Easy to Learn Java: Programming Articles, Examples and Tips

Start with Java in a few days with Java Lessons or Lectures

Home

Code Examples

Java Tools

More Java Tools!

Java Forum

All Java Tips

Books

Submit News
Search the site here...
Search...
 
Search the JavaFAQ.nu
1000 Java Tips ebook

1000 Java Tips - Click here for the high resolution copy!1000 Java Tips - Click here for the high resolution copy!

Java Screensaver, take it here

Free "1000 Java Tips" eBook is here! It is huge collection of big and small Java programming articles and tips. Please take your copy here.

Take your copy of free "Java Technology Screensaver"!.

Easy Learn Java: Programming Articles, Examples and Tips - Page 415


Previous 1060 Stories (530 Pages, 2 Per Page) Next

Lack of Streaming leads to Screaming

Go to all tips in Story by Dr. Kabutz

The Java Specialists' Newsletter [Issue 047] - Lack of Streaming leads to Screaming

Author: Dr. Heinz M. Kabutz

JDK version:

Category: Language

You can subscribe from our home page: http://www.javaspecialists.co.za (which also hosts all previous issues, available free of charge Smile

Welcome to the 47th edition of The Java(tm) Specialists' Newsletter, read by over 3600 Java programmers in 82 countries. I have put my Mauritius trip with photos on my website http://www.javaspecialists.co.za, please have a look under "Courses". Warning: prolonged looking at that webpage is known to cause envy - please enter at own risk!

I am planning Java and Design Patterns courses in South Africa for June. Please have a look at our website for more information.

Is your company thinking of venturing into new business but you do not have the necessary resources? Are you scared of subcontracting work to a non-English speaking country due to the communication problems involved? (no offense to my non-English readers - this is my advert, ok? Wink Then South Africa is your dream come true. In South Africa we speak and write English fluently, so you will not have the typical communication problems that you would find in non-English speaking countries. Our software developers are highly skilled, very good at solving problems and able to pull up their sleeves and get to work. Internationally, South Africans are known for their hard work and dedication to the task at hand. If your company is looking for such resources, please contact me by simply replying to this email, and I will personally see to it that you are contacted within 24 hours.

In the last two weeks I received two questions from readers who ran out of memory when trying to read a big object from the database. In this newsletter I want to explore how you can read a big object from a database without killing your poor JVM, and be scalable as well.

Lack of Streaming leads to Screaming

How do you retrieve big objects from the database in Java? Say you have a database containing previews of movies in DivX format, stored as IMAGE columns. How do you retrieve the 25 megabyte file from the database using JDBC?

Simple. We write a SELECT statement, execute it, and say result_set.getBytes(1). We run the code and it works well for small movie snippets, but as soon as we have a 25 megabyte file, our poor JVM throws an OutOfMemoryError. What's annoying about an OutOfMemoryError is that the stack trace is not filled in (because hey, you've run out of memory!), so you cannot exactly determine where the error occurred, unless you add trace logging. What makes it even more tricky is that some JDBC drivers try to be too clever, resulting in OutOfMemoryErrors.

Let's look at some test code. I have written two test classes, TestDatabaseBlobInsert and TestDatabaseBlobFetch. What surprised me was that of the several drivers that I tested (DataDirect, iNet SPRINTA, Avenir, MS SQL Server Type 4, JDBC/ODBC bridge), the JDBC/ODBC bridge was the fastest for inserting big objects into MS SQL Server. For fetching the data it was the slowest:


import java.sql.DriverManager;
import java.sql.Statement;
import java.sql.SQLException;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.io.ByteArrayInputStream;

public class TestDatabaseBlobInsert {
  private static final String TABLE_DROP =
    "DROP TABLE MovieArchive";
  private static final String TABLE_CREATE =
    "Create Table MovieArchive (moviedata image, title varchar(255))";
  private static final String TABLE_INSERT =
    "INSERT INTO MovieArchive (title, moviedata) VALUES (?,?)";

  private static final int size = 25 * 1024 * 1024;
  private final byte[] data = new byte[size];

  private final Connection con;

  public TestDatabaseBlobInsert(String driver, String url,
      String user, String password)
      throws SQLException, ClassNotFoundException {
    Class.forName(driver);
    con = DriverManager.getConnection(url, user, password);
    System.out.println("Driver: " + driver);
    for (int i=0; ibyte)(Math.random()*255);
  }

  public void setUp() throws SQLException {
    Statement st = con.createStatement();
    try {
      System.out.println("Dropping old table");
      st.executeUpdate(TABLE_DROP);
    } catch(SQLException ex) {} // table might not exist
    System.out.println("Creating new table");
    st.executeUpdate(TABLE_CREATE);
    st.close();
  }

  public void testInsertWithBinaryStream() throws SQLException {
    long start = -System.currentTimeMillis();
    System.out.println("Inserting via BinaryStream");
    PreparedStatement stmt = con.prepareStatement(TABLE_INSERT);
    ByteArrayInputStream bis = new ByteArrayInputStream(data);
    stmt.setString(1, "Babe");
    stmt.setBinaryStream(2, bis, data.length);
    stmt.executeUpdate();
    start += System.currentTimeMillis();
    System.out.println("That took " + start + "ms");
    stmt.close();
  }

  public void testInsertWithSetBytes() throws SQLException {
    long start = -System.currentTimeMillis();
    System.out.println("Inserting via setBytes()");
    PreparedStatement stmt = con.prepareStatement(TABLE_INSERT);
    stmt.setString(1, "On Her Majesty's Secret Service");
    stmt.setBytes(2, data);
    stmt.executeUpdate();
    start += System.currentTimeMillis();
    System.out.println("That took " + start + "ms");
    stmt.close();
  }

  public void testAll() throws SQLException {
    setUp();
    testInsertWithBinaryStream();
    testInsertWithSetBytes();
  }

  public static void main(String[] args) throws Exception {
    if (args.length != 4) usage();
    TestDatabaseBlobInsert test = new TestDatabaseBlobInsert(
      args[0], args[1], args[2], args[3]);
    test.testAll();
  }

  private static void usage() {
    System.out.println(
      "Usage: TestDatabaseBlobInsert driver url username password");
    System.exit(1);
  }
}

I ran this code by setting up an ODBC source pointing to the MS SQL Server database Movies, and then running it with a maximum heap space of 256MB:

java -Xmx256m -classpath . TestDatabaseBlobInsert sun.jdbc.odbc.JdbcOdbcDriver jdbc:odbc:Movies sa ""

The result on my little notebook was the following:

Driver: sun.jdbc.odbc.JdbcOdbcDriver
Dropping old table
Creating new table
Inserting via BinaryStream
That took 78975ms
Inserting via setBytes()
That took 73419ms

Back to the issue at hand - how do we get this data out of the database? The seemingly easiest way is to do the following:

1:  PreparedStatement st = con.prepareStatement(
  "SELECT moviedata FROM MovieArchive WHERE title = ?");
2:  st.setString(1, "Babe");
3:  ResultSet rs = st.executeQuery();
4:  if (rs.next()) {
5:    byte[] data = rs.getBytes(1);
}

This code can easily cause an OutOfMemoryError if the available heap memory is less than the size of the data that you are reading. Now for the 1'000'000 dollar question: Where does OutOfMemoryError occur? That depends on your driver. If you are using the iNet SPRINTA or the Avenir drivers, then you will run out of memory on line 4, i.e. when you call rs.next(). If you are using the DataDirect, Microsoft or ODBC bridge drivers, you will only get the out of memory error on line 5.

How can we write this so that we won't get an out of memory error? Here is some sample code. It is very important that you read the data blocks of bytes at a time, rather than in one big chunk, otherwise your system will definitely not scale to support many users.

import java.sql.*;
import java.io.*;

public class TestDatabaseBlobFetch {
  private static final String TABLE_SELECT =
    "SELECT moviedata FROM MovieArchive WHERE title = ?";

  private final Connection con;

  public TestDatabaseBlobFetch(String driver, String url,
      String user, String password)
      throws SQLException, ClassNotFoundException {
    Class.forName(driver);
    con = DriverManager.getConnection(url, user, password);
    System.out.println("Driver: " + driver);
  }

  public void testSelectBlocksAtATime() throws SQLException {
    long start = -System.currentTimeMillis();
    System.out.println("SELECT: 64kb blocks at a time");
    PreparedStatement stmt = con.prepareStatement(TABLE_SELECT);
    stmt.setString(1, "Babe");
    ResultSet rs = stmt.executeQuery();
    int count=0;
    if (rs.next()) {
      try {
        System.out.println("Retrieving Data");
        OutputStream out = new BufferedOutputStream(
          new FileOutputStream("Data.1"));
        InputStream in = new BufferedInputStream(
          rs.getBinaryStream(1));
        byte[] buf = new byte[65536];
        int i;
        while((i = in.read(buf, 0, buf.length)) != -1) {
          out.write(buf, 0, i);
          count += i;
        }
        out.close();
      } catch(IOException ex) { ex.printStackTrace(); }
    }
    System.out.println("fetched " + count + " bytes");
    start += System.currentTimeMillis();
    System.out.println("That took " + start + "ms");
    stmt.close();
  }

  public void testSelectWithGetBytes() throws SQLException {
    long start = -System.currentTimeMillis();
    System.out.println("SELECT: all at once");
    PreparedStatement stmt = con.prepareStatement(TABLE_SELECT);
    stmt.setString(1, "Babe");
    ResultSet rs = stmt.executeQuery();
    byte[] data = null;
    if (rs.next()) {
      System.out.println("Retrieving Data");
      data = rs.getBytes(1);
      try {
        FileOutputStream out = new FileOutputStream("Data.2");
        out.write(data, 0, data.length);
        out.close();
      } catch(IOException ex) { ex.printStackTrace(); }
    }
    System.out.println("fetched " + data.length + " bytes");
    start += System.currentTimeMillis();
    System.out.println("That took " + start + "ms");
    stmt.close();
  }

  public void testAll() throws SQLException {
    testSelectBlocksAtATime();
    testSelectWithGetBytes();
  }

  public static void main(String[] args) throws Exception {
    if (args.length != 4) usage();
    TestDatabaseBlobFetch test = new TestDatabaseBlobFetch (
      args[0], args[1], args[2], args[3]);
    test.testAll();
  }

  private static void usage() {
    System.out.println(
      "usage: TestDatabaseBlobFetch driver url username password");
    System.exit(1);
  }
}

I tried this with several JDBC drivers, the only Type 4 driver that worked correclty was the DataDirect driver (now released under the Microsoft label). I will not go into the differences between the Microsoft driver and the others, that's for another article. If you want to try this out, you can run it like this:

java -Xmx2m -classpath .;msbase.jar;mssqlserver.jar;msutil.jar
  TestDatabaseBlobFetch com.microsoft.jdbc.sqlserver.SQLServerDriver
  jdbc:microsoft:sqlserver://localhost:1433;DatabaseName=Movies sa ""

Naturally you have to download the Microsoft SQL Server Type 4 driver and put the jar files into the directory from which you are running this code. The output from using the DataDirect Microsoft driver is the following on my machine:

Driver: com.microsoft.jdbc.sqlserver.SQLServerDriver
SELECT: 64kb blocks at a time
Retrieving Data
fetched 26214400 bytes
That took 62746ms
SELECT: all at once
Retrieving Data
Exception in thread "main" java.lang.OutOfMemoryError
        <>

The iNet SPRINTA driver falls over much sooner - actually when you call rs.next():

Driver: com.inet.tds.TdsDriver
SELECT: 64kb blocks at a time
Exception in thread "main" java.lang.OutOfMemoryError
        <>

What is wrong here?

I am finding it very hard to think of a reason to store 25mb files in a database. They are too big to stay in the database's cache for very long. I think that the design is flawed to start with. I would personally rather store the URL to the file in the database, instead of the actual data, and then retrieve the data directly from the file system.

I don't know all the conditions why someone would want to do that, but just remember, you have to stream such big data out of the database chunk by chunk, otherwise you have a serious problem.

Until the next issue ...

Heinz


Copyright 2000-2004 Maximum Solutions, South Africa

Reprint Rights. Copyright subsists in all the material included in this email, but you may freely share the entire email with anyone you feel may be interested, and you may reprint excerpts both online and offline provided that you acknowledge the source as follows: This material from The Java(tm) Specialists' Newsletter by Maximum Solutions (South Africa). Please contact Maximum Solutions for more information.

Java and Sun are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. Maximum Solutions is independent of Sun Microsystems, Inc.

15551 bytes more | comments? | Printer Friendly Page  Send to a Friend | Score: 0
Posted by jalex on Saturday, October 29, 2005 (00:00:00) (2394 reads)

Q:Inheritance seems error-prone. How can I guard against these errors?

Go to all tips in Java IAQ by Peter Norvig

Q:Inheritance seems error-prone. How can I guard against these errors?

Answer: The previous two questions show that a programmer neeeds to be very careful when extending a class, and sometimes just in using a class that extends another class. Problems like these two lead John Ousterhout to say "Implementation inheritance causes the same intertwining and brittleness that have been observed when goto statements are overused. As a result, OO systems often suffer from complexity and lack of reuse." (Scripting, IEEE Computer, March 1998) and Edsger Dijkstra to allegedly say "Object-oriented programming is an exceptionally bad idea which could only have originated in California." (from a collection of signature files). I don't think there's a general way to insure being safe, but there are a few things to be aware of:
  • Extending a class that you don't have source code for is always risky; the documentation may be incomplete in ways you can't foresee.
  • Calling super tends to make these unforeseen problems jump out.
  • You need to pay as much attention to the methods that you don't over-ride as the methods that you do. This is one of the big fallacies of Object-Oriented design using inheritance. It is true that inheritance lets you write less code. But you still have to think about the code you don't write.
  • You're especially looking for trouble if the subclass changes the contract of any of the methods, or of the class as a whole. It is difficult to tell when a contract is changed, since contracts are informal (there is a formal part in the type signature, but the rest appears only in comments). In the Properties example, it is not clear if a contract is being broken, because it is not clear if the defaults are to be considered "entries" in the table or not.


This tip is reprinted on JavaFAQ.nu by by courtesy of Peter Norvig I am thankful for his important contributions to my site - 21 Infrequently Answered Java Questions. Alexandre Patchine



comments? | Printer Friendly Page  Send to a Friend | Score: 0
Posted by jalex on Wednesday, October 26, 2005 (00:00:00) (2250 reads)

Previous 1060 Stories (530 Pages, 2 Per Page) Next

530| 529| 528| 527| 526| 525| 524| 523| 522| 521| 520| 519| 518| 517| 516| 515| 514| 513| 512| 511| 510| 509| 508| 507| 506| 505| 504| 503| 502| 501| 500| 499| 498| 497| 496| 495| 494| 493| 492| 491| 490| 489| 488| 487| 486| 485| 484| 483| 482| 481| 480| 479| 478| 477| 476| 475| 474| 473| 472| 471| 470| 469| 468| 467| 466| 465| 464| 463| 462| 461| 460| 459| 458| 457| 456| 455| 454| 453| 452| 451| 450| 449| 448| 447| 446| 445| 444| 443| 442| 441| 440| 439| 438| 437| 436| 435| 434| 433| 432| 431| 430| 429| 428| 427| 426| 425| 424| 423| 422| 421| 420| 419| 418| 417| 416|
415
| 414| 413| 412| 411| 410| 409| 408| 407| 406| 405| 404| 403| 402| 401| 400| 399| 398| 397| 396| 395| 394| 393| 392| 391| 390| 389| 388| 387| 386| 385| 384| 383| 382| 381| 380| 379| 378| 377| 376| 375| 374| 373| 372| 371| 370| 369| 368| 367| 366| 365| 364| 363| 362| 361| 360| 359| 358| 357| 356| 355| 354| 353| 352| 351| 350| 349| 348| 347| 346| 345| 344| 343| 342| 341| 340| 339| 338| 337| 336| 335| 334| 333| 332| 331| 330| 329| 328| 327| 326| 325| 324| 323| 322| 321| 320| 319| 318| 317| 316| 315| 314| 313| 312| 311| 310| 309| 308| 307| 306| 305| 304| 303| 302| 301| 300| 299| 298| 297| 296| 295| 294| 293| 292| 291| 290| 289| 288| 287| 286| 285| 284| 283| 282| 281| 280| 279| 278| 277| 276| 275| 274| 273| 272| 271| 270| 269| 268| 267| 266| 265| 264| 263| 262| 261| 260| 259| 258| 257| 256| 255| 254| 253| 252| 251| 250| 249| 248| 247| 246| 245| 244| 243| 242| 241| 240| 239| 238| 237| 236| 235| 234| 233| 232| 231| 230| 229| 228| 227| 226| 225| 224| 223| 222| 221| 220| 219| 218| 217| 216| 215| 214| 213| 212| 211| 210| 209| 208| 207| 206| 205| 204| 203| 202| 201| 200| 199| 198| 197| 196| 195| 194| 193| 192| 191| 190| 189| 188| 187| 186| 185| 184| 183| 182| 181| 180| 179| 178| 177| 176| 175| 174| 173| 172| 171| 170| 169| 168| 167| 166| 165| 164| 163| 162| 161| 160| 159| 158| 157| 156| 155| 154| 153| 152| 151| 150| 149| 148| 147| 146| 145| 144| 143| 142| 141| 140| 139| 138| 137| 136| 135| 134| 133| 132| 131| 130| 129| 128| 127| 126| 125| 124| 123| 122| 121| 120| 119| 118| 117| 116| 115| 114| 113| 112| 111| 110| 109| 108| 107| 106| 105| 104| 103| 102| 101| 100| 99| 98| 97| 96| 95| 94| 93| 92| 91| 90| 89| 88| 87| 86| 85| 84| 83| 82| 81| 80| 79| 78| 77| 76| 75| 74| 73| 72| 71| 70| 69| 68| 67| 66| 65| 64| 63| 62| 61| 60| 59| 58| 57| 56| 55| 54| 53| 52| 51| 50| 49| 48| 47| 46| 45| 44| 43| 42| 41| 40| 39| 38| 37| 36| 35| 34| 33| 32| 31| 30| 29| 28| 27| 26| 25| 24| 23| 22| 21| 20| 19| 18| 17| 16| 15| 14| 13| 12| 11| 10| 9| 8| 7| 6| 5| 4| 3| 2| 1|


Home Code Examples Java Forum All Java Tips Books Submit News, Code... Search... Offshore Software Tech Doodling

RSS feed Java FAQ RSS feed Java FAQ News     

    RSS feed Java Forums RSS feed Java Forums

All logos and trademarks in this site are property of their respective owner. The comments are property of their posters, all the rest 1999-2006 by Java FAQs Daily Tips.

Interactive software released under GNU GPL, Code Credits, Privacy Policy