Igwans comment is not really a solution. Sometimes reading files into memory works, sometimes it doesn’t. Sometimes your files are just do big to read into memory.

When comparing the in memory solution to a Java/Scala solution, one should implement the Scala solution with NIO which just uses memory mapped files and OS memory managment. This is much faster than reading files to a buffer by hand (I did use NIO to parse large log files for analysis).

And if one is supposed to develop a buffered implementation or a memory mapped one then my main concern with all languages beside Java comes into play: With Java you usually do not develop applications but assemble them. There are so many great open source libraries around (lucene, svnkit, spring, seam, camel, …) that most of the time you only write plumbing code.

Peace
-stephan

–
Stephan Schmidt :: stephan@reposita.org
Reposita Open Source - Monitor your software development
http://www.reposita.org
Blog at http://stephan.reposita.org - No signal. No noise.

Igwan advocates buffering the whole file before processing it and claims faster times, also Java’s buffered reader implements (who’d have thought it) a reasonable sized buffer (ISTR its 8kb) by default. This means each disk read for Erlang is 1 or 2 bytes and it repeats the process to identify lines, but Java hoovers up a few Kb each time and works out where the lines are afterwards.

With this in mind, the stats you posted are no great surprise - Erlang will be working the IO subsystem pretty hard and that will slow down the whole thing.