Semergence

So Tim Bray finds out that Erlang IO is slow. I can attest to this fact, as my recent work on reading large files in Erlang has shown that IO and string manipulation is much slower than I would have wanted.

Yes, like Bray, my file reading is single threaded (although, what I do with the line is very multi-threaded) so I suppose using a single thread for Erlang isn’t very Erlang-like in the first place.

In the meantime, I’m porting my OLAP cube generator to Scala. The assumption (and shortly, hopefully proof) is that the JVM can do file IO much better than Erlang, yet I can still take advantage of Scala’s Actors to retain my concurrency.

Update: OK, some numbers and code. This is a benchmark for Erlang and Scala to read in a file line by line.

First, the Erlang code:

	process_file2(Filename) ->
		{ok, File} = file:open(Filename, read),
		process_lines2(File).

	process_lines2(File) ->
		case io:get_line(File, '') of
			eof -> file:close(File);
			_ -> process_lines2(File)
		end.

Now the Scala code:

object LineReader {

  def foreachline(in: BufferedReader, f: String => Unit): Unit = {
    val line = in.readLine()
    if (line == null) return
    else f(line)
    foreachline(in, f)
  }

  def forLines(filename: String, f: String => Unit) = {
    val in = new BufferedReader(new FileReader(filename))
    foreachline(in, f)
    in.close()
  }

}

OK, so these aren’t exactly the same. The Scala example is dispatching to a function, so Scala is even at a disadvantage.

The timings, three runs each, on my MacBook Pro 2.2 Ghz Intel Core 2 Duo. Erlang is the BEAM emulator 5.5.5 and Scala is 2.6 running on JDK 1.5 on Mac OS X. Erlang code was compiled with HIPE.

I am reading in a 1028071833 bytes file with 10037355 lines.

I was trying to enable SMP on my Windows Erlang install, but I was shocked to find out that SMP is not supported in the official Windows Erlang distribution.

I confirmed this by reading the Errata for Programming Erlang:

#8893: “SMP Erlang has been enabled … since R11B-0 on all platforms where SMP Erlang is known to work.”
PLEASE mention that Windows is NOT among them.

What the heck? That seems like a horrible little secret of the Erlang world. I don’t want to compile my own Erlang on windows just to get SMP (which, imo, is the killer feature of erlang).

When I was testing performance in Erlang using funs, I mentioned I wanted to see what would happen if I made the functions tail recursive. I just took a crack at it, and it looks like making this particular function tail recursive didn’t help performance. (as always, please let me know if my assumptions are incorrect.)

Tail Recursive Line Processing

process_file(Filename) ->
	{ok, File} = file:open(Filename, read),
	process_lines(File, first, 0).

process_line(eof) -> done;
process_line(Line) ->
	L = string:tokens(string:strip(Line, both, $n), "	"),
	{Dimensions, Measures} = lists:split(10, L),
	lists:map(fun(X) -> {I,_} = string:to_integer(X), I end, Measures).

process_lines(_, eof, _) -> done;
process_lines(File, PreviousLine, LineNum) ->
	Line = io:get_line(File, ''),
	process_line(Line),
	process_lines(File, Line, LineNum + 1).

Running this code agains a file of 10037355 lines and 1028071833 bytes big, it takes on average 401.40 seconds. This is only slightly better than the previous code attempts that are not tail recursive (and imo easier to read).

As I code more and more in Erlang, I’m very interested how to achieve the best performance possible. I’m very new to the language, so I’m still unsure what the best practices are concerning performance. Tonight I was testing the effects of using a fun() in an algorithm.

My test concerns reading a tab delimited text file, tokenizing it, and converting any numbers into integers. I’ve split the program into two conceptual parts: 1) the file IO and line reading, and 2) the handling of the line. I wanted to test the performance differences between using a fun() for the line handling vs just including the line handling code directly.

My test text file is 10037355 lines long and 1028071833 bytes big. I compiled my code using HIPE.

The quick answer is, using a fun() is slightly slower than not using it (which is to be expected). For my particular test, using a fun() was approximately 20% slower.

Here’s the code. I stole the erlang timing functions from David King (thanks David!).

Using a fun()

time_takes(Mod,Fun,Args) ->
  Start=erlang:now(),
  Result = apply(Mod,Fun,Args),
  Stop=erlang:now(),
  io:format("~p~n",[time_diff(Start,Stop)]),
  Result.

time_diff({A1,A2,A3}, {B1,B2,B3}) ->
  (B1 - A1) * 1000000 + (B2 - A2) + (B3 - A3) / 1000000.0 .

handle_line(Line, SplitOn) ->
	L = string:tokens(string:strip(Line, both, $n), "	"),
	{Dimensions, Measures} = lists:split(SplitOn, L),
	lists:map(fun(X) -> {I,_} = string:to_integer(X), I end, Measures).

process_file(Filename, Proc) ->
	{ok, File} = file:open(Filename, read),
	process_lines(File, Proc, 0).

process_lines(File, Proc, LineNum) ->
	case io:get_line(File, '') of
		eof -> file:close(File);
		Line ->
			Proc(Line),
			process_lines(File, Proc, LineNum + 1)
	end.

Including Code Directly (no fun())

(I’m just showing the difference.)

process_lines(File, Proc, LineNum) ->
	case io:get_line(File, '') of
		eof -> file:close(File);
		Line ->
			L = string:tokens(string:strip(Line, both, $n), "	"),
			{Dimensions, Measures} = lists:split(10, L),
			lists:map(fun(X) -> {I,_} = string:to_integer(X), I end, Measures),
			process_lines(File, Proc, LineNum + 1)
	end.

Of course, it’s easy to argue that Erlang isn’t the best language for string manipulation. But this part of the application is hardly the bottleneck, so I’m willing to take the bloat in order to take advantage of the concurrency later on.

Next up, I’ll do timing experiments testing if tail recursion speeds anything up.

If you are working with Erlang on Mac OS X and have installed it via Mac Ports, then you might not be running the SMP enabled erlang.

To check, start erlang as

erlang -smp

Argument '-smp' not supported.

sudo port uninstall erlang
sudo port install erlang +smp