A NullPointerException came out of a line with 3 dots in it.
def zipOf(order: Order): String = order.customer.address.zipThe message was empty. The stack trace named the method and the line, which I already knew from the exception being thrown at all. Any of the 3 could have been null and the exception looks the same either way.
2 things I wanted to settle. Whether Option really takes this class of error away, which everyone around me says and I half believe. And what it costs, because I have been repeating that Option allocates on every lookup and I have never measured that once. Everything below is Scala 2.9.1 on jdk 7.
complete: 00-001
no address: NullPointerException, message=[null] at zipOf line 8
no customer: NullPointerException, message=[null] at zipOf line 8
no order: NullPointerException, message=[null] at zipOf line 83 different defects arrive as one line of output repeated 3 times. To find out which link was broken I had to go back and split the chain up by hand. The argument for Option is about this class of error and it is fair.
The same path with the fields typed as Option:
def zipOf(order: Option[OrderO]): Option[String] =
order.flatMap(_.customer).flatMap(_.address).map(_.zip)complete: Some(00-001)
no address: None
no customer: None
no order: NoneNothing is thrown. The 3 defects are still there and the program still cannot produce a zip, but the absence now travels as a value and arrives where I decided to handle it. The compiler also stops me reading the zip out without saying what happens when it is missing.
That holds for code that goes through map and flatMap. get is still on the type:
none.get: NoSuchElementException, message=[None.get]Same crash under a different class name. The message says None.get, which beats an empty one, but the guarantee only covers the calls I choose to make.
Where it leaks
Most of the code around this is java and java returns null. So the wrapper gets written:
def zipOf(id: String): Option[String] = Some(Legacy.zip(id))That is what comes out when I write it quickly. It compiles and it is wrong:
zipOf("c-1") = Some(null)
zipOf("c-1").isDefined = true
zipOf("c-1").getOrElse("none") = null
lengthOfZip("c-1") ! java.lang.NullPointerException
Interop$$anonfun$lengthOfZip$1.apply(Interop.scala:9) <- Interop$$anonfun$lengthOfZip$1.apply(Interop.scala:9) <- scala.Option.map(Option.scala:133)Some(null) is a value that reports itself as present and holds nothing. isDefined is true, getOrElse hands back the null it was there to keep away from me and the NPE moves inside the lambda in map. That is a worse place to meet it than the original chain, because now the stack goes through library code. Writing Option(...) instead of Some(...) fixes all of it, since Option.apply checks for null and gives back None.
The second leak I did not expect. Option is a reference like any other, so it can be null itself:
val o: Option[String] = null = null
o.isDefined ! java.lang.NullPointerException
Interop$.main(Interop.scala:25) <- Interop.main(Interop.scala)
o.getOrElse("none") ! java.lang.NullPointerException
Interop$.main(Interop.scala:27) <- Interop.main(Interop.scala)That compiles without a warning. I ran the compiler again with -deprecation in case I had missed one. The only 2 warnings in the whole build are about an Integer alias in a different file. An uninitialised field of type Option[String] holds null rather than None. Every call on it fails the old way.
2.9.1 has a flag aimed at exactly this. -Xcheck-null warns on the selection of a nullable reference. On these 2 small files it produced 54 warnings. 3 of them are the real dereferences on line 8. It also flags o.isDefined on the null Option, which is the case I had just been surprised by. The rest are things like an arrow on a string literal and label.+, because a string concatenation is a selection on a reference too. 17 of the 54 are concatenations and 8 are that arrow. I did not find a way to read the 3 I wanted out of the other 51.
What I thought it cost
I have been repeating that Option costs an allocation on every lookup. I had never measured it. A table of 100000 entries, 20 million lookups per loop, 3 warmups then 5 timed runs with the median printed. Same 2.9.1 and jdk 7, heap pinned with -Xms256m -Xmx256m so that compressed oops stay on:
1 java null median 192 ms runs 192,192,176,201,182 15.9802 bytes/lookup
2 scala Option median 436 ms runs 453,447,436,430,412 15.9795 bytes/lookup
3 java + Option() median 188 ms runs 184,188,197,181,195 15.9795 bytes/lookup
4 scala apply, no Option median 434 ms runs 434,438,424,432,466 15.9795 bytes/lookupLoops 1 and 3 run against the same java.util.HashMap. The only difference is that loop 3 wraps every result in Option(...). 188 against 192. Across the 3 runs I kept, the 2 sit at 181 to 189 against 186 to 199. Wrapping costs nothing I can see. The interesting gap is loop 3 against loop 2, which is 248 ms between 2 loops that both build an Option per lookup. What separates those 2 is scala.collection.mutable.HashMap against java.util.HashMap.
The allocation column is the part I got wrong. Loops 2, 3 and 4 all allocate 15.9795 bytes per lookup and loop 1 sits a rounding hair away at 15.9802. That number is the key rather than the Some. The key i % SIZE is boxed into an Integer. The cache covers -128 to 127 by default while the keys run from 0 to 99999. So 128 lookups in every 100000 come out of the cache and the rest allocate 16 bytes each. That is 16 × (1 − 128/100000) or 15.97952.
Loop 4 was meant to be the control. sm(k) returns the value directly and the word Option does not appear in it. It came out as slow as loop 2, a little slower in 2 of the 3 runs. I read that as confirmation until I turned escape analysis off:
1 java null median 195 ms runs 192,189,195,195,210 15.9802 bytes/lookup
2 scala Option median 544 ms runs 540,552,536,562,544 31.9795 bytes/lookup
3 java + Option() median 229 ms runs 209,229,239,230,227 31.9795 bytes/lookup
4 scala apply, no Option median 535 ms runs 495,545,553,535,518 31.9795 bytes/lookupLoop 1 holds at 15.9802 and the other 3 gain exactly 16 bytes. Loop 4 gains them too, so there is a Some in it after all: apply calls get, get builds the Some and apply unwraps it and drops it. My control loop contained the thing I was controlling for. I only noticed because the allocation counter disagreed with the source I had written.
16 bytes is one Some, a 12 byte header plus one 4 byte reference under compressed oops. With escape analysis on, which is the default, the jit works out that it never leaves the method and skips the allocation. Loop 3 goes from 188 ms to 229 when I take that away. The object is real and it costs time when the jit cannot remove it. It just never gets built.
The same block settles the other question better than my first pairing did. With allocation forced on both, loops 2 and 3 build the same number of Some objects and allocate the same 31.9795 bytes per lookup. They are still 315 ms apart. Whatever that gap is, it is not the Option.
What I did not check
Where those 248 ms go. I measured that it is not the Option and stopped there, so scala.collection.mutable.HashMap is still on my list.
Whether any of this survives outside a tight loop. 20 million lookups with nothing else running is a friendly case for escape analysis. In a request handler with a stack of frames above it I do not know that the Some stays out of the heap. I did not build that test.