crash testing, 1 import failure

moggi described here our crash testing infrastructure. Basically we have a document horde mostly populated through get-bugzilla-attachments-by-mimetype which downloads all the attachments from our bugzilla (and a whole bunch of other bugzillas) that are in formats which LibreOffice can open. We then import the lot of them with the above testing harness looking for crashes and aborts. A new report tends to appear every 1-3 days.

These documents are filed in bugzillas. In many cases they were filed specifically because they were causing some sort of trouble for someone, so there are a lot of hostile documents in there.

We currently have 76221 documents in the horde, the most recent run reports one, one single solitary failure (an assert in a .doc about invalid positioning of a cross-reference bookmark in a document with change-tracking enabled).

Here's a graph over time of our failure rate. Where failure is either a straight forward crash, or a triggered assert. The builds are dbgutil, extra-debugging, extra-checking, assert-enabled, exception-specification-enabled builds.

You get temporary local peaks every now and then when either a new assert is added or someone introduces a bug. We have two purposes here, immediate regression discovery and historic bug removal.

We also have export crash testing, where the numbers aren't as shiny yet, but are on an equivalent downward trend. More on that in a while when we figure out how to fix this final import stinker.