How Parsing is Done

On my last post, I talked about the Comparing Test Results project that I am currently working on. In that post, it also mentioned parsing of logs to extract useful information. So far this has already been implemented and tested with data from Fennec builds. The intention of this post is to explain how the parsing is done so it can be a reference for improvements that may need to be made to extend this project to other builds.

There are two parts to the parsing. The first part is to look at the entire log and to find metadata used to characterize and identify a set of data. Here the code is looking for: the build ID, the product, the operating system, and the type of tests that were run.

1. To get the build ID, the code is looking for the line that contains the string “BuildID=” such as this “BuildID=20090715053040”. The information to be extracted will then be whatever after the character ‘=’.
2. Similarly for the product, the code is looking for the string ”Name=”.
3. For the OS and test type, the code is looking for the line that contains the string “tinderbox: build: ” such as this “tinderbox: build: Maemo mozilla-central crashtest”. The OS will be taken from the first word after the string “tinderbox: build:” and the test type will be taken from the last word.

The second part is to parse the log per line to find lines that contain the output of a test run. On each line, if it’s an output of a test run, it is looking for: the result of the test (PASS/FAIL/TODO), the location of the test code (normally the path to the test source file), and if any the message that may describe the intent of the test.

1. To determine if a line is an output of a test the code is looking for either of these strings: “TEST-PASS”, “TEST-FAIL”, “TEST-UNEXPECTED-FAIL”, “TEST-TIMEOUT”, “TEST-KNOWN-FAIL”.
2. The line is then split into three sections as separated by the divider ‘|’. The first section will determine the code will go into either of these conditions: the test passed, the test is marked as TODO, or the test failed.
3. The second section is simply taken as the location of the test code. Since only the relative path is needed, the beginning is striped as it appears for ‘reftest’ and ‘xpcshell’.
4. The third section is taken as the message, if any, from the test. It is taken if the test did not pass.
5. The code also increments the count of the number of passes, failures, and todos.
6. As a result, in any of the three conditions, the code will see if it is a new test. If it is, it is added into the data structure with a count of 1 and if it is not a test that passes, its message is added as well. Otherwise, it will update the count of an existing test and if it is not a test that passes, the message is appended and separated by a coma.

Everything mentioned is not final, still continuously being improved, and is open to suggestions and extensions. The post will be updated as the code is updated.

see more Fail Blog

Working on Comparing Test Results project

What? For two weeks now I have been working on a new project. I decided to work on this get exposure to other areas at Mozilla. Especially for me this particular project would get me to work on tools such as python and couchdb that I have never used before so that opportunity also got me excited. A description for the project can be found here on this Wiki page.

Why? In short the idea is to make test results as generated into the build logs easier to view and to get it into some representations that would be more meaningful for people to monitor and compare these results.

How? One method that we are going with is to parse the more meaningful data from the build logs, as needed for information as well as identification, into a persistent storage which can then be queried for different purposes. So there are two parts to this process. One part is parsing the logs and saving the data into a DB. The other part is about creating aggregations and reports of these data and presenting it for different monitoring scenarios.

Seriously how? Specifically we have a python script that will parse the logs from different builds and extract metadata such as: build ID, product, OS, and test type and the test data including the status of a test run, the location of test code, and the message. The script will need to be hooked and run automatically.

The resulting data will be formatted into a dictionary like or JSON structure. Since CouchDB is used for data storage, sending data to CouchDB in this format will automatically save the data into documents. As documents in CouchDB, data is queried through Map and Reduce procedures which are stored as ‘Views’ documents. Users will then view the results generated for CouchDB’s views through web pages.

What now? So far parsing data and saving it to CouchDB are working well using data from Fennec builds. Basic views have been generated on CouchDB’s Futon. The remaining pieces will be to get the script included into the build process, to have it read from actual logs, to figure out scenarios and generate more complex views needed, and to have server generated web pages to fetch reports from the data storage. Working in Progress.

see more Lolcats and funny pictures