Test Coverage Measurement in Practice, or the Mysteries of Bytecode Instrumentation
FrontEndART Ltd. began developing its source code coverage measurement system called TestNavigator in the early 2010s, with its first version offering method-level measurements. This solution served our banking clients well in 2012. However, branch-level coverage has now become the standard, supported by IDEs (e.g., Eclipse) during unit testing. Identifying branches from source code might seem straightforward, but what about on the compiled bytecode level, where compiler optimizations can further complicate our task?
One special challenge during instrumentation is determining block-level measurement points, which can be achieved through analyzing the bytecode level of the source code. Java bytecode sits between source code and machine code in the hierarchy and consists of instructions for the JVM, generated by the compiler from the program. It cannot be executed on its own and cannot be read by the CPU. (This implementation of bytecode is one of the reasons behind Java’s platform independence.) Every bytecode instruction of a method consists of a one-byte opcode and, optionally, operands. Due to the architecture of the JVM, variables are stored in a stack, and bytecode operations are primarily executed on stack elements. Java bytecode uses Labels to handle jumps/switches between blocks, and by iterating through these Labels, we can determine the relevant jump points. And we could go on listing many more fascinating technical challenges. As you can see, determining relevant jump points is far from trivial for us... Hansel and Gretel used breadcrumbs, but we’ll need more serious tools to ensure success.
If you're curious about what this development looks like in practice, and if joining the development of such a system and working in a fantastic SCRUM team sounds appealing to you, then don’t miss our meetup! The development of the business part of our new TestNavigator system begins in 2022, and we’re looking for enthusiastic applicants. We're looking for candidates interested in challenges, building large, robust systems, Big Data (coverage data from many versions of many systems), AI (test selection), and fresh, modern technologies (Spring, Spring Boot, Angular, Kubernetes, Docker, ...) as well as working according to a streamlined methodology where you deliver stories in 2-week sprints.
We’re looking forward to seeing you on 2021.12.08 at 6:00 PM at the following link: meet.google.com/ejb-wtur-mod