Solidsands – Toolchain Qualification for Safety-Critical Systems
Toolchain qualification is the foundation of safety-critical engineering. In the engineering of safety-critical systems — whether for automotive, industrial, aerospace, railway, or medical applications — the validity of the final software product is inextricably linked to the integrity of the tools used to create it. When developers treat compilers and standard libraries as unexplored components, significant vulnerabilities emerge.
In a functional safety context, this black-box approach is no longer acceptable. Functional safety standards explicitly mandate evidence of toolchain qualification. Through its partnership with Solid Sands, Sightsys enables organizations to shift from passive trust to active validation, providing infrastructure for systematic compiler and library qualification tailored to specific use cases.
The Reality of Standards: Certification Levels
Functional safety frameworks establish rigor tiers matching failure severity potential. Compliance demands objective proof and adherence to documented processes.
ISO 26262 governs automotive applications with target safety levels from ASIL A through ASIL D. IEC 61508 addresses industrial contexts with SIL rankings 1–4. EN 50716 applies to railway systems ranging from Basic Integrity to SW-SIL 4. DO-178C covers avionics with DAL classifications E through A. IEC 62304 manages medical device classifications A, B, and C.
For ASIL D (Automotive) or SW-SIL 4 (Railway), the certification process requires that every tool that could introduce an error or fail to detect one must be qualified. A compiler translation error could embed bugs undetectable by conventional verification testing of source code alone.
The Engineering Scale: Quantifying Compiler Integrity
High-integrity certification demands extensive, methodically organized test collections probing language specifications comprehensively. The Solid Sands offering remains entirely compiler-agnostic and compatible with any C or C++ toolchain, employing dual validation techniques:
1. Handwritten Precision (The Foundation)
The SuperTest suite includes approximately 25,000 handwritten test files, representing 2 million lines of code, derived from decades of compiler expertise. Tests deliberately correlate to ISO C and C++ language standards, delivering the documentation traceability auditors require linking code constructs to confirmed behavior.
2. Generated Complexity (The Stress Test)
Sophisticated test generators address non-obvious edge cases:
Arithmetic and Logic: Over 11 million lines of code are dedicated to verifying arithmetic correctness, including complex unsigned overflows and data type behaviors.
Optimization Verification: Compilers are inherently dangerous because they transform your code. Solid Sands provides over 1 million lines of code designed specifically to verify that optimization passes — such as loop unrolling, dead code elimination, and instruction scheduling — do not alter the observable program behavior.
Understanding Structural Coverage and Evidence
For standard libraries (validated via SuperGuard), demonstrating how the test collection exercises library implementation becomes essential. High-assurance settings (SIL 4, ASIL D) frequently require specific metrics preventing untested logic:
Statement Coverage ensures every executable instruction executes at least once. Branch/Decision Coverage confirms all control structures (conditionals, switches) experience both outcomes. Modified Condition/Decision Coverage (MC/DC) represents the high-safety benchmark, demanding proof that each condition within composite decisions independently influences final outcomes. SuperGuard facilitates mapping test execution results against coverage criteria, furnishing objective validation that library implementations achieve full certification.
The ASIC Industry and GCC: A Real-World Necessity
Picture an ASIC manufacturer developing advanced automotive driver assistance semiconductors requiring a complete toolchain. Many depend on GNU Compiler Collection (GCC).
However, GCC is an open-source toolchain. When automotive suppliers must achieve ASIL D on that hardware, they cannot designate standard GCC as “qualified.” The ASIC vendor must perform full toolchain qualification on the particular GCC version, binutils edition, target processor architecture, and compilation parameters. The semiconductor manufacturer applies the SuperTest qualification kit executing requirements-based validation of customized GCC. This allows the chip company to distribute a “Safety Package” to automotive clients, substantially reducing supplier certification demands.
Emerging Robotics: Ensuring Deterministic Safety
Functional safety requirements in robotics accelerate as robots transition from restricted industrial cages into collaborative human environments. Autonomous logistics and surgical robotics contexts define “quality software” through deterministic operation and verifiable safety integrity.
Why Qualified Software is Non-Negotiable in Robotics
Compiler-Induced Bugs: Robotics frequently employs aggressive compiler optimizations meeting real-time processing requirements for sensor data processing. If the compiler contains a subtle bug in its register allocation or loop unrolling logic, the robot might execute a movement command incorrectly. In an autonomous warehouse, a “glitch” in path-planning logic caused by a compiler error is not just a software bug — it is a collision hazard.
The Library Risk: Contemporary robotics leverages open-source C++ libraries for motion algorithms utilizing extensive template structures. Standard testing cannot guarantee consistent behavior across boundary conditions. Applying SuperGuard validates mathematical and logical foundations matching safety-critical kernel assurance levels.
Integration and Determinism: Toolchain qualification of the compiler (GCC or Clang) constructing robot firmware establishes a “Safety Package,” supplying the traceability matrix demonstrating compiler absence as failure source, permitting engineers to implement static checkers that automatically forbid patterns triggering identified compiler anomalies.
Toolchain Qualification in CI/CD Workflows
Qualification is valid only for a specific use case. If you change your compiler flags, switch architectures, or update the compiler version, the qualification state changes. The Sightsys / Solid Sands approach to toolchain qualification guarantees testing conditions precisely mirror production circumstances, employing identical compiler parameters and hardware abstraction layers.
The philosophy avoids pursuing “zero-defect” compilers — practically unachievable — favoring instead a “known-defect” compiler approach. Pinpointing compiler inconsistencies enables developing mitigation strategies, facilitating secure compiler employment for safety-critical development. SuperTest and SuperGuard generate standardized outputs (JUnit XML, CSV, HTML) integrating directly into CI/CD systems, empowering engineers to execute impact analysis comparing new traceability against prior baselines.
Talk to Sightsys
Does your current development process include a formal traceability matrix for your toolchain qualification, or are you looking to implement an RBT strategy for your next project? Contact the Sightsys team to discuss the technical requirements of your specific target architecture and to see a demonstration of how these qualification suites can be integrated into your existing build pipeline.
Read also: AI generated unit testing with Claude Code and Cantata — closing the loop between AI-assisted test creation and qualified coverage.
