Reference: Sommerville, Software Engineering, 10 ed., Chapter 12

Safety is a property of a system that reflects the system's ability to operate, normally or abnormally, without danger of causing human injury or death and without damage to the system's environment. It is important to consider software safety as most devices whose failure is critical now incorporate software-based control systems.

Safety and reliability are related but distinct. Reliability is concerned with conformance to a given specification and delivery of service. Safety is concerned with ensuring system cannot cause damage irrespective of whether or not it conforms to its specification. System reliability is essential for safety but is not enough.

Reliable systems can be unsafe:

• Dormant faults in a system can remain undetected for many years and only rarely arise.
• Specification errors: the system can behave as specified but still cause an accident.
• Hardware failures could generate spurious inputs that are hard to anticipate in the specification.
• Context-sensitive commands (the right command at the wrong time) are often the result of operator error.

In safety-critical systems it is essential that system operation is always safe i.e. the system should never cause damage to people or the system's environment. Examples: control and monitoring systems in aircraft, process control systems in chemical manufacture, automobile control systems such as braking and engine management systems.

Two levels of safety criticality:

• Primary safety-critical systems: embedded software systems whose failure can cause the associated hardware to fail and directly threaten people.
• Secondary safety-critical systems: systems whose failure results in faults in other (socio-technical) systems, which can then have safety consequences.

Safety terminology

Safety achievement strategies:

Hazard avoidance

The system is designed so that some classes of hazard simply cannot arise.

Hazard detection and removal

The system is designed so that hazards are detected and removed before they result in an accident.

Damage limitation

The system includes protection features that minimize the damage that may result from an accident.

Accidents in complex systems rarely have a single cause as these systems are designed to be resilient to a single point of failure. Almost all accidents are a result of combinations of malfunctions rather than single failures. It is probably the case that anticipating all problem combinations, especially, in software controlled systems is impossible so achieving complete safety is impossible. However, accidents are inevitable.

The goal of safety requirements engineering is to identify protection requirements that ensure that system failures do not cause injury or death or environmental damage. Safety requirements may be 'shall not' requirements i.e. they define situations and events that should never occur. Functional safety requirements define: checking and recovery features that should be included in a system, and features that provide protection against system failures and external attacks.

Hazard-driven analysis:

Hazard identification

Identify the hazards that may threaten the system. Hazard identification may be based on different types of hazard: physical, electrical, biological, service failure, etc.

Hazard assessment

The process is concerned with understanding the likelihood that a risk will arise and the potential consequences if an accident or incident should occur. Risks may be categorized as: intolerable (must never arise or result in an accident), as low as reasonably practical - ALARP (must minimize the possibility of risk given cost and schedule constraints), and acceptable (the consequences of the risk are acceptable and no extra costs should be incurred to reduce hazard probability).

The acceptability of a risk is determined by human, social, and political considerations. In most societies, the boundaries between the regions are pushed upwards with time i.e. society is less willing to accept risk (e.g., the costs of cleaning up pollution may be less than the costs of preventing it but this may not be socially acceptable). Risk assessment is subjective.

Hazard assessment process: for each identified hazard, assess hazard probability, accident severity, estimated risk, acceptability.

Hazard analysis

Concerned with discovering the root causes of risks in a particular system. Techniques have been mostly derived from safety-critical systems and can be: inductive, bottom-up: start with a proposed system failure and assess the hazards that could arise from that failure; and deductive, top-down: start with a hazard and deduce what the causes of this could be.

Fault-tree analysis is a deductive top-down technique.:

• Put the risk or hazard at the root of the tree and identify the system states that could lead to that hazard.
• Where appropriate, link these with 'and' or 'or' conditions.
• A goal should be to minimize the number of single causes of system failure.

Risk reduction

The aim of this process is to identify dependability requirements that specify how the risks should be managed and ensure that accidents/incidents do not arise. Risk reduction strategies: hazard avoidance; hazard detection and removal; damage limitation.

Safety engineering processes are based on reliability engineering processes. Regulators may require evidence that safety engineering processes have been used in system development.

Agile methods are not usually used for safety-critical systems engineering. Extensive process and product documentation is needed for system regulation, which contradicts the focus in agile methods on the software itself. A detailed safety analysis of a complete system specification is important, which contradicts the interleaved development of a system specification and program. However, some agile techniques such as test-driven development may be used.

Process assurance involves defining a dependable process and ensuring that this process is followed during the system development. Process assurance focuses on:

• Do we have the right processes? Are the processes appropriate for the level of dependability required. Should include requirements management, change management, reviews and inspections, etc.
• Are we doing the processes right? Have these processes been followed by the development team.

Process assurance is important for safety-critical systems development: accidents are rare events so testing may not find all problems; safety requirements are sometimes 'shall not' requirements so cannot be demonstrated through testing. Safety assurance activities may be included in the software process that record the analyses that have been carried out and the people responsible for these.

Safety-related process activities:

• Creation of a hazard logging and monitoring system;
• Appointment of project safety engineers who have explicit responsibility for system safety;
• Extensive use of safety reviews;
• Creation of a safety certification system where the safety of critical components is formally certified;
• Detailed configuration management.

Formal methods can be used when a mathematical specification of the system is produced. They are the ultimate static verification technique that may be used at different stages in the development process. A formal specification may be developed and mathematically analyzed for consistency. This helps discover specification errors and omissions. Formal arguments that a program conforms to its mathematical specification may be developed. This is effective in discovering programming and design errors.

Advantages of formal methods

Producing a mathematical specification requires a detailed analysis of the requirements and this is likely to uncover errors. Concurrent systems can be analyzed to discover race conditions that might lead to deadlock. Testing for such problems is very difficult. They can detect implementation errors before testing when the program is analyzed alongside the specification.

Disadvantages of formal methods

Require specialized notations that cannot be understood by domain experts. Very expensive to develop a specification and even more expensive to show that a program meets that specification. Proofs may contain errors. It may be possible to reach the same level of confidence in a program more cheaply using other V & V techniques.

Model checking involves creating an extended finite state model of a system and, using a specialized system (a model checker), checking that model for errors. The model checker explores all possible paths through the model and checks that a user-specified property is valid for each path. Model checking is particularly valuable for verifying concurrent systems, which are hard to test. Although model checking is computationally very expensive, it is now practical to use it in the verification of small to medium sized critical systems.

Static program analysis uses software tools for source text processing. They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team. They are very effective as an aid to inspections - they are a supplement to but not a replacement for inspections.

Three levels of static analysis:

Characteristic error checking

The static analyzer can check for patterns in the code that are characteristic of errors made by programmers using a particular language.

User-defined error checking

Users of a programming language define error patterns, thus extending the types of error that can be detected. This allows specific rules that apply to a program to be checked.

Assertion checking

Developers include formal assertions in their program and relationships that must hold. The static analyzer symbolically executes the code and highlights potential problems.

Static analysis is particularly valuable when a language such as C is used which has weak typing and hence many errors are undetected by the compiler. Particularly valuable for security checking - the static analyzer can discover areas of vulnerability such as buffer overflows or unchecked inputs. Static analysis is now routinely used in the development of many safety and security critical systems.

Safety engineering in software focuses on identifying, analyzing, and mitigating hazards that can lead to harmful outcomes - for users, the environment, or even other systems. It seeks to ensure that failures do not lead to unacceptable risk. Large Language Models (LLMs) can assist with brainstorming hazards, summarizing safety concepts, and drafting analysis documents, but they do not themselves guarantee safety or replace the rigorous analysis, evidence, and validation required to build safe systems. In safety-critical domains, human judgment, domain expertise, and formal reasoning remain essential.

Where LLMs help in safety engineering? Below are common uses and what to watch out for:

Brainstorming hazards and failure scenarios

LLMs can generate lists of potential hazards, unsafe conditions, and failure scenarios from a textual system description. This can jumpstart hazard identification. The risk is that the model may produce generic hazards or omit domain-specific conditions that only domain experts know, and it may present manufactured hazards with unwarranted confidence.

Drafting safety requirement text

LLMs can draft candidate safety requirements, constraints, and conditions to check (e.g., "system shall fail gracefully when X"). This helps with early documentation. The risk is that drafted safety requirements may not be measurable or rooted in actual risk analysis, and they can lull engineers into thinking safety is "handled."

Summarizing analysis (e.g., FMEA, FTA)

LLMs can help summarize results of failure mode and effect analysis (FMEA), fault tree analysis (FTA), or other safety assessments into prose. This supports communication. The risk is that such summaries can misrepresent severity, likelihood, or conditions - especially if they are based on incomplete prompts or missing context.

Safety checklist generation

LLMs can produce checklists of safety considerations (e.g., boundary conditions, operator errors, environmental impacts). These can serve as reminders. The risk is that checklists are generic, may not reflect actual system usage, and can create false reassurance if relied on too heavily without validation.

Integrating with testing and verification plans

LLMs can suggest connections between identified hazards and test or verification cases. This supports planning. The risk is that the model lacks real evidence about system behavior; it cannot measure whether tests truly address hazards or prove properties.

Bottom line: LLMs can assist in *generating ideas, drafting text, and reminding teams of common safety concerns*, but safety engineering is fundamentally about validated reasoning, evidence, and understanding of how a system behaves under varied conditions. AI output must be critically reviewed and tied to data, formal analysis, and expert judgment.

A simple guided example

Scenario:

You want to draft a starting set of hazards for an automated door control system (e.g., a door that must detect obstruction, respond to emergency signals, and correct lock/unlock behavior).

Prompt to an LLM:

"List possible hazards for an automated door control system, including what could go wrong, potential causes, and consequences for users."

Typical Output (Excerpt):

"Hazard: door fails to detect obstruction; Cause: sensor error; Consequence: injury. Hazard: door won't open on emergency signal; Cause: logic fault; Consequence: trapped users..."

Why this helps:

The model produces a structured starting point that names probable hazards and organizes them into cause/consequence pairs that the team can refine.

Why this can mislead:

The model may miss domain-specific hazards (e.g., specific sensor failure modes in the environment, interactions with other building systems) and can present plausible but unsupported causes or consequences. It has no real understanding of how the system operates under stress or failure.

Discussion point:

Which hazards require domain data or simulation to validate? Which require input from actual users or subject matter experts? How would you prioritize safety analysis tasks based on this draft?

Key Takeaways

• LLMs can help generate hazard candidates, draft safety requirements, and summarize analyses, but they do not replace formal safety engineering methods.
• Safety engineering requires precise definitions, measurable conditions, evidence, and expert validation.
• Use AI output as a starting point for deeper, evidence-based safety work, not as a definitive artifact.