A Guide to online information about:

System Safety

by

As part of my former Day Job, in August of 1999, I attended the workshop "Programmable Electronic Mining Systems: An Introduction to Safety".  Since the software I write for a living controls large (tens of tons) mobile equipment, I wanted to know more about how to implement my systems in a safer manor for all concerned.  Sometimes relying on a simple Watchdog Timer is not enough.  After all, I don't want to be on the witness stand explaining to lawyers why my system went off and killed and maimed people.  How would you feel about being is such a position? 

So, this month's Resource Page looks at "System Safety".  As was pointed out in the Workshop, a common misconception is to think of the problem as Software Safety. Software does not exist in isolation from the rest of the system.  The system as a whole must be designed to be safe.

Mr. Sammarco recommended the following Books:

"Safer C" by Less Hatton.
"Safety-Critical Computer Systems" by Neil Story.
"Safeware" by Nancy Leveson.
"System Safety: Hazop and Software Hazop" by Felix Redmill. 

From the Workshop:

"Functional and operational safety starts at the system level. Safety cannot be assured if efforts are focused only on software. The software can be totally free of 'bugs' and employ numerous safety features, yet the equipment can be unsafe because of how the software and all the other parts interact in the system. In other words, the sum can be less safe than the individual parts!

Thus, a system approach is needed. How does one address the safety of this system? Is it done by making the system more reliable, employing redundancy, or conducting extensive testing? All of these are necessary but not sufficient to ensure
safety! Making a system more reliable is not sufficient if the system has unsafe functions. What you could have is a system that reliably functions to cause unsafe conditions!  Employing redundancy is not sufficient if both redundant parts are not safe.

Testing alone is not sufficient for safety. Studies show that testing doesn't find all the 'bugs' and some systems are too complex to test every condition. 

The key to safety is to 'design in' safety early in the design by looking at the entire system, identifying hazards, designing to eliminate or reduce hazards, and doing this approach over the system life cycle!

While we'd all love to place 100% of the blame on the Boss Person Hierarchy, it actually means the management of the project, not to say bosses don't contribute to the problem in many forms.

Most causes of system faults are created before the first line of code is ever written or first schematic is ever drawn. The errors are caused by not understanding the requirements of the system.

One simple way of understanding the requirements is to ask yourself how you would test this requirement. If you can't specify a test that can clearly show that the requirement has been meet, then the requirement or the understanding needs refinement.

The earlier in the design cycle that the requirements are clearly understood the lower the cost of any needed changes.

(i)    each safety-related function to be implemented?

(ii)   the information to be given to the operator at any time?

(iii)  the required action on each operator command including illegal or unexpected commands?

(iv)  the communications requirements between the embedded system and other equipment?

(v)    the initial states for all internal variables and external interfaces?

[A big help here is to use that nitt-pickiest of all programs, Lint, on the variable issue.]

(vii) each modes and the initiators of mode transition? (e.g. start-up, normal operation, shutdown)

[If you go from the Auto Mode, to the Maintenance Mode, then back to the Auto Mode, starting over where you left off in the Auto Mode, could be deadly in some cases.]

[What happens when you have channel-to-channel leakage in your A/D because the input that you are not reading is driving the entire mux into saturation?]

(x)   the constraints put on the software by the hardware? (e.g. speed, memory size, word length)

(xi)   internal self-checks to be carried out and the action on detection of a failure?

Does the software contain adequate error detection facilities allied to error containment, recovery. or safe shutdown procedures?

Are safety critical areas of the software identified?

"Design for Assessment"

• Does design implement safety functions and attributes?
• Are design criteria satisfied?
• Is the design feasible?
• Are safety arguments credible?
• Is the approach cost-effective?
• • To minimize costs and risks:
  • • use a simple design (eases analysis)
    • avoid novelty (use established designs)
    • ensure supporting evidence is readily available

An easy place to start designing a safe system actually has little to do with any style of coding or design.  It is a method of documentation/source code archiving, which is called by several names.  The two most common are Version Control and Configuration Management.  The hardest part about this is establishing the discipline to use it.  Once you start using it, you'll find you become dependent on it and wonder how you got along without it.  I know using Version Control has saved me many hassles a few times.

Susan Dart once wrote : "The goals of using CM (Configuration Management) are to ensure the integrity of a product and to make its evolution more manageable. Although there is overhead involved in using CM, it is generally agreed that the consequences of not using CM can lead to many problems and inefficiencies. The overhead of using CM relates to time, resources, and the effects on other aspects of the software lifecycle."

The Concurrent Versions System (CVS) provides network-transparent source control for groups of developers.  CVS, the most popular version control system in the Open Source community, is available on many platforms.
CVS FAQ !  Open Source Version Control Software.

Features:

• Maintains a history of all changes made to each directory tree it manages.
• Provides hooks to support process control and change control.
• Provides reliable access to its directory trees from remote hosts using Internet protocols.
• Supports parallel development allowing more than one developer to work on the same sources at the same time. 

Considered the main manual for CVS, Version Management with CVS (170 pages) explains underlying concepts, describes how to use all documented features of CVS, and gives examples of command use.

CVS is normally a command line program, but there are versions available for Windows and Mac's.

Online manual for CVS.  The manual gives a explanation of what CVS is, and more importantly, what CVS is not.

"CVS is a version control system. Using it, you can record the history of your source files.

For example, bugs sometimes creep in when software is modified, and you might not detect the bug until a long time after you make the modification. With CVS, you can easily retrieve old versions to see exactly which change caused the bug. This can be a big help. 

You could, of course, save every version of every file you have ever created. This would however waste an enormous amount of disk space. CVS stores all the versions of a file in a single file in a clever way that only stores the differences between versions.

CVS also helps you if you are part of a group of people working on the same project. It is all too easy to overwrite each others' changes unless you are extremely careful. Some editors, like GNU Emacs, try to make sure that the same file is never modified by two people at the same time. Unfortunately, if someone is using another editor, that safeguard will not work. CVS solves this problem by insulating the different developers from each other. Every developer works in his own directory, and CVS merges the work when each developer is done.

CVS is not a substitute for management.  Your managers and project leaders are expected to talk to you frequently enough to make certain you are aware of schedules, merge points, branch names and release dates. If they don't, CVS can't help. CVS is an instrument for making sources dance to your tune. But you are the piper and the composer. No instrument plays itself or writes its own music."

RCS is a version control system. It offers a basic level of functionality (for example, it operates on one file at a time). We generally recommend a more powerful system, such as CVS, even for beginners.

RCS has been widely ported and reimplemented. The free version of RCS is often called GNU RCS to distinguish it from the non-free implementations.

"RCS is [analogous to using] assembly language, while CVS is [like using] Pascal."

Practical Software Configuration Management by Tim Mikkelsen and Suzanne Pherigo, 1997, gets individuals and small teams started with configuration management. The book covers basic RCS usage and also discusses the larger issues that are illustrated by the RCS examples.

---

There are several commercial companies offering Configuration Management tools:

The GUI (graphical) TLIB for Windows supports long filenames under Windows-95/98 and Windows-NT.

I have used TLIB and always been happy with it.
They think enough of their product to put links to their competitors sites on their own site.

   Do you think they'll put links to Burton Systems Software on their Web sites?:

   www.intersolv.com
Intersolv/Merant PVCS

   www.mks.com
Mortice Kern Source Integrity (formerly MKS RCS)

msdn.microsoft.com or shop.microsoft.com
Microsoft Visual SourceSafe

[I've used this product as well, and personally, I like TLIB better.  TLIB never left me wondering, "Why did it do that?."]

   www.rational.com or www.atria.com
Rational/Atria Clearcase

TLIB Conversion Tool are now available.

---

The short CVS overview and the longer CVS datasheet lists CVS features.

Read a description of the complete SourceForge package available free to opensource developers.

Who are we? What are we doing? Why are we doing it?  There is too much information about this project to fit in this introductory page. You should really take the time to visit our Frequently Asked Questions.

Robert,

Per our discussion at the NIOSH/MSHA conference, here is the URL to our Division of Small Manufacturers Administration (DSMA).

http://www.fda.gov/cdrh/dsma/dsmamain.html

The FDA's main web page is at: http://www.fda.gov

Enjoy :)

[The PowerPoint Windows system crashed during Mr. Jones presentation and he said "This is why we never allow any Microsoft products to be used in equipment."  He did say later they used NT in some products, "but not where it could hurt anyone if it failed."  If I where to start my own business, I'd give serious consideration to building it around FDA guidelines, at least as far as documentation goes.]

"Nancy Leveson Professor of Aeronautics and Astronautics started a new area of research, software safety, which is concerned with the problems of building software for real-time systems where failures can result in loss of life or property. One advantage of this topic is that nobody questions its goals, except for a few misanthropes (who don't matter anyway). She and her students produced a formal requirements specification for TCAS II, a real collision-avoidance system required on all commercial aircraft in U.S. airspace. One of the lessons she learned from this project is never to do anything like it again. The FAA was pleased with it though, and adopted it as their official specification. She claims you should not read anything into the fact she has been taking the train a lot lately."

The System and Software Safety Research Project is also working on modeling and analysis of various aerospace and transportation systems. Subtopics in this research area include modeling and analysis of safety, system and software requirements specification, safe software design, software fault tolerance, and verification and validation of safety.

Dr. Leveson and some students (both former and present) started a company to commercialize their ideas. Safeware Engineering Corporation began as a partnership in 1991 to perform applied research and development for real-time, safety-critical systems.  The goal is to act as a technology transfer conduit from university research to industrial research and practice.

Software has become the driving force behind most new technologies. But, the engineering of software is becoming increasingly complicated. A software engineer must balance a variety of competing factors, including functionality, quality, performance, safety, usability, time to market, and cost. Moreover, the size of software systems being built is rapidly growing.

The Software Engineering Research Laboratory (SERL) in the Department of Computer Science at the University of Colorado at Boulder is pursuing the discovery of principles and the development of technologies to support the engineering of large, complex software systems. The challenging targets for this work are organizations and software systems operating in the wide-area, heterogeneous, distributed, and decentralized context of wide-area networks such as the Internet.

Broadly speaking, there are six themes that underlie their research:

The purpose of Technology Validation Assurance (TVA) is to provide a short-cycle, low-cost approach to building in quality through design optimization, and quality and reliability analysis for critical electronic hardware. Concurrent engineering, theoretical analysis, and empirical data are used to produce information about a technology's mechanical, thermal, and electrical limitations and its most likely failure mechanisms.  If you want reliability, you've come to the right place!

The Handbook now has more than 650 pages and contains a compilation of 101 analysis techniques and methodologies, plus other related information for the seasoned safety veteran as well as the new practitioner. New material includes 11 added techniques, an updated software system safety section, and a new section on the application of fuzzy and hybrid mathematics to safety analysis. A glossary is now included. The handbook is in an 81/2 X 11 loose-leaf binder so that additions, changes, and your own personal notes can be easily accommodated. 

Since I played a small part in the production of  "MicroC/OS-II The Real-Time Kernel; A complete portable, ROMable scalable preemptive RTOS", ISBN: 0-87930-543-6, Copyright 1999,  (see page XIX of the book) I found particular interest that µC/OS-II is being readied for Safety Critical Systems.

Work is under way to generate a complete validation suite for µC/OS-II.  The validation suite provides all of the documentation necessary to prove that µC/OS-II is suitable for use in safety critical systems common to aviation (FAA standard DO-178B level B) and medical products (FDA/CDRH "General Principles of Software Validation, Draft Guidance, Version 1.1").  The validation suite was produced by Validated Software Corp http://www.validatedsoftware.com. When the suite is formally released, µC/OS-II will have been certified for use in an avionics device to FAA standard DO-178B level B.  A follow-on package in the 1st quarter of 2000 will bring µC/OS-II and the suite up to full DO-178B level A certifiability.

The Raskasky Group provides Computer System Validation Support Services and Seminars to the Pharmaceutical Industry.  They provide online resources and pointers to important internet sites.

The Real-Time Software Engineering Branch develops ground data systems for integration and test and on-orbit operations of Earth and space science missions. Branch personnel participate in teams with flight projects, principal investigators, AETD centers, and other organizations to develop integrated hardware and software systems for real-time mission support. The system functionality includes spacecraft, instrument, ground system monitoring and control, launch and tracking services, and data display and analysis.

Recent advances in composite super-pressure balloon materials greatly enhanced the prospects for long duration balloon flights on Earth, as well as possible use for planetary exploration. NASA is embarking on the development of technologies to support extended balloon missions lasting up to 100 days (~5 circumnavigations of the globe) above 99.9% of Earth's atmosphere.

The purpose of the ULDB flight software effort is to process, monitor, and control data received and collected on the airborne instrumentation package.  The flight software will facilitate all communications with the instruments on board and the ground mthrough continuous line of sight and over the horizon communications.

The goal of this project is to develop formal-method based system design tools with an emphasis on reliability, safety, and security. The tools will utilize models, generic in construct but domain specific for each application. This project is a joint effort with Sandia National Laboratories.

Objectives: 

The primary objectives of this project involve increasing the reliability, safety, and security of systems by performing analysis at design time. Analyzing these aspects of a system can greatly reduce the time required to produce a safe, secure, and reliable system. The analyses will rely heavily on formal verification and validation methods. 

The specific analysis tools that are being identified and developed include: verification of the consistency of requirements, validation of design models versus requirement models, simulation of system behavior (including forward and backward system execution), hazard analysis, safety and reliability analysis (using proprietary tools), and automatic fault-tree generation.

Technical Overview: 

The modeling approach is based on multiple-aspect structural and behavioral models. The Graphical Modeling Editor (GME) is used to implement the modeling environment. System behavior is described in a discrete state space using hierarchical, parallel state machines. System structure and component failures are modeled in the structural aspect. Relationships between the structure and behavioral aspects are explicitly modeled. This ensures the relationship between physical failure modes and system behavior are explicitly represented. 

When the models are interpreted, Ordered Binary Decision Diagrams (OBDDs) are used to symbolically represent the system. Using a symbolic representation allows exhaustive search techniques to be used on the system without the need to enumerate the discrete state space. OBDDs have been used to represent and analyze systems with 10^100 unique, discrete states. OBDDs allow for the analysis of "real" systems.

The State Space Analysis Tool (SSAT) was designed to provide model analysis capabilities. SSAT converts from the GME models to a symbolic (OBDD) representation of the system. Then, the SSAT allows the user to validate their models through simulation, reachability analysis, loop detection, and deterministic analysis. Once the models have been validated, the user can use the SSAT to generate safety and reliability information for further analysis. A tool developed at Sandia, WinR, is used to analyze the safety and reliability information SSAT generates.

"Multi-Domain Surety Modeling and Analysis for High Assurance Systems," Proceedings of the Engineering of Computer Based Systems Conference, Nashville, TN, March 1999.

"An Integrated Multi-Domain Analysis Environment for High Consequence Systems," Proceedings of the 1998 ASME Design Engineering Technical Conference / Computers in Engineering, Atlanta, GA, September 1998.

"Integrated Analysis Environment for High Impact Systems," Proceedings of the Engineering of Computer Based Systems Conference, Jerusalem, Israel, April 1998.

"Formal-Methods" might deserve their own Resource Page, but in the meantime, the work of Associate Professor Steven D. Johnson
at the Computer Science Department, Indiana University, would be a good place for you to start.  His information is now an official sub-collection of the World Library for Formal Methods.

"Formal Methods: < mathematics, specification> Mathematically based techniques for the specification, development, and verification of software and hardware systems." from --- Free On-Line Dictionary of Computing.

Phyllis Frankl is an Associate Professor in the Department of Computer and Information Science at Polytechnic University, in Brooklyn, NY. His main research interests are in the area of software testing, including development of software testing tools and comparing the effectiveness of software testing techniques.  He has many links to areas such as Effectiveness of testing techniques, Testing Object-oriented Programs.

Example:

"Adaptive Testing of Non-deterministic Communications Protocols," 6th International Workshop on Protocol Test Systems, Sept. 1993, (with M. Ghriga). 

"Ada Reports & Papers (Part 8)" has many abstracts covering Software Safety and Engineering, mostly done by or for the Military.

Q. Does DoD offer guidance for programming language selection? 

A. Yes. The DoD Joint Technical Architecture V2.0 references the Information Technology Standards Guidance V3.1, which includes the following guidance: 

from section 3.2.3.3.3: "the intrinsically low level nature of C and lack of direct support for modern software engineering approaches and discipline make it an undesirable language for the development of large, general purpose DoD software applications."

"Because the mechanics of the C language are embedded in C++, it is susceptible to many of the above noted difficulties with C, despite the introduction of OOP software engineering into the language."

from section 3.2.3.3.6: "Use of C++ for the development of critical systems applications is not recommended."

" While C is a popular programming language, it is not the best for safety-critical systems.

* In particular, care needs taken in the software design phase with regards to:"
 

If your unsure of what Ada is about then check out this free compiler:

ezl1998a.zip    GNAT Ada 95 compiler readme files, install...
ezl1998b.zip    Mini-DJGPP built for GNAT/DOS Ada 95 compiler
ezl1998c.zip    GNAT Ada 95 compiler v3.10p, part #1
ezl1998d.zip    GNAT Ada 95 compiler v3.10p, part #2
ezl1998e.zip    GNAT Ada 95 compiler v3.10p, part ezl1998e.zip    GNAT Ada 95 compiler v3.10p, part #3
ezl1998f.zip    GNAT Ada 95 compiler v3.10p, part #4
ezl1998g.zip    GNAT Ada 95 compiler components, demos, etc.
ezl1998h.zip    GNAT Ada 95 tutorials
ezl1998i.zip    GNAT Ada 95 tutorials

EZ2LOAD 1998 edition is a distribution of the famous GNAT Ada 95
compiler, built by volunteers for the DOS platform. There are also
software components and tutorials.  Read the documentations and "readme" files that are in EZ2LOAD.

Public Domain.

http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998a.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998b.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998c.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998d.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998e.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998f.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998g.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998h.zip
http://www.simtel.net/pub/simtelnet/msdos/ada/ezl1998i.zip
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998a.zip 992295 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998b.zip 1299960 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998c.zip 1455651 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998d.zip 1453997 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998e.zip 1453473 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998f.zip  1101571 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998g.zip 1343279 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998h.zip 1321924 bytes
ftp://ftp.simtel.net/pub/simtelnet/msdos/ada/ezl1998i.zip  1224073 bytes

Delorie Software provides DJGPP, which is a complete 32-bit C/C++ development system for Intel 80386 (and higher) PCs running DOS. It includes ports of many GNU development utilities. The development tools require a 80386 or newer computer to run, as do the programs they produce. In most cases, the programs it produces can be sold commercially without license or royalties.  CVS has been ported to DJGPP as well. 

[Update Jan/2007: The reliability analysis center is now the Alion System Reliability Center (SRC). They operated the Reliability Analysis Center (RAC) under contract to the Department of Defense for 37 years.]

The Reliability Analysis Center (RAC) is one of 13 DoD Information Analysis Centers chartered to collect, analyze, and disseminate information on reliability, maintainability, and related assurance technologies such as quality, availability, and now supportability. RAC maintains a technical library, a staff of technical specialists, and computerized databases. It performs special studies under approved technical area tasks (TATs), publishes a current awareness bulletin (The RAC Journal), and has produced a variety of technical products for sale. RAC also conducts standard and customized training courses in assurance related technologies.

The Reliability Analysis Center (RAC) is DoD Information Analysis Centers (IACs). The IACs are chartered by the DoD to collect, analyze, and disseminate data and information in a designated technical area of specialization. Information is distributed to DoD and industry via databases, methodology handbooks, state-of -the-art technology reviews, training courses, and consulting services.

RAC's scope is the reliability, maintainability, quality, and supportability of microcircuits, semiconductors, electromechanical and mechanical parts, and equipment/systems employing these parts.

Information Resources

Bibliographic Data
Comprehensive searchable bibliographic data (books, standards, journal articles, symposium papers, and other documents) covering the spectrum of issues related to reliability, maintainability, quality, and supportability.

Reliability Data
Quantitative and qualitative reliability data on electrical, mechanical, and electromechanical components and systems.

The "Related Web Sites" page presents a representative selection of World Wide Web sites that provide an excellent starting point for reliability related information available on the Web.
"Resources Related to Software" should be of particular interest to you.

"Applying Software Reliability Engineering (SRE) to Build Reliable Software (Start 98-3)
"Why don't we do what we know how to do?" The knowledge, the technology, and the methods for building reliable software are known, and have been known for some time - yet few organizations are reporting successful implementation of software reliability programs. "People" issues are at the forefront of our problem in and slow movement along the path to "reliable" software. Each organization needs someone to "champion" and that person need not be the manager, or the department head. The effort need not be full scale but participants need to identify and sustain a common goal."

This START attempts to clarify the common goal, to identify the essential components of the software reliability engineering discipline which, if implemented, can guide an organization to developing more reliable software.

PRISM is the new Reliability Analysis Center (RAC) software tool that ties together several tools into a comprehensive system reliability prediction methodology.  The PRISM concept accounts for the myriad of factors that can influence system reliability, combining all those factors into an integrated system reliability assessment resource.

PRISM was developed to overcome inherent limitations in MIL-HDBK-217, which is no longer being actively maintained or updated by the Department of Defense (DoD).

The Third Quarter-1999 issue of "The Journal of the RAC Reliability Analysis Center" had a great deal of information on PRISM.

This site contains pointers to information on Safety-Critical Systems, where human lives may be at risk, especially involving software and computers, available around the world on the World Wide Web (WWW).  Pointers to relevant newsgroups, books, journals, repositories, and mailing lists can be found here as well.

This site, first developed in 1994, contains a listing of over 2,000 health and safety Internet based resources.

Safety critical systems are defined briefly as those where software is used in a way which might affect safety.  The Institution has undertaken a number of tasks relating to it, details of which can be found here.

The Software Engineering Institute (SEI) is a federally funded research and development center established in 1984 by the U.S. Department of Defense with a broad charter to address the transition of software engineering technology. The SEI is an integral component of Carnegie Mellon University and is sponsored by the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics [OUSD (AT&L)] .

The first International Conference on the Safety of Industrial Automated Systems welcomed over 200 participants. The quality of the presentations, the range of subjects, and the presence of 15 exhibitors made this event a truly valuable survey of the field for the researchers, equipment designers, industry and prevention specialists in attendance.

The Conference Proceedings, containing the full text of 44 presentations, are now available at a price CAN $50.00 (tax included).