Design considerations for safety-critical systems

Topic area Safety Critical Systems
Target audience Undergrad and grad IS/CS/CE majors in general.
Activity type Worksheet, reading assignment, position paper, class discussion.
Time required The activity to read the articles and complete the worksheets at home will take 1 hour. The first discussion will consume one class period. The reading of the case pairs, reading of the Code of Ethics, and writing the two page paper will take from 1.5 to 3 hours, depending on student reading and writing ability. The discussion of the papers should take no more than 25 minutes.
Attachments Writing Assignment
Additional materials Copies of the following articles:
  1. "The Mystery of the Airbus 300" U.S. NEWS vol 105 No 3 pg15.
  2. "Report admits crew mistakes cause downing of iranian jet" Xinhua News Agency item no 0819050.
  3. "Crisis in the Gulf; The Radar System that should be foolproof", by David White in Financial Times Limited, 5 July 98 section 1,pg.6.
  4. "Navy to improve Aegis system," by George Lobsenz in the Washington News, 19 August 1988.
  5. "US Naval error 'led to downing of Iran Airbus'" by Air Cdre G.S. Cooper, The Daily Telegraph 6 December 1988, International page 12.
  6. "Three case pairs- Isomorphic ethical issues in safety critical and non-safety critical systems." By Don Gotterbarn, attached.
Copy of ACM Code of Ethics and Guidelines for Professional Conduct. www.acm.org/sigcas or Software Engineering Code of Ethics and Professional Conduct www.cs.etsu.edu/seeriTD.
Background needed to complete the assignment Students need to be given a copy of the articles or otherwise be able to locate the articles about the USS Vincennes' use of the Aegis radar system to shot down an Iranian commercial airliner killing 290 civilians. They also need copies of the ACM Code of Conduct and Guidelines for Professional Conduct or similarly detailed code such as the Software Engineering Code of Ethics and Professional Practice. They also need a copy of the small case pairs.
References L.Lee, The Day the Phones Stopped, Primus Books, N.Y. 1992.
Last modified August 1998
Abstract:
The essence of the assignment is to have the students read a detailed description( distributed over several short articles) of a situation which was responsible for significant loss of life. The case is interesting because the Captain followed a reasonable course of action to assure that he was firing on an attacking enemy jet fighter. The articles make clear, after all of the international and national posturing about why one is at war etc., that the single most significant problem is the poor interface design. The design fails to take into account the real situation in which the radar system would be used — a high stress war-room with little outside contact —and the system does not present the data in a way that is clear to the user. A worksheet is used to ensure that students understand the circumstances. To dispel the view that computer ethics is only about safety critical issues, a series of case pairs are provided with computer ethics issues not related to safety. These stories are structured to map the ethical issues of safety critical cases.

Goals for the activity:
There are three primary goals of this activity. The first goal is to have students relate professional standards to the development of a safety critical piece of software. The second goal is to show students that non-safety critical software also generates significant risks (mission critical risks) and ethical problems. This is done using case pairs which show students that the ethical issues that generate problems for safety critical software can be identical to the ethical issues of non-safety critical software. The goal here is to show students that the ethicality of the problem is not based in the safety issues, but that the ethical issue can reside elsewhere. The third goal is to give students practice at identifying difficult ethical situations using professional standards.

Knowledge / skills / attitudes to be developed (behavioral objectives):
Students should develop:

  1. An ability to identify the social significance of interface design.
  2. An awareness that non-safety critical systems also have ethical hot spots.

Procedure:
This activity is done in several stages. Students are asked to read the articles about the Vincennes and complete the work sheets. Ask the students to come to the next class with the worksheets completed and be prepared for a discussion about who is at fault for the loss of 290 civilian's lives. The work sheets can be collected to facilitate a graded assignment, if so desired. For a portion of the next class discuss the circumstances of the event, identifying the various stakeholders and relevant ethical facts, Care needs to be taken to reduce nationalistic rhetoric.

Once the event has been described, call on the students to discuss the technical issues and design failures—the major system with Most of the significant airplane data details was displayed on four 4 foot wide screens while the critical piece of information that the captain needed to make his decision was displayed on a different monitor with different font, etc. Ask pointed questions. "If you were the programmer, what could you do to fix this problem?" (Answer: Put the important data on the same screen.) It is important to immediately distinguish friend from foe. What display method could you use to make it easy to distinguish friend from foe in a stressful situation? (Answer Display foes in a different color, or blinking or reverse video, etc. make a siren sound for foes.) Spend some time with these solutions thinking about the war room situation in which the Aegis is used. The sound is a bad idea as a solution because of the potential for gun blasts which will cover up the siren sound. Be sure to allow some time at the end of this class to discuss the question " Are the only significant computer ethics issues safety critical issues?" Discuss the obvious case of the loss of life in the Vincennes example. Ask "Is it the loss of life that makes this an issues in computer ethics?" Assign the students the two-three page paper assignment to use the Code of Ethics to Identify the Professional Ethical Issues in the attached case pairs.

Assessing outcomes:
Grading the worksheets provides an objective assessment of there understanding of the situation. It is more important to give feedback on the short essay. Your primary focus should be on their ability to identify relevant standards from the Codes. A purely mechanical list of Code imperatives should receive less credit than a response that recognizes the tension between some of the imperatives that apply, especially in the third case.

Additional remarks:
None.

Author contact information:
Don Gotterbarn
Software Engineering Ethics Research Institute
East Tennessee State University
Box 70711

Page maintained by: kwb@csee.usf.edu