Performance Assessment Guide


INTRODUCTION

The DoD Performance Assessment Guide contains 3 stand-alone modules that work together to help you: (1) benchmark your organization's quality climate and quality management strengths and weaknesses (The Quality and Productivity Self-Assessment Guide); (2) track your organization's performance over time (The Guide for Developing Performance Measures); and, (3) track what your organization's customers think about the service they receive (The Guide For Measuring Customer Satisfaction).

Each module will be discussed in separate sections of the User's Guide. Guided practice is available for each module and it is recommended that you follow the numbered steps before attempting to collect data.


INSTALLATION

The DoD Performance Assessment Guide was designed to operate on an IBM-compatible personal computer (PC). At least a 386 sx PC is necessary. Requirements include 2.5 megabytes of available hard disk space and 540 K of available RAM. An EGA or VGA monitor may be required for displaying the graphics characters. A printer capable of printing IBM graphics characters is also desirable if you want to print screens showing the graphic results.

The DoD Performance Assessment Guide has been designed to operate with MS-DOS 3.0 or higher. It can also be "run" from Windows.

1. Save the self extracting, compressed program file "padisk1.exe" in a local directory.
2, Using Windows "file manager" or "run" command, execute the "padisk1.exe". All necessary programs will be copied to the directory. Then the "padisk1.exe" file can be deleted.
3. To start the program type:

4. Read introductory screens or press <Enter> key to continue.

5. Use the arrow keys on screen to highlight the module of your choice and press the <Enter> key. Choose Module 1 (Quality and Productivity Self-Assessment Guide.)


MODULE 1

Quality and Productivity Self-Assessment User's Guide


TABLE OF CONTENTS

Topic Page

Introduction
Objectives
Accessing the Guide
Administering Survey
Organizing Surveys and Results
Reviewing Results
Reports
Combining Organization Data
Guide Utilities

Appendix A: Definitions of Statistical Terms A-1


INTRODUCTION

The Quality and Productivity Self-Assessment Guide (Guide) is an easy way to benchmark your organization's quality climate and quality management strengths and weaknesses. The survey is based on a validated model measuring organizational quality climate, processes, tools, outcomes, and results. Results are also presented based on the Malcolm Baldrige National Quality Award categories and items.

The Guide obtains information from the individuals who know your organization best - your people! You decide the administration procedures that are best for your organization, collect the survey responses, and receive comprehensive evaluation reports generated by the software.

The Guide reports:

The Guide provides people with immediate feedback about their own scores. If using the automated survey, scores are presented immediately upon completion of the survey. People may print out a copy of their scores to keep as a reference point for later use. We've found that people enjoy comparing their scores to overall organizational scores as they become available. Individual scores point people to ideas and reference material that they can review in a companion "Ideas and Sources" booklet.

The Guide also serves as an "awareness" or educational tool. By completing the survey and reviewing the companion "Ideas and Sources" booklet, people become increasingly aware of the importance of, and what they can do to contribute to, quality excellence within their organization.

OBJECTIVES

The User's Guide will show you how to:


It is recommended that you walk through the manual, following the numbered steps as practice, prior to collecting data. The practice allows you to use both surveys -each illustrating various aspects of the software. The guided practice should take about 2 hours to complete.
ACCESSING THE GUIDE

Access the Guide

1. From the DoD Performance Assessment Guide's initial screen, choose Module 1, Quality and Productivity Self-Assessment Guide.
2. Type in the organization and work unit names. Follow the example shown on the screen.
3. A prompt appears asking you to select which survey you'd like to administer. Press <Enter> to select the climate survey for all respondents.

Note: If you have entered this module after using Module 2 (Guide for Developing Performance Measures) the organization and work unit (at bottom left of screen) will reflect the names you used for Module 2. These names can be easily changed using the Edit Org/Work Unit/Surv Adm option in the Utilities menu.

4. The Main Menu appears.

The Main Menu

The Main Menu consists of three elements:


ADMINISTERING SURVEY

Overview

The Guide software has been designed to provide you with a number of administrative alternatives. The surveys may be responded to on 1 or more PCs where answers are stored anonymously. After all responses have been entered, an administrator collects the data on floppy diskettes, copies them onto his/her PC, and chooses a menu option which calculates results and presents reports.

Should these procedures prove impractical for your organization, people may complete the hard copy survey (anonymously). After collecting the hard copies, the administrator can quickly enter the scores on his/her PC to receive results.

Administration Options Summary

1. People may complete the automated survey at 1 or more centralized PC locations.

2. People may complete the hard copy survey.
Any combination of the procedures may be used. All responses, whether entered via the software or from hard copy, will eventually be entered or copied onto the survey administrator's PC. Software options are used to calculate responses, display results graphically, and print results.
Decide which survey version to use

(screen image)

The entire survey may be taken by everyone in the organization, in which case results will be obtained for the entire model (climate, processes, tools, outcomes, and results) and for the Baldrige criteria. Using the entire survey (174 questions) is recommended for organizations whose entire work force has been exposed to quality concepts, tools, etc. The survey should take between 30 - 60 minutes to complete.

The climate portion of the survey (56 questions) may be taken by everyone, in which case results are available for the climate portion of the model only. This option is recommended for organizations who have not yet implemented quality practices, tools, etc. within the organization.

Finally, the automatic determination option presents the entire survey to anyone indicating that he/she is a supervisor and the climate portion to everyone else. This option is recommended for organizations who have only recently implemented quality practices, tools, etc. within the organization. Choose the Edit Org/Work Unit/Surv Adm option from the Utilities menu to change the option. Your choice is reflected at the bottom left of the screen.

Define Organization and Work Unit

The Guide has been designed to provide information about an "organization" and its sub-elements or "work units". These terms are defined by you and reflected near the bottom left of the screen. A hypothetical example will be used for illustrative purposes. The overall organization may be the ABC Corporation. The ABC Corporation has several regional locations. Each regional location is responsible for several departments.

The typical way to define organization and work unit is at the office level. In this case the organization might be defined as "Department 1" and the work unit as "Office 1". People from Office 1 would take the survey at one PC. People from Office 2 would use a different PC where the organization is defined as Department 1 and the work unit as Office 2. The same survey (entire/climate/automatic) should be used so that later, the data from each separate organization/work unit element, may later be combined at the department level, regional level, and/or overall organizational level depending upon the interest of the user.

A good rule of thumb for defining work units is to have at least 10 people, and preferably more, comprising a work unit. This will ensure individuals greater anonymity and increase the likelihood of frank responses.


The survey asks people questions about their organization and work unit so it is very important that these terms be defined for them. The administrator should enter this information in the software. If hard copies are used, the administrator types in this information on the hard copies prior to distribution.
ORGANIZING SURVEYS AND RESULTS

After deciding the organization/work unit combinations that will work best for you, the Guide provides a convenient way for you to store your survey responses separately. This is done by setting up directories within the Survey Directory on your computer. For practice purposes, you will be setting up 3 new directories.

Setting Up Directories

1. On the Main Menu (from Utilities), select New File Set. The Disk Drive Specification window appears:



2. Complete the entries as specified below and press <Enter>.

4. A prompt appears asking if you'd like to create a new directory. Type Y.

5. The Organization Name and Work Unit Name prompt appears.

6. Type the organization to which this survey is to be administered, (for practice, type in Organization 1 and Work Unit 1) then press <Enter>.

7. Choose climate survey option

8. Practice by creating 2 additional directories. Use Organization 1 and Work Unit 2 for 1 directory. Choose Climate Survey option. Use Organization 1 and Work Unit 3 for the other directory and choose Climate Survey option.

Selecting Directories

1. From the Utilities menu, select New File Set.

2. Select the directory in which Organization 2/Work Unit 2 is located, then press <Enter>. The directory you selected is displayed in the lower left corner of the Guide Main Menu screen.

Responding to Automated Survey

1. Choose Survey from the Questionnaire Menu.

2. Complete demographic question screen. (Choose any answers for practice.)

3. Press <F10> to save your answers and continue. (All questions must be answered in order to save and continue.)

4. Complete survey questions. (Choose any answers for practice.) All questions must be answered in order to be saved.

5. After reviewing scores, press <Esc> to continue.

6. Choose Y at prompt to print out scores.

7. Choose Ideas and Sources option from Questionnaire menu to review information about score categories on screen or print the entire file using the Print Ideas/Sources option from the Utilities menu.

Setting Up Automated Survey For Respondents

1. The survey administrator should create one or more directories and enter the appropriate organization and work unit names. Note: If there is more than one organization/work unit directory, a survey administrator will need to be on hand to make sure respondents are in the correct directory prior to completing the survey.

2. Leave the Survey option on the Main Menu highlighted. A screen saver will appear after a few minutes and direct the next respondent to press a key and choose this option.

Entering Data From Hard Copy Surveys

1. Choose Hard Copy Scores from the Utilities menu.

2. You will be prompted to select the survey type; Entire survey or Climate survey. Choose Entire survey.

3. Enter data for all questions. (Enter random answers for practice.) Press <F10> to save responses.


REVIEWING RESULTS

Once you have received and entered all survey responses, you can view the results. The Guide provides graphs and reports that help you interpret scores and understand what the data represents.

To View Results (Guide Categories)

The original model was developed by General Research Corporation (GRC) by conducting an extensive quality and productivity literature review including the 1989 Baldrige Award, Deming, Crosby, Juran, and other distinguished authorities on the subject. Based on the literature review, the GRC model was developed and questions written or revised to address each portion of the model. The questions were pre-tested with 6 organizations, revised and tested with 49 organizations. The model has since undergone 2 revisions; each based on testing and knowledge gained with previous versions.

The GRC model includes 5 areas related to quality improvement: Climate, Processes, Tools, Outcomes, and Results. Each area is further subdivided into categories and items.

The Guide areas, categories, and items define an alternative breakdown of Baldrige category scores. The Guide categories and items (based on the original GRC model) are smaller in scope than the Baldrige categories and can point to more specific goals and objectives for improvement. The Ideas and Sources booklet define the categories and items for you.

The Guide results are presented by area, category, item, and overall Guide result. For each of these, a mean score, standard deviation, and 95% confidence intervals are presented (across all individuals responding). These are described in Appendix A.

1. From Consolidator, select Calculate. You will be prompted to choose Baldrige Award breakout or DoD Survey breakout. Choose DoD Survey.

2. The demographic screen is presented, by default, with all categories checked. Press <F10> to continue.




3. The results by the DoD survey breakout are presented on screen. Your results might look something like this.

Survey scoring consists of three statistical elements:

Mean ScoreThe average score across all respondents for each question and for the survey as a whole.
Standard DeviationAn expression of variability about the mean. A large standard deviation indicates that people do not agree about the question. Some people answered with a score of 1 and others with a score of 6. A small standard deviation (1.00 or lower) indicates that people are agreeing about the statement.
95% Confidence LimitMean scores are "best guesses" about the true mean. Confidence limits define an interval likely to contain the "true" category mean. The 95% interval allows that 95 times out of 100, the true mean lies somewhere between the upper and lower limit.


NOTE: See Appendix A for definitions of statistical terms.

4. Press <Esc>. The question distribution report is generated. To view the reports, choose View from the Consolidator menu. Then, choose DoD Survey. Choose Summary-by-Category Report to view the overall report which was reviewed above. Choose Question Specific Report to view statistics for each question. Select a category to view the associated questions. The picture describes the statistics that are presented in the question-specific report.

5. Press <Esc> to return to the Main Menu.

To View Results (Baldrige Categories)

The results are presented by Baldrige category and item. An overall Baldrige score is also presented. For each category, item, and the overall Baldrige result, a mean score, standard deviation, and 95% confidence intervals are presented (across all individuals responding). These are described in further detail in Appendix A.

1. From Consolidator, select Calculate. Choose Baldrige Award.

2. The demographic screen is presented, by default, with all categories checked. Press <F10> to continue.

3. The results by the Baldrige survey breakout are presented on screen. As with the DoD Survey, similar statistics are presented for Baldrige categories and items.

4. Press <Esc> to continue. The question-specific report is generated and may be viewed by selecting View, Baldrige Award, and Question-Specific Report.

What the Results Mean

The Baldrige results should be interpreted with caution! The scores achieved on the Guide self-assessment cannot be directly related to the scores you may achieve by the Baldrige examination team. The Baldrige award is based on an extremely stringent variety of data. The Guide is a self-assessment which is not as thorough or accurate as the actual examination conducted by the Baldrige evaluators. When a Baldrige evaluation team actually comes to assess your organization, they will conduct extensive interviews, complete tours of your facilities, and examine a wide variety of data and materials that relate to quality.

The Guide Baldrige results CAN point out your organization's strengths and weaknesses as they relate to Baldrige categories. As a yardstick, it is safe to assume that unless your mean scores include "6" at the upper end of the confidence limits, you will not receive full points by the Baldrige examination team for that item. On the other hand, many items cannot be fully assessed through use of a survey. Therefore, a mean score of "6" on an item does not ensure that you will meet all Baldrige criteria. Consult the Baldrige Application Guidelines for a full description of each item and category.

Converting Guide Scores into Baldrige Points

This section is designed for those individuals who are curious about how to convert the mean scores presented by the Guide into Baldrige points. Remember! The conversion will point to areas where improvement is needed but there can be no one-to-one comparison of Guide points with Baldrige Examiner awarded points.

1. Divide the mean score obtained on a category or item by 6. The result is the percentage of Baldrige points you have achieved for that category or item.

For example, suppose that you obtained a mean score of 5.2 on the Baldrige Item 1.1 Senior Executive Leadership. Dividing 5.10 by 6 = .85. The Baldrige Application indicates that 45 points are possible for Item 1.1. Eighty-five percent of 45 = 38 which means that you have achieved 38 points out of 45.

To Print Results

1. Select Print from the Consolidator menu.

2. Select DoD Survey and then the Question-Specific Report or Summary-by-Category Report to print these reports.

3. Select Baldrige Award and then the Question-Specific Report or Summary-by-Category Report to print these reports.


If you have trouble printing, make sure you have correctly identified the printer port and printer type. These options are available on the Utilities menu. If you are still unable to obtain a printout, you might try opening the associated text file in any word processing program and printing the file from there. The following files are found in the "survey" directory you created earlier. Open:

To View Graphs

1. Choose Graphics from the Consolidator menu. Choose either DoD Survey or Baldrige Award. Choose from among the categories to view graphs depicting mean scores and significance.


COMBINING ORGANIZATION DATA

You can combine data from different directories which represent different organization/work unit combinations. This option allows you to review the "big picture" for larger departments and for the entire organization. Use the following procedures to combine results from 2 or more organization/work unit combinations. Remember, the same survey (either climate, entire, or automatic) must be used in order to combine results!

Combining Data Using Floppy Diskettes

Export Data

1. Access the file set for an organization.
2. From Utilities, select Export Data. The Drive Selection window appears.
3. Select the floppy drive, then press <Enter>. The Insert Diskette prompt appears.
4. Insert a formatted floppy diskette, then press <Enter>. The file set is copied to the diskette.
5. Press any key. The Main Menu appears.

Import Data

1. On the Main Menu, create a New File Set to store your combined data.
2. From Utilities, select Import Data. The Drive Selection list appears.
3. Select the floppy drive on which the exported data is stored.
4. Insert the floppy with the data. Follow the prompts until the responses are imported into the directory.
5. Access the second organization's file set.
6. From Utilities, select Export Data. Follow the prompts until the data is exported.
7. Access the file set to which the data is to be imported.
8. From Utilities, select Import Data. Follow the prompts until the data is added to the file set.

Combining Data Using the Hard Drive

1. Practice by responding to 2 surveys for the data files you created earlier representing organization 1/work unit 2 and organization 1/work unit 3.
2. Access the file set for organization 1/work unit 1.
3. From Utilities, select Export Data. The hard drive directory prompt appears.
4. Type in a hard drive directory, then press <Enter>. The file set answers are copied into the new directory. Name the organization "Organization 1" and the work unit "All Work Units".
5. Access the file set for organization 1/work unit 2.
6. From Utilities, select Export Data. The hard drive directory prompt appears.
7. Type in the same name for the new file set you created in step 4. Then press <Enter>. The file set answers are added to the first organization's in the new directory.
8. Access the file set for organization 1/work unit 3.
9. From Utilities, select Export Data. The hard drive directory prompt appears.
10. Type in the same name for the new file set you created in step 4. Then press <Enter>. The file set answers are added to the first organization's in the new directory.
11. Practice by consolidating the responses and viewing the results.


GUIDE UTILITIES

New File Set

Use this option to create or change directories. Additional directories are helpful for storing data separately for organization/work unit combinations.

Edit Org/Work Unit/Surv Adm

Choose this option to change the organization and work unit names in the event that they are misspelled. Also, use this option to change the type of survey you'd like to administer. The choices are climate only, the entire survey, or automatic determination. These choices are explained on page 5.

Survey Administration

Use this option to select the type of survey you'd like to administer. The choices are climate only, the entire survey, or automatic determination. These choices are explained on page 5.

Survey Version (Red/Blue)

Select this option to change the survey version. Both versions contain identical questions but the questions are presented in a different order.

Import Data

Copies response data from a floppy diskette into the currently selected directory.

Hard Copy Scores

Use this option to enter data quickly from hard copy surveys.

Export Data

Copies response data from the currently selected directory to a floppy diskette or to the hard drive.

Print Ideas/Sources

Use this option to print the entire Ideas/Sources reference material. If you have trouble printing, check your printer port and printer type selections. Open the sfideas.txt file stored in your "survey" directory into any word processing program. Print the file from there. Or, make copies of the printed "Ideas and Sources" reference material which accompanies the software.

Re-Index Files

Use this option after program execution is interrupted (e.g., a power failure).

Change System Date

Use this option to set a new system date.

Change Printer Type

Use this option to specify the printer on which reports are to be printed.

Change Printer Port

Use this option to specify the connection port on your computer to which the selected printer (above) is connected.


APPENDIX A: DEFINITIONS OF STATISTICAL TERMS

Mean

The mean (arithmetic average) is the sum of the values of all responses divided by the number of responses.

Standard Deviation

A standard deviation is the square root of the variance of a distribution. The variance is computed by summing the squared differences from the mean for all responses and then dividing by 1 less than the number of respondents. If all responses are identical, there is no variance and the standard deviation would be 0. The larger the standard deviation, the more spread out, or less agreement, there exists among responses.

95% Confidence Limits

The category means obtained are "best guesses" about the true mean. Confidence limits define an interval likely to contain the true category mean. The 95% interval allows that 95 times out of 100, the true mean lies somewhere between the upper and lower limit.

For any given category mean, confidence limits that do not include the scale mean (3.5) are thought to be significantly above or significantly below the scale mean.

For example, suppose that a category mean of 4.30 was obtained. The confidence limits are 4.00 and 4.60. Since 3.5 is not included within the confidence interval, the category mean is considered significantly above the scale mean. Similarly, consider a category mean of 2.70 with accompanying confidence limits of 2.30 and 3.10. Since 3.5 is not included within the limits, the category mean is considered significantly lower than the scale mean.

Consult an introductory statistics text for the formula to calculate confidence intervals.