The DoD Performance Assessment Guide contains 3 stand-alone modules
that work together to help you: (1) benchmark your organization's
quality climate and quality management strengths and weaknesses
(The Quality and Productivity Self-Assessment Guide); (2)
track your organization's performance over time (The Guide
for Developing Performance Measures); and, (3) track what
your organization's customers think about the service they receive
(The Guide For Measuring Customer Satisfaction).
Each module will be discussed in separate sections of the User's
Guide. Guided practice is available for each module and it is
recommended that you follow the numbered steps before attempting
to collect data.
The DoD Performance Assessment Guide was designed to operate on
an IBM-compatible personal computer (PC). At least a 386 sx PC
is necessary. Requirements include 2.5 megabytes of available
hard disk space and 540 K of available RAM. An EGA or VGA monitor
may be required for displaying the graphics characters. A printer
capable of printing IBM graphics characters is also desirable
if you want to print screens showing the graphic results.
The DoD Performance Assessment Guide has been designed to operate with MS-DOS 3.0 or higher. It can also be "run" from Windows.
1. Save the self extracting, compressed program file
"padisk1.exe" in a local directory.
2, Using Windows "file manager" or "run" command, execute the "padisk1.exe". All necessary programs will be copied to the directory. Then the "padisk1.exe" file can be deleted.
3. To start the program type:
5. Use the arrow keys on screen to highlight the
module of your choice and
press the <Enter> key. Choose Module 1 (Quality and Productivity
Quality and Productivity
Self-Assessment User's Guide
Accessing the Guide
Organizing Surveys and Results
Combining Organization Data
Appendix A: Definitions of Statistical Terms A-1
The Quality and Productivity Self-Assessment Guide (Guide) is an easy way to benchmark your organization's quality climate and quality management strengths and weaknesses. The survey is based on a validated model measuring organizational quality climate, processes, tools, outcomes, and results. Results are also presented based on the Malcolm Baldrige National Quality Award categories and items.
The Guide obtains information from the individuals who know your organization best - your people! You decide the administration procedures that are best for your organization, collect the survey responses, and receive comprehensive evaluation reports generated by the software.
The Guide reports:
The Guide provides people with immediate feedback about their
own scores. If using the automated survey, scores are presented
immediately upon completion of the survey. People may print out
a copy of their scores to keep as a reference point for later
use. We've found that people enjoy comparing their scores to overall
organizational scores as they become available. Individual scores
point people to ideas and reference material that they can review
in a companion "Ideas and Sources" booklet.
The Guide also serves as an "awareness" or educational tool. By completing the survey and reviewing the companion "Ideas and Sources" booklet, people become increasingly aware of the importance of, and what they can do to contribute to, quality excellence within their organization.
The User's Guide will show you how to:
Access the Guide
1. From the DoD Performance Assessment Guide's initial screen,
choose Module 1, Quality and Productivity Self-Assessment Guide.
2. Type in the organization and work unit names. Follow the example shown on the screen.
3. A prompt appears asking you to select which survey you'd like to administer. Press <Enter> to select the climate survey for all respondents.
Note: If you have entered this module after using Module 2 (Guide for Developing Performance Measures) the organization and work unit (at bottom left of screen) will reflect the names you used for Module 2. These names can be easily changed using the Edit Org/Work Unit/Surv Adm option in the Utilities menu.
4. The Main Menu appears.
The Main Menu
The Main Menu consists of three elements:
The Guide software has been designed to provide you with a number of administrative alternatives. The surveys may be responded to on 1 or more PCs where answers are stored anonymously. After all responses have been entered, an administrator collects the data on floppy diskettes, copies them onto his/her PC, and chooses a menu option which calculates results and presents reports.
Should these procedures prove impractical for your organization, people may complete the hard copy survey (anonymously). After collecting the hard copies, the administrator can quickly enter the scores on his/her PC to receive results.
Administration Options Summary
1. People may complete the automated survey at 1 or more centralized PC locations.
The entire survey may be taken by everyone in the organization, in which case results will be obtained for the entire model (climate, processes, tools, outcomes, and results) and for the Baldrige criteria. Using the entire survey (174 questions) is recommended for organizations whose entire work force has been exposed to quality concepts, tools, etc. The survey should take between 30 - 60 minutes to complete.
The climate portion of the survey (56 questions) may be taken by everyone, in which case results are available for the climate portion of the model only. This option is recommended for organizations who have not yet implemented quality practices, tools, etc. within the organization.
Finally, the automatic determination option presents the entire survey to anyone indicating that he/she is a supervisor and the climate portion to everyone else. This option is recommended for organizations who have only recently implemented quality practices, tools, etc. within the organization. Choose the Edit Org/Work Unit/Surv Adm option from the Utilities menu to change the option. Your choice is reflected at the bottom left of the screen.
Define Organization and Work Unit
The Guide has been designed to provide information about an "organization" and its sub-elements or "work units". These terms are defined by you and reflected near the bottom left of the screen. A hypothetical example will be used for illustrative purposes. The overall organization may be the ABC Corporation. The ABC Corporation has several regional locations. Each regional location is responsible for several departments.
The typical way to define organization and work unit is at the office level. In this case the organization might be defined as "Department 1" and the work unit as "Office 1". People from Office 1 would take the survey at one PC. People from Office 2 would use a different PC where the organization is defined as Department 1 and the work unit as Office 2. The same survey (entire/climate/automatic) should be used so that later, the data from each separate organization/work unit element, may later be combined at the department level, regional level, and/or overall organizational level depending upon the interest of the user.
A good rule of thumb for defining work units is to have at least 10 people, and preferably more, comprising a work unit. This will ensure individuals greater anonymity and increase the likelihood of frank responses.
After deciding the organization/work unit combinations that will work best for you, the Guide provides a convenient way for you to store your survey responses separately. This is done by setting up directories within the Survey Directory on your computer. For practice purposes, you will be setting up 3 new directories.
Setting Up Directories
1. On the Main Menu (from Utilities), select New File Set. The Disk Drive Specification window appears:
2. Complete the entries as specified below and press <Enter>.
4. A prompt appears asking if you'd like to create a new directory. Type Y.
5. The Organization Name and Work Unit Name prompt appears.
6. Type the organization to which this survey is to be administered, (for practice, type in Organization 1 and Work Unit 1) then press <Enter>.
7. Choose climate survey option
8. Practice by creating 2 additional directories. Use Organization 1 and Work Unit 2 for 1 directory. Choose Climate Survey option. Use Organization 1 and Work Unit 3 for the other directory and choose Climate Survey option.
1. From the Utilities menu, select New File Set.
2. Select the directory in which Organization 2/Work Unit 2 is located, then press <Enter>. The directory you selected is displayed in the lower left corner of the Guide Main Menu screen.
Responding to Automated Survey
1. Choose Survey from the Questionnaire Menu.
2. Complete demographic question screen. (Choose any answers for practice.)
3. Press <F10> to save your answers and continue. (All questions must be answered in order to save and continue.)
4. Complete survey questions. (Choose any answers for practice.) All questions must be answered in order to be saved.
5. After reviewing scores, press <Esc> to continue.
6. Choose Y at prompt to print out scores.
7. Choose Ideas and Sources option from Questionnaire menu to review information about score categories on screen or print the entire file using the Print Ideas/Sources option from the Utilities menu.
Setting Up Automated Survey For Respondents
1. The survey administrator should create one or more directories and enter the appropriate organization and work unit names. Note: If there is more than one organization/work unit directory, a survey administrator will need to be on hand to make sure respondents are in the correct directory prior to completing the survey.
2. Leave the Survey option on the Main Menu highlighted. A screen saver will appear after a few minutes and direct the next respondent to press a key and choose this option.
Entering Data From Hard Copy Surveys
1. Choose Hard Copy Scores from the Utilities menu.
2. You will be prompted to select the survey type; Entire survey or Climate survey. Choose Entire survey.
3. Enter data for all questions. (Enter random answers for practice.) Press <F10> to save responses.
Once you have received and entered all survey responses, you can view the results. The Guide provides graphs and reports that help you interpret scores and understand what the data represents.
To View Results (Guide Categories)
The original model was developed by General Research Corporation (GRC) by conducting an extensive quality and productivity literature review including the 1989 Baldrige Award, Deming, Crosby, Juran, and other distinguished authorities on the subject. Based on the literature review, the GRC model was developed and questions written or revised to address each portion of the model. The questions were pre-tested with 6 organizations, revised and tested with 49 organizations. The model has since undergone 2 revisions; each based on testing and knowledge gained with previous versions.
The GRC model includes 5 areas related to quality improvement: Climate, Processes, Tools, Outcomes, and Results. Each area is further subdivided into categories and items.
The Guide areas, categories, and items define an alternative breakdown of Baldrige category scores. The Guide categories and items (based on the original GRC model) are smaller in scope than the Baldrige categories and can point to more specific goals and objectives for improvement. The Ideas and Sources booklet define the categories and items for you.
The Guide results are presented by area, category, item, and overall Guide result. For each of these, a mean score, standard deviation, and 95% confidence intervals are presented (across all individuals responding). These are described in Appendix A.
1. From Consolidator, select Calculate. You will be prompted to choose Baldrige Award breakout or DoD Survey breakout. Choose DoD Survey.
2. The demographic screen is presented, by default, with all categories checked. Press <F10> to continue.
3. The results by the DoD survey breakout are presented on screen. Your results might look something like this.
Survey scoring consists of three statistical elements:
|Mean Score||The average score across all respondents for each question and for the survey as a whole.|
|Standard Deviation||An expression of variability about the mean. A large standard deviation indicates that people do not agree about the question. Some people answered with a score of 1 and others with a score of 6. A small standard deviation (1.00 or lower) indicates that people are agreeing about the statement.|
|95% Confidence Limit||Mean scores are "best guesses" about the true mean. Confidence limits define an interval likely to contain the "true" category mean. The 95% interval allows that 95 times out of 100, the true mean lies somewhere between the upper and lower limit.|
4. Press <Esc>. The question distribution report
is generated. To view the reports, choose View from the
Consolidator menu. Then, choose DoD Survey. Choose
Summary-by-Category Report to view the overall report which
was reviewed above. Choose Question Specific Report to
view statistics for each question. Select a category to view the
associated questions. The picture describes the statistics that
are presented in the question-specific report.
5. Press <Esc> to return to the Main Menu.
To View Results (Baldrige Categories)
The results are presented by Baldrige category and item. An overall Baldrige score is also presented. For each category, item, and the overall Baldrige result, a mean score, standard deviation, and 95% confidence intervals are presented (across all individuals responding). These are described in further detail in Appendix A.
1. From Consolidator, select Calculate. Choose Baldrige Award.
2. The demographic screen is presented, by default, with all categories checked. Press <F10> to continue.
3. The results by the Baldrige survey breakout are presented on screen. As with the DoD Survey, similar statistics are presented for Baldrige categories and items.
4. Press <Esc> to continue. The question-specific report is generated and may be viewed by selecting View, Baldrige Award, and Question-Specific Report.
What the Results Mean
The Baldrige results should be interpreted with caution! The scores
achieved on the Guide self-assessment cannot be directly related
to the scores you may achieve by the Baldrige examination team.
The Baldrige award is based on an extremely stringent variety
of data. The Guide is a self-assessment which is not as thorough
or accurate as the actual examination conducted by the Baldrige
evaluators. When a Baldrige evaluation team actually comes to
assess your organization, they will conduct extensive interviews,
complete tours of your facilities, and examine a wide variety
of data and materials that relate to quality.
The Guide Baldrige results CAN point out your organization's strengths and weaknesses as they relate to Baldrige categories. As a yardstick, it is safe to assume that unless your mean scores include "6" at the upper end of the confidence limits, you will not receive full points by the Baldrige examination team for that item. On the other hand, many items cannot be fully assessed through use of a survey. Therefore, a mean score of "6" on an item does not ensure that you will meet all Baldrige criteria. Consult the Baldrige Application Guidelines for a full description of each item and category.
Converting Guide Scores into Baldrige Points
This section is designed for those individuals who are curious about how to convert the mean scores presented by the Guide into Baldrige points. Remember! The conversion will point to areas where improvement is needed but there can be no one-to-one comparison of Guide points with Baldrige Examiner awarded points.
1. Divide the mean score obtained on a category or item by 6. The result is the percentage of Baldrige points you have achieved for that category or item.
For example, suppose that you obtained a mean score of 5.2 on the Baldrige Item 1.1 Senior Executive Leadership. Dividing 5.10 by 6 = .85. The Baldrige Application indicates that 45 points are possible for Item 1.1. Eighty-five percent of 45 = 38 which means that you have achieved 38 points out of 45.
To Print Results
1. Select Print from the Consolidator menu.
2. Select DoD Survey and then the Question-Specific Report or Summary-by-Category Report to print these reports.
3. Select Baldrige Award and then the Question-Specific Report or Summary-by-Category Report to print these reports.
To View Graphs
1. Choose Graphics from the Consolidator menu. Choose
either DoD Survey or Baldrige Award. Choose from
among the categories to view graphs depicting mean scores and
You can combine data from different directories which represent
different organization/work unit combinations. This option allows
you to review the "big picture" for larger departments
and for the entire organization. Use the following procedures
to combine results from 2 or more organization/work unit combinations.
Remember, the same survey (either climate, entire, or automatic)
must be used in order to combine results!
Combining Data Using Floppy Diskettes
1. Access the file set for an organization.
2. From Utilities, select Export Data. The Drive Selection window appears.
3. Select the floppy drive, then press <Enter>. The Insert Diskette prompt appears.
4. Insert a formatted floppy diskette, then press <Enter>. The file set is copied to the diskette.
5. Press any key. The Main Menu appears.
1. On the Main Menu, create a New File Set to store your
2. From Utilities, select Import Data. The Drive Selection list appears.
3. Select the floppy drive on which the exported data is stored.
4. Insert the floppy with the data. Follow the prompts until the responses are imported into the directory.
5. Access the second organization's file set.
6. From Utilities, select Export Data. Follow the prompts until the data is exported.
7. Access the file set to which the data is to be imported.
8. From Utilities, select Import Data. Follow the prompts until the data is added to the file set.
Combining Data Using the Hard Drive
1. Practice by responding to 2 surveys for the data files you
created earlier representing organization 1/work unit 2 and organization
1/work unit 3.
2. Access the file set for organization 1/work unit 1.
3. From Utilities, select Export Data. The hard drive directory prompt appears.
4. Type in a hard drive directory, then press <Enter>. The file set answers are copied into the new directory. Name the organization "Organization 1" and the work unit "All Work Units".
5. Access the file set for organization 1/work unit 2.
6. From Utilities, select Export Data. The hard drive directory prompt appears.
7. Type in the same name for the new file set you created in step 4. Then press <Enter>. The file set answers are added to the first organization's in the new directory.
8. Access the file set for organization 1/work unit 3.
9. From Utilities, select Export Data. The hard drive directory prompt appears.
10. Type in the same name for the new file set you created in step 4. Then press <Enter>. The file set answers are added to the first organization's in the new directory.
11. Practice by consolidating the responses and viewing the results.
New File Set
Use this option to create or change directories. Additional directories are helpful for storing data separately for organization/work unit combinations.
Edit Org/Work Unit/Surv Adm
Choose this option to change the organization and work unit names in the event that they are misspelled. Also, use this option to change the type of survey you'd like to administer. The choices are climate only, the entire survey, or automatic determination. These choices are explained on page 5.
Use this option to select the type of survey you'd like to administer.
The choices are climate only, the entire survey, or automatic
determination. These choices are explained on page 5.
Survey Version (Red/Blue)
Select this option to change the survey version. Both versions contain identical questions but the questions are presented in a different order.
Copies response data from a floppy diskette into the currently selected directory.
Hard Copy Scores
Use this option to enter data quickly from hard copy surveys.
Copies response data from the currently selected directory to a floppy diskette or to the hard drive.
Use this option to print the entire Ideas/Sources reference material. If you have trouble printing, check your printer port and printer type selections. Open the sfideas.txt file stored in your "survey" directory into any word processing program. Print the file from there. Or, make copies of the printed "Ideas and Sources" reference material which accompanies the software.
Use this option after program execution is interrupted (e.g., a power failure).
Change System Date
Use this option to set a new system date.
Change Printer Type
Use this option to specify the printer on which reports are to be printed.
Change Printer Port
Use this option to specify the connection port on your computer
to which the selected printer (above) is connected.
APPENDIX A: DEFINITIONS OF STATISTICAL TERMS
The mean (arithmetic average) is the sum of the values of all responses divided by the number of responses.
A standard deviation is the square root of the variance of a distribution. The variance is computed by summing the squared differences from the mean for all responses and then dividing by 1 less than the number of respondents. If all responses are identical, there is no variance and the standard deviation would be 0. The larger the standard deviation, the more spread out, or less agreement, there exists among responses.
95% Confidence Limits
The category means obtained are "best guesses" about the true mean. Confidence limits define an interval likely to contain the true category mean. The 95% interval allows that 95 times out of 100, the true mean lies somewhere between the upper and lower limit.
For any given category mean, confidence limits that do not include the scale mean (3.5) are thought to be significantly above or significantly below the scale mean.
For example, suppose that a category mean of 4.30 was obtained. The confidence limits are 4.00 and 4.60. Since 3.5 is not included within the confidence interval, the category mean is considered significantly above the scale mean. Similarly, consider a category mean of 2.70 with accompanying confidence limits of 2.30 and 3.10. Since 3.5 is not included within the limits, the category mean is considered significantly lower than the scale mean.
Consult an introductory statistics text for the formula to calculate confidence intervals.