DoDD 5000.4 authorizes the CAIG to establish criteria and procedures for preparing and presenting cost estimates of defense systems requiring DAB review. O&S estimates reviewed by the CAIG should be developed according to a standard set of procedures. This section discusses the CAIG review criteria and presents a costing methodology and conventions for preparing and documenting O&S estimates.
3.2 CAIG REVIEW
The OSD CAIG is charged with improving the quality of cost analysis within the Department of Defense. As such, it serves as the principal advisory body to the Under Secretary of Defense for Acquisition on matters relating to cost. For major programs, the CAIG is responsible for ensuring that the DAB has available to it an independent estimate of program costs. In addition to presenting the CAIG's own cost estimates, the CAIG report to the cognizant DAB committee and to the DAB contains an evaluation of the estimates developed by the DoD component responsible for the program. It therefore is incumbent upon the DoD components to submit well-documented, credible estimates that are ready for review and can be replicated by a third party.
Accordingly, the CAIG has established criteria and procedures for preparing and presenting cost estimates. The criteria apply to the development, investment, and O&S phases of a system's life-cycle cost. The CAIG criteria and procedures are discussed in DoDD 5000.4 (Enclosure 2), which addresses the scope of the analysis, analytical methods, and presentation of results.
3.2.1 Cost Assessment Criteria
Variations in cost-estimating situations are too great to permit an exhaustive listing of the factors the CAIG may consider. The following criteria, however, will be considered in all cases:
If an estimating methodology or supporting documentation is insufficient and does not permit a reasonably firm appraisal of the estimate, the CAIG will recommend that the DAB committee review and the full DAB review be deferred until the deficiency is remedied.
O&S costs typically exceed both development and investment costs over a system's useful life. Therefore, in assessing the total costs of two competing systems, the costs of operating and supporting each system are a primary consideration.
3.2.2 Lessons Learned from CAIG Reviews
The lessons learned from previous CAIG reviews are presented here to highlight some of the areas in which cost estimates have been deficient or additional emphasis has been needed:
3.3 O&S COST-ESTIMATING PROCESS
O&S cost estimates focus on the costs likely to be incurred by a defense system under specified conditions. Although the cost analysis must consider historical costs, it should do more than just extrapolate from past cost trends. The proper approach is to present normalized empirical data derived from the ground rules provided or to show the relationship between an assumption and its related cost impacts. The O&S estimating process described in the sections below is most appropriate for major acquisition programs reviewed by the CAIG. However, the approach is applicable to all acquisition programs, regardless of the review authority.
Preparation for a CAIG review requires
formal identification of the estimating approach, coordination
with the CAIG action officer, updating of the current estimate,
and preparation of an independent estimate. Exhibit 3-1 lists
the primary activities entailed in preparing cost estimates for
review by the CAIG.
3.3.1 Develop Approach
The selection of an estimating approach involves four basic steps: identification of key O&S issues; selection of a reference system; development of ground rules and assumptions; and selection of a cost element structure. The paragraphs that follow describe each of these steps in turn, and provide some guidelines for accomplishing them.
22.214.171.124 Identify Primary O&S Issues
analytical approach and presentation structure selected must be
able to accommodate the unique aspects of the program being assessed.
A good way to ensure that all pertinent issues have been identified
is for representatives of the CAIG to meet with staff members
from the DoD component conducting the analysis. As shown in Exhibit
3-2, the objective of such "pre-CAIG review" meetings
is to discuss the issues addressed in the analysis and agree on
the approach that will be used to estimate O&S costs (or to
update a prior O&S estimate). Where necessary, specific guidance
will be provided by the CAIG representative with respect to the
estimating approach and CAIG expectations. This guidance must
be documented by the organization preparing the estimate, and
the estimating approach should be revised accordingly.
126.96.36.199 Select Reference System
A reference system is an existing operational system with a mission similar to that of the proposed system. It is usually the system being replaced, unless some other system provides a better basis for the analysis. Where data on the proposed new system are not available, a reference system is used as a proxy for that system as well.
Since reference systems are normally in current operation, there are a variety of sources of data on their cost, technical characteristics, and performance. One source of historical cost data specifically directed by DoDD 5000.4 is the Visibility and Management of Operating and Support Costs (VAMOSC) system. DoDD 5000.4 requires that historical data be used to identify and allocate functional costs among major defense systems and subsystems. One of VAMOSC's objectives is to enhance the visibility of O&S costs in DoD cost analyses. Validated VAMOSC data should be used to calculate the O&S costs of the reference system unless some other sources or data bases are clearly more appropriate.
Much of the technical and performance information for the reference system can be obtained from contractor reports, historical government acquisition reports, maintenance data collection systems, and current operational concepts. Similar data for the proposed system may be found in the Cost Analysis Requirements Document (CARD), which is prepared by the DoD component responsible for the program. The CARD provides detailed information on the scope of the program estimate.
assumptions, ground rules, and cost-estimating methodologies for
both the reference and proposed system should be similar. This
enables the analyst to pinpoint differences in resource consumption
that arise from differences in weapon system characteristics.
If a comparison of the characteristics of the reference and proposed
systems has not been completed for the CARD, a suggested format
is provided in Exhibit 3-3. This summary may be presented and
discussed at the pre-CAIG review meeting. If a reference system
is not available or applicable, the sample format can be used
to describe the proposed system.
|Goal (prototype/100th prod unit)|
|Number of Systems|
|No. of Equipped Deployable Units|
|Average No. Systems/Unit|
|Interim Contractor Support|
|Contractor Logistics Support|
|No. of Maintenance Levels|
|Operational Ready Rate (%)|
|(Mean Time Between Removals)|
|(Mean Time Between Maintenance Action (inherent))|
|Maintenance Hours Per|
|(Fly Hr/Oper Hr)|
|Major Overhaul Point|
| (Fly Hrs/Oper Hrs/Miles)
NOTE: 1. The elements
under each category should be expanded, deleted, or revised to
accommodate the system under study.
2. Include additional columns for subsequent milestone reviews as appropriate.
188.8.131.52 Develop Ground Rules and Assumptions
A prerequisite to the development of useful O&S cost estimates is a detailed definition of how a system will be operated, maintained, and supported in peacetime. The basic ground rules and assumptions are derived from this information. The ground rules include descriptions of relevant missions and system characteristics, as well as manning, maintenance, support, and logistics policies. The assumptions "bound" the estimate (that is, they define its scope). The ground rules and assumptions are identified in the study documentation. They must be stated clearly and presented in a way that permits them to be referenced easily. Also, the ground rules and assumptions used to estimate O&S costs must be consistent with those employed for non-O&S elements (such as initial spares). Specific O&S cost conventions to consider in developing ground rules and assumptions are discussed in Section 3.4.
184.108.40.206 Select Relevant Cost Element Structure
A cost element structure (CES) establishes a standard vocabulary for identifying and classifying the costs of a system. A generic CES format is presented in Chapter 4 and discussed in Appendix B. Weapon-system-specific CESs are presented in Appendices C through G. The CESs are designed to capture as many relevant costs as practical within the O&S phase. However, due to differences among programs, the basic structure may have to be modified to accommodate the unique characteristics of the system(s) being investigated. The revised structure should be checked to ensure that all relevant costs are represented and defined in a manner compatible with the cost estimate. If a decision will affect costs not explicitly described in this guide, those costs should be identified and included in the O&S estimate. For cost estimates requiring CAIG review, modifications to the CES format recommended here must be approved by the CAIG prior to the start of the analysis.
3.3.2 Hold Pre-CAIG Meeting
The CAIG action officer, his or her program office counterpart, and the independent cost estimate (ICE) team chief should meet to discuss the scope, methodology, and ground rules and assumptions for the cost analysis. As presented in Exhibit 3-2, topics covered in these discussions include system characteristics, operating schemes, the specifications of each alternative and of the reference system, the cost element structure, historically relevant O&S cost drivers, costs to be included and excluded, data sources, estimating methodologies, cost models, back-up documentation, and significant trade-off issues. All of these matters should be discussed and agreed upon with the CAIG representative. The initial pre-CAIG meeting should be held shortly after the DAB planning meeting, which is normally conducted 180 days in advance of a DAB review.
3.3.3 Prepare Estimate
The paragraphs below describe how to construct an O&S estimate once the CAIG has approved the initial estimating approach. The discussion summarizes the steps entailed in selecting a cost-estimating technique, choosing a cost model, collecting data, and estimating the relevant costs. The requirements for conducting sensitivity analyses and documenting cost estimates also are discussed.
220.127.116.11 Determine Cost-Estimating Techniques
A number of techniques may be employed to estimate the O&S costs of a weapon system. The specific approach chosen will depend to a large degree on the maturity of the program and the level of detail of the available data. Most O&S analyses are accomplished using a combination of three estimating techniques: parametric, analogy, and engineering. As a system progresses from concept development to operations, more detailed information becomes available about it. Initial parametric variables or analogous data may then be updated or replaced with test or operational data.
The paragraphs below summarize the key features of the three estimating techniques used in O&S cost assessments. Regardless of the technique employed, the approach must be documented in the materials accompanying the estimate.
18.104.22.168 Select/Construct Cost Model
O&S estimating is most often accomplished using computer models of cost methodologies. Specific models are not prescribed in this guide because the characteristics of a program normally determine the estimating process. However, most O&S models fall into one of three categories:
In general, when selecting or constructing a model for O&S cost estimating, one should consider not only the system whose costs are being assessed, but also the acquisition milestone to which the system has progressed and the accuracy expected from the estimate. For example, a cost estimate in support of a Milestone I review is not expected to be as detailed as an estimate prepared for Milestones II or III. As a program matures, the cost model and estimating methodology should be adapted to accommodate more precise information. To satisfy both estimating and review requirements, a good model should have the following characteristics:
22.214.171.124 Identify Data and Data Sources
Three types of data are used in O&S estimates: programmatic, technical, and cost. Data of each type must be provided for both the proposed system and the reference system.
Programmatic data are facts or assumptions about the fielding, operating concepts, and support requirements of a weapon system. Technical data focus on system specifications and characteristics, and on performance capabilities or goals. For acquisition programs, this information is available from many of the source documents required by DoDI 5000.2 and from Title 10 of the United States Code. (Appendix A provides a brief summary of the purpose and scope of the major source documents.) Each DoD component may have additional source documents from which similar information may be obtained.
The basic source of cost data for most weapons analyses is the historical resource-consumption rates of analogous systems. These values are expressed in terms of dollars, manning levels, and material quantities. Although the concept is straightforward, most categories of historical O&S costs are actually derived values because large portions of the operations and maintenance budget are not broken out by weapon system. Rather, the budget figures are presented by functional area (e.g., depot maintenance, base operating support, training). To calculate O&S costs for individual weapon systems, various allocation techniques have been developed.
An important consideration when collecting data is the type of cost represented. Three types of data typically are used:
As a general rule, one should use the type of data that best describes the system-related O&S cost. Most O&S cost estimates are derived from two forms of data: raw and processed.
126.96.36.199 Estimate and Evaluate Relevant Costs
The selection, design, development, maintenance, and support concepts for a defense system influence how its O&S costs are analyzed and evaluated. In order to compare system alternatives fully, it is necessary to identify and estimate all relevant O&S costs. For example, the number of crew members needed to operate a weapon system may be identical regardless of the alternative selected. However, the reliability of alternative systems may result in very different maintenance manning and material costs, which are relevant to the decision at hand. The purpose of O&S cost analyses changes as a system moves through the various acquisition phases. For example, one of the primary aims of a Milestone I analysis is to explore and quantify the relative cost advantages of different system alternatives, concepts, and design options (e.g., changes in system availability, requirements, alternative support policies, etc.). As a program matures, the cost analysis focuses on trade-offs within the selected alternative, such as deployment, support, and affordability issues. A fundamental objective of this process is to ensure that the alternative selected satisfies mission requirements at the lowest possible life-cycle cost.
188.8.131.52 Assess Uncertainty
Estimates of future O&S costs are subject to some degree of uncertainty. It is important to identify and bound the scope of variables that contribute to uncertainty in ground rules and assumptions. The following are some of the areas that can cause uncertainty and that must be addressed before developing an O&S estimate: mission scenario, deployment plans, operating tempo, system quantity, system availability, operating environment, and cost-estimating methods. These areas are typically identified as non-cost elements that may affect the O&S costs of a system. A sensitivity analysis is conducted to determine the cost impact of the uncertainties inherent in each system estimate. When quantifying uncertainty proves impractical, a qualitative assessment must be made.
The CAIG must also be informed of any major risks in the acquisition program that may alter O&S costs, and of all feasible alternatives for reducing risk. For example, an O&S cost estimate is often sensitive to the goals established for the reliability and maintainability of a system and selected subsystems. If attainment of these goals appears unlikely or will be too costly during the development phase, the costs of reducing reliability and maintainability thresholds should be estimated and, if significant, presented to the CAIG.
184.108.40.206 Perform Sensitivity Analysis
Given the number of variables that affect O&S costs, it is useful to perform sensitivity analyses to identify explicitly the nature and magnitude of the uncertainty. The documentation provided with the analysis should explain the methods used to bound the estimate and describe the elements included in the sensitivity analysis. The key cost drivers must be identified in order to determine the impact on cost of changes in performance characteristics, reliability, maintainability, and operating tempo. Sensitivity analysis of suspected cost drivers places the uncertainty associated with an estimate in the proper perspective. The sensitivity analysis requires estimating the effect of the high and low uncertainty ranges for significant input factors and showing the effect of this range on total O&S costs. Each parameter should be examined and tested independently.
220.127.116.11 Document Results
Detailed documentation must be provided for each O&S cost estimate presented to the CAIG. The documentation serves as an audit trail of backup information, methodologies, and results that is used to track a program's costs as it moves from one milestone to the next. The documentation also provides the information needed for the estimate to be replicated by another experienced analyst. The value of a well-documented estimate is so important that Chapter 5 is devoted entirely to the issue of preparing documentations for O&S cost estimates.
3.3.4 Present Results to OSD CAIG
After completing a cost estimate, the results must be communicated in a clear, concise manner to the CAIG. The presentation should focus not only on total life-cycle O&S costs, but also on those aspects of a program that have the greatest effect on O&S costs and, therefore, affect decisionmaking. To aid in structuring this material, Chapter 6 reviews the requirements for presenting O&S cost estimates and provides some formats for displaying the results. Although the formats represent a standardized approach for presenting costs, they should not preclude good judgment in providing additional detail relevant to the program under review.
3.4 O&S COST CONVENTIONS
This section discusses some conventions that should be observed when developing O&S estimates for CAIG review. Some deal with ground rules; others address the scope of the analysis. The specific conventions applied should be documented in the cost estimate. The best way to ensure that the appropriate conventions are applied is through early coordination with the CAIG representative. Deviations from the standards presented here should be discussed with the CAIG representative and documented in the estimate.
|SMALL MISSILES (AIRCRAFT)|
|LARGE MISSILES (ICBM)|
|-- BY CLASS|
|GROUND COMBAT VEHICLES|
| -- TRACKED VEHICLES
-- NONTRACKED VEHICLES
When developing a time-phased estimate, the expected rate of maturity (e.g., reliability growth) should be considered, as well as the rate at which the new system will be added to the force. Different rates of maturity are particularly significant when comparing alternatives that differ markedly in their use of common subsystems. Differences in maturity rates are also important considerations in determining support strategies for the early years of a system's deployment. Clear documentation of the system characteristics assumed in each phase is essential.
Go to Chapter 1 2 3 4 5 6 or Table of Contents
Go to top of page