3. O&S COST METHODOLOGY

3.1    INTRODUCTION

    DoDD 5000.4 authorizes the CAIG to establish criteria and procedures for preparing and presenting cost estimates of defense systems requiring DAB review. O&S estimates reviewed by the CAIG should be developed according to a standard set of procedures. This section discusses the CAIG review criteria and presents a costing methodology and conventions for preparing and documenting O&S estimates.

3.2    CAIG REVIEW

    The OSD CAIG is charged with improving the quality of cost analysis within the Department of Defense. As such, it serves as the principal advisory body to the Under Secretary of Defense for Acquisition on matters relating to cost. For major programs, the CAIG is responsible for ensuring that the DAB has available to it an independent estimate of program costs. In addition to presenting the CAIG's own cost estimates, the CAIG report to the cognizant DAB committee and to the DAB contains an evaluation of the estimates developed by the DoD component responsible for the program. It therefore is incumbent upon the DoD components to submit well-documented, credible estimates that are ready for review and can be replicated by a third party.

    Accordingly, the CAIG has established criteria and procedures for preparing and presenting cost estimates. The criteria apply to the development, investment, and O&S phases of a system's life-cycle cost. The CAIG criteria and procedures are discussed in DoDD 5000.4 (Enclosure 2), which addresses the scope of the analysis, analytical methods, and presentation of results.

     3.2.1    Cost Assessment Criteria

        Variations in cost-estimating situations are too great to permit an exhaustive listing of the factors the CAIG may consider. The following criteria, however, will be considered in all cases:

        If an estimating methodology or supporting documentation is insufficient and does not permit a reasonably firm appraisal of the estimate, the CAIG will recommend that the DAB committee review and the full DAB review be deferred until the deficiency is remedied.

        O&S costs typically exceed both development and investment costs over a system's useful life. Therefore, in assessing the total costs of two competing systems, the costs of operating and supporting each system are a primary consideration.

     3.2.2    Lessons Learned from CAIG Reviews

        The lessons learned from previous CAIG reviews are presented here to highlight some of the areas in which cost estimates have been deficient or additional emphasis has been needed:

3.3    O&S COST-ESTIMATING PROCESS

    O&S cost estimates focus on the costs likely to be incurred by a defense system under specified conditions. Although the cost analysis must consider historical costs, it should do more than just extrapolate from past cost trends. The proper approach is to present normalized empirical data derived from the ground rules provided or to show the relationship between an assumption and its related cost impacts. The O&S estimating process described in the sections below is most appropriate for major acquisition programs reviewed by the CAIG. However, the approach is applicable to all acquisition programs, regardless of the review authority.

    Preparation for a CAIG review requires formal identification of the estimating approach, coordination with the CAIG action officer, updating of the current estimate, and preparation of an independent estimate. Exhibit 3-1 lists the primary activities entailed in preparing cost estimates for review by the CAIG.



Exhibit 3-1. O&S COST ESTIMATE PREPARATION PROCESS

3.3.1    Develop Approach

        The selection of an estimating approach involves four basic steps: identification of key O&S issues; selection of a reference system; development of ground rules and assumptions; and selection of a cost element structure. The paragraphs that follow describe each of these steps in turn, and provide some guidelines for accomplishing them.

         3.3.1.1    Identify Primary O&S Issues

                The analytical approach and presentation structure selected must be able to accommodate the unique aspects of the program being assessed. A good way to ensure that all pertinent issues have been identified is for representatives of the CAIG to meet with staff members from the DoD component conducting the analysis. As shown in Exhibit 3-2, the objective of such "pre-CAIG review" meetings is to discuss the issues addressed in the analysis and agree on the approach that will be used to estimate O&S costs (or to update a prior O&S estimate). Where necessary, specific guidance will be provided by the CAIG representative with respect to the estimating approach and CAIG expectations. This guidance must be documented by the organization preparing the estimate, and the estimating approach should be revised accordingly.


  • Potential modifications to CAIG-recommended cost element structure
  • Proposed reference system characteristics and description
  • Description of proposed alternatives
  • Identification of historically relevant cost drivers for reference system and differences for the proposed system
  • Identification of unique operations and support philosophies that could influence requirements
  • Specifications of ground rules and assumptions, including costs to be included or excluded
  • Specification of significant trade-off issues to be quantified and presented
  • Proposed presentation formats

  • Exhibit 3-2. PRE-CAIG REVIEW MEETING TOPICS

             3.3.1.2    Select Reference System

                    A reference system is an existing operational system with a mission similar to that of the proposed system. It is usually the system being replaced, unless some other system provides a better basis for the analysis. Where data on the proposed new system are not available, a reference system is used as a proxy for that system as well.

                    Since reference systems are normally in current operation, there are a variety of sources of data on their cost, technical characteristics, and performance. One source of historical cost data specifically directed by DoDD 5000.4 is the Visibility and Management of Operating and Support Costs (VAMOSC) system. DoDD 5000.4 requires that historical data be used to identify and allocate functional costs among major defense systems and subsystems. One of VAMOSC's objectives is to enhance the visibility of O&S costs in DoD cost analyses. Validated VAMOSC data should be used to calculate the O&S costs of the reference system unless some other sources or data bases are clearly more appropriate.

                    Much of the technical and performance information for the reference system can be obtained from contractor reports, historical government acquisition reports, maintenance data collection systems, and current operational concepts. Similar data for the proposed system may be found in the Cost Analysis Requirements Document (CARD), which is prepared by the DoD component responsible for the program. The CARD provides detailed information on the scope of the program estimate.

                    The assumptions, ground rules, and cost-estimating methodologies for both the reference and proposed system should be similar. This enables the analyst to pinpoint differences in resource consumption that arise from differences in weapon system characteristics. If a comparison of the characteristics of the reference and proposed systems has not been completed for the CARD, a suggested format is provided in Exhibit 3-3. This summary may be presented and discussed at the pre-CAIG review meeting. If a reference system is not available or applicable, the sample format can be used to describe the proposed system.




    CHARACTERISTICS

    REFERENCE
    SYSTEM
    PROPOSED
    SYSTEM MS-I
    AS OF:___
    PROPOSED
    SYSTEM MS-II
    AS OF:___

    System Designation/Name
    General
        Crew Composition
    Performance
        Speed (max)
        Speed (sustained)
        Range
        Payload
    Configuration
        Weight (empty)
        Weight (gross)
        Dimensions
            Height
            Width
            Length
    Acquisition
        Unit Cost/Design-to-Cost
        Goal (prototype/100th prod unit)
        Number of Systems
            Acquire(d) (proposed)
            Deploy(ed) (proposed)
        Operating Concept
        No. of Equipped Deployable Units
            (sqd/companies)
        Average No. Systems/Unit
        Operating Hours/Year/System
    Maintenance Concept
        Interim Contractor Support
        Contractor Logistics Support
        In-House Support
            No. of Maintenance Levels
    Performance Goals
        Operational Ready Rate (%)
        System Reliability
            (Mean Time Between Removals)
            (Mean Time Between Maintenance Action (inherent))
        Maintenance Hours Per
            (Fly Hr/Oper Hr)
        Major Overhaul Point
            (Fly Hrs/Oper Hrs/Miles)

    NOTE:    1.    The elements under each category should be expanded, deleted, or revised to accommodate the system under study.
                    2.    Include additional columns for subsequent milestone reviews as appropriate.


    Exhibit 3-3. SYSTEM DESCRIPTION SUMMARY

    3.3.1.3    Develop Ground Rules and Assumptions

                    A prerequisite to the development of useful O&S cost estimates is a detailed definition of how a system will be operated, maintained, and supported in peacetime. The basic ground rules and assumptions are derived from this information. The ground rules include descriptions of relevant missions and system characteristics, as well as manning, maintenance, support, and logistics policies. The assumptions "bound" the estimate (that is, they define its scope). The ground rules and assumptions are identified in the study documentation. They must be stated clearly and presented in a way that permits them to be referenced easily. Also, the ground rules and assumptions used to estimate O&S costs must be consistent with those employed for non-O&S elements (such as initial spares). Specific O&S cost conventions to consider in developing ground rules and assumptions are discussed in Section 3.4.

             3.3.1.4    Select Relevant Cost Element Structure

                    A cost element structure (CES) establishes a standard vocabulary for identifying and classifying the costs of a system. A generic CES format is presented in Chapter 4 and discussed in Appendix B. Weapon-system-specific CESs are presented in Appendices C through G. The CESs are designed to capture as many relevant costs as practical within the O&S phase. However, due to differences among programs, the basic structure may have to be modified to accommodate the unique characteristics of the system(s) being investigated. The revised structure should be checked to ensure that all relevant costs are represented and defined in a manner compatible with the cost estimate. If a decision will affect costs not explicitly described in this guide, those costs should be identified and included in the O&S estimate. For cost estimates requiring CAIG review, modifications to the CES format recommended here must be approved by the CAIG prior to the start of the analysis.

         3.3.2    Hold Pre-CAIG Meeting

            The CAIG action officer, his or her program office counterpart, and the independent cost estimate (ICE) team chief should meet to discuss the scope, methodology, and ground rules and assumptions for the cost analysis. As presented in Exhibit 3-2, topics covered in these discussions include system characteristics, operating schemes, the specifications of each alternative and of the reference system, the cost element structure, historically relevant O&S cost drivers, costs to be included and excluded, data sources, estimating methodologies, cost models, back-up documentation, and significant trade-off issues. All of these matters should be discussed and agreed upon with the CAIG representative. The initial pre-CAIG meeting should be held shortly after the DAB planning meeting, which is normally conducted 180 days in advance of a DAB review.

         3.3.3    Prepare Estimate

            The paragraphs below describe how to construct an O&S estimate once the CAIG has approved the initial estimating approach. The discussion summarizes the steps entailed in selecting a cost-estimating technique, choosing a cost model, collecting data, and estimating the relevant costs. The requirements for conducting sensitivity analyses and documenting cost estimates also are discussed.

             3.3.3.1    Determine Cost-Estimating Techniques

                    A number of techniques may be employed to estimate the O&S costs of a weapon system. The specific approach chosen will depend to a large degree on the maturity of the program and the level of detail of the available data. Most O&S analyses are accomplished using a combination of three estimating techniques: parametric, analogy, and engineering. As a system progresses from concept development to operations, more detailed information becomes available about it. Initial parametric variables or analogous data may then be updated or replaced with test or operational data.

                    The paragraphs below summarize the key features of the three estimating techniques used in O&S cost assessments. Regardless of the technique employed, the approach must be documented in the materials accompanying the estimate.

             3.3.3.2    Select/Construct Cost Model

                    O&S estimating is most often accomplished using computer models of cost methodologies. Specific models are not prescribed in this guide because the characteristics of a program normally determine the estimating process. However, most O&S models fall into one of three categories:

                    In general, when selecting or constructing a model for O&S cost estimating, one should consider not only the system whose costs are being assessed, but also the acquisition milestone to which the system has progressed and the accuracy expected from the estimate. For example, a cost estimate in support of a Milestone I review is not expected to be as detailed as an estimate prepared for Milestones II or III. As a program matures, the cost model and estimating methodology should be adapted to accommodate more precise information. To satisfy both estimating and review requirements, a good model should have the following characteristics:

             3.3.3.3    Identify Data and Data Sources

                    Three types of data are used in O&S estimates: programmatic, technical, and cost. Data of each type must be provided for both the proposed system and the reference system.

                    Programmatic data are facts or assumptions about the fielding, operating concepts, and support requirements of a weapon system. Technical data focus on system specifications and characteristics, and on performance capabilities or goals. For acquisition programs, this information is available from many of the source documents required by DoDI 5000.2 and from Title 10 of the United States Code. (Appendix A provides a brief summary of the purpose and scope of the major source documents.) Each DoD component may have additional source documents from which similar information may be obtained.

                    The basic source of cost data for most weapons analyses is the historical resource-consumption rates of analogous systems. These values are expressed in terms of dollars, manning levels, and material quantities. Although the concept is straightforward, most categories of historical O&S costs are actually derived values because large portions of the operations and maintenance budget are not broken out by weapon system. Rather, the budget figures are presented by functional area (e.g., depot maintenance, base operating support, training). To calculate O&S costs for individual weapon systems, various allocation techniques have been developed.

                    An important consideration when collecting data is the type of cost represented. Three types of data typically are used:

    As a general rule, one should use the type of data that best describes the system-related O&S cost. Most O&S cost estimates are derived from two forms of data: raw and processed.

             3.3.3.4    Estimate and Evaluate Relevant Costs

                    The selection, design, development, maintenance, and support concepts for a defense system influence how its O&S costs are analyzed and evaluated. In order to compare system alternatives fully, it is necessary to identify and estimate all relevant O&S costs. For example, the number of crew members needed to operate a weapon system may be identical regardless of the alternative selected. However, the reliability of alternative systems may result in very different maintenance manning and material costs, which are relevant to the decision at hand. The purpose of O&S cost analyses changes as a system moves through the various acquisition phases. For example, one of the primary aims of a Milestone I analysis is to explore and quantify the relative cost advantages of different system alternatives, concepts, and design options (e.g., changes in system availability, requirements, alternative support policies, etc.). As a program matures, the cost analysis focuses on trade-offs within the selected alternative, such as deployment, support, and affordability issues. A fundamental objective of this process is to ensure that the alternative selected satisfies mission requirements at the lowest possible life-cycle cost.

             3.3.3.5    Assess Uncertainty

                    Estimates of future O&S costs are subject to some degree of uncertainty. It is important to identify and bound the scope of variables that contribute to uncertainty in ground rules and assumptions. The following are some of the areas that can cause uncertainty and that must be addressed before developing an O&S estimate: mission scenario, deployment plans, operating tempo, system quantity, system availability, operating environment, and cost-estimating methods. These areas are typically identified as non-cost elements that may affect the O&S costs of a system. A sensitivity analysis is conducted to determine the cost impact of the uncertainties inherent in each system estimate. When quantifying uncertainty proves impractical, a qualitative assessment must be made.

                    The CAIG must also be informed of any major risks in the acquisition program that may alter O&S costs, and of all feasible alternatives for reducing risk. For example, an O&S cost estimate is often sensitive to the goals established for the reliability and maintainability of a system and selected subsystems. If attainment of these goals appears unlikely or will be too costly during the development phase, the costs of reducing reliability and maintainability thresholds should be estimated and, if significant, presented to the CAIG.

            3.3.3.6    Perform Sensitivity Analysis

                    Given the number of variables that affect O&S costs, it is useful to perform sensitivity analyses to identify explicitly the nature and magnitude of the uncertainty. The documentation provided with the analysis should explain the methods used to bound the estimate and describe the elements included in the sensitivity analysis. The key cost drivers must be identified in order to determine the impact on cost of changes in performance characteristics, reliability, maintainability, and operating tempo. Sensitivity analysis of suspected cost drivers places the uncertainty associated with an estimate in the proper perspective. The sensitivity analysis requires estimating the effect of the high and low uncertainty ranges for significant input factors and showing the effect of this range on total O&S costs. Each parameter should be examined and tested independently.

             3.3.3.7    Document Results

                    Detailed documentation must be provided for each O&S cost estimate presented to the CAIG. The documentation serves as an audit trail of backup information, methodologies, and results that is used to track a program's costs as it moves from one milestone to the next. The documentation also provides the information needed for the estimate to be replicated by another experienced analyst. The value of a well-documented estimate is so important that Chapter 5 is devoted entirely to the issue of preparing documentations for O&S cost estimates.

        3.3.4    Present Results to OSD CAIG

            After completing a cost estimate, the results must be communicated in a clear, concise manner to the CAIG. The presentation should focus not only on total life-cycle O&S costs, but also on those aspects of a program that have the greatest effect on O&S costs and, therefore, affect decisionmaking. To aid in structuring this material, Chapter 6 reviews the requirements for presenting O&S cost estimates and provides some formats for displaying the results. Although the formats represent a standardized approach for presenting costs, they should not preclude good judgment in providing additional detail relevant to the program under review.

    3.4    O&S COST CONVENTIONS

        This section discusses some conventions that should be observed when developing O&S estimates for CAIG review. Some deal with ground rules; others address the scope of the analysis. The specific conventions applied should be documented in the cost estimate. The best way to ensure that the appropriate conventions are applied is through early coordination with the CAIG representative. Deviations from the standards presented here should be discussed with the CAIG representative and documented in the estimate.


    SYSTEM TYPE
    YEARS
    CARGO AIRCRAFT
    25
    BOMBER AIRCRAFT
    25
    FIGHTER AIRCRAFT
    20
    HELICOPTER AIRCRAFT
    20
    SMALL MISSILES (AIRCRAFT)
    15
    LARGE MISSILES (ICBM)
    20
    ELECTRONIC EQUIPMENT
    10
    SHIPS
        -- BY CLASS
    20-40
    GROUND COMBAT VEHICLES
        -- TRACKED VEHICLES

        -- NONTRACKED VEHICLES

    20

    20

    Exhibit 3-4. DESIGNED SYSTEM LIFE EXPECTANCY


    Exhibit 3-5. SYSTEM LIFE EXPECTANCY O&S PHASES

        When developing a time-phased estimate, the expected rate of maturity (e.g., reliability growth) should be considered, as well as the rate at which the new system will be added to the force. Different rates of maturity are particularly significant when comparing alternatives that differ markedly in their use of common subsystems. Differences in maturity rates are also important considerations in determining support strategies for the early years of a system's deployment. Clear documentation of the system characteristics assumed in each phase is essential.

    Go to Chapter 1 2 3 4 5 6 or Table of Contents

    Go to top of page