Modeling

Adopted as-is — accepted in the original. The Kazakhstan Chamber of Actuaries (KCA) adopts this Actuarial Standard of Practice in its original English form, as published by the Actuarial Standards Board (ASB), United States, without modification. The full original ASB text is reproduced below for reference. The Russian and Kazakh catalogue entries are KCA adaptations; in case of any discrepancy the original ASB text prevails.

---

Actuarial Standard of Practice No. 56

Modeling

Developed by the Modeling Task Force of the General Committee of the Actuarial Standards Board

Adopted by the Actuarial Standards Board December 2019

Doc. No. 195

ASOP No. 56—Doc. No. 195

T A B L E O F C O N T E N T S

Transmittal Memorandum

Transmi	ttal Memorandum	iv
	STANDARD OF PRACTICE
Section	1. Purpose, Scope, Cross References, and Effective Date	1
1.1	Purpose	1
1.2	Scope	1
1.3	Cross References	2
1.4	Effective Date	2
Section	2. Definitions	2
2.1	Assumption	2
2.2	Data	2
2.3	Governance and Controls	2
2.4	Hold-out Data	2
2.5	Input	2
2.6	Intended Purpose	2
2.7	Intended User	3
2.8	Model	3
2.9	Model Risk	3
2.10	Model Run	3
2.11	Output	3
2.12	Overfitting	3
2.13	Parameter	3
Section	3. Analysis of Issues and Recommended Practices	3
3.1	Model Meeting the Intended Purpose	3
	3.1.1 Designing, Developing, or Modifying the Model	3
	3.1.2 Selecting, Reviewing, or Evaluating the Model	4
	3.1.3 Using the Model	4
	3.1.4 Model Structure	4
	3.1.5 Data	4
	3.1.6 Assumptions Used As Input	4
3.2	Understanding the Model	6
3.3	Reliance on Data or Other Information Supplied by Others	6
3.4	Reliance on Models Developed by Others	6
3.5	Reliance on Experts	6
3.6	Evaluation and Mitigation of Model Risk	7
	3.6.1 Model Testing	7
	3.6.2 Model Output Validation	8
	3.6.3 Review by Another Professional	8
	3.6.4 Reasonable Governance and Controls	8
	3.6.5 Mitigating Misuse and Misinterpretation	8

ii

ASOP No. 56—Doc. No. 195

3.7 Documentation	8
Section 4. Communications and Disclosures	9
4.1 Required Disclosures in an Actuarial Report	9
4.2 Additional Disclosures in an Actuarial Report	9
4.3 Confidential Information	9
APPENDIXES
Appendix 1—Background and Current Practices	10
Appendix 2—Comments on the Fourth Exposure Draft and Responses	12

iii

ASOP No. 56—Doc. No. 195

December 2019

TO: Members of Actuarial Organizations Governed by the Standards of Practice of the Actuarial Standards Board and Other Persons Interested in Modeling

FROM: Actuarial Standards Board (ASB)

SUBJ: Actuarial Standard of Practice (ASOP) No. 56

This document contains ASOP No. 56, Modeling .

History of the Standard

The ASB first began work on a standard for modeling in the late 1990s. Motivated primarily to address the role catastrophe modeling of earthquakes and hurricanes played in casualty ratemaking, this work was focused on the use of specialized models where actuaries would have to rely on a model that was developed by professionals other than actuaries. As a result of this work, ASOP No. 38, Using Models Outside the Actuary’s Area of Expertise , was approved by the ASB in June of 2000 with the scope of the standard limited to the Property/Casualty area of practice. Historically, ASOP No. 38 had been the only ASOP that specifically addressed modeling.

Recently, the number and importance of modeling applications in actuarial science have increased, with the results of actuarial models sometimes being reflected in financial statements.

Recognizing this trend, the ASB asked the Life Committee in 2010 to begin work on an ASOP focused on modeling. The Life Committee formed a task force to address this issue and, in February of 2012, a discussion draft titled Modeling in Life Insurance and Annuities was released and nineteen comment letters were received. The transmittal letter also mentioned that the scope might be expanded to all practice areas and asked for comments on this idea.

Based upon the feedback received, and numerous other discussions on the topic of modeling, in December of 2012 the ASB created two multi-disciplinary task forces under the direction of the General Committee: i) a general Modeling Task Force, charged with developing an ASOP to address modeling applications in all practice areas, and ii) a Catastrophe Modeling Task Force to consider expanding ASOP No. 38 to all practice areas while focusing exclusively on using catastrophe models. The membership of these task forces has experience in all actuarial practice areas, including enterprise risk management.

First Exposure Draft

The first exposure draft was released in June 2013 with a comment deadline of September 30, 2013. Forty-eight comment letters were received and considered in making changes that were reflected in the second exposure draft.

iv

ASOP No. 56—Doc. No. 195

Second Exposure Draft

A second exposure draft was released in November 2014 with a comment deadline of March 1, 2015. Thirty-seven comment letters were received and considered in making changes that were reflected in the third exposure draft.

Third Exposure Draft

A third exposure draft was released in June 2016 with a comment deadline of October 31, 2016. Twenty-eight comment letters were received and considered in making changes that were reflected in the fourth exposure draft.

Fourth Exposure Draft

A fourth exposure draft was released in December 2018 with a comment deadline of May 15, 2019. Twenty-six comment letters were received and considered in making changes that were reflected in this final ASOP. For a summary of the issues contained in these comment letters, please see appendix 2.

Notable Changes from the Fourth Exposure Draft

Notable changes made to the fourth exposure draft are summarized below. Additional changes were made to improve readability, clarity, or consistency.

1.     Section 3.1.6(b), Margins, was deleted because it did not provide sufficiently clear guidance. While margins are appropriately used, or even required, for certain intended purposes, margins are inappropriate and not used for other intended purposes. 
   

2.     “Hold-out data” in predictive modeling was defined and added to the list of items that may be included in the model output validation in section 3.6.2(b). 
   

3.     The term “parameter” was eliminated from section 3 of the ASOP, referencing it only within the definition of “assumption” because the two terms often are synonymous and the guidance often was identical.
   

As a next step, the ASB will review the previously approved but pending ASOP No. 38, Catastrophe Modeling (for All Practice Areas) , for any changes necessitated by this ASOP and take appropriate action.

The ASB thanks everyone who took the time to contribute comments and suggestions on the exposure drafts.

v

ASOP No. 56—Doc. No. 195

The ASB also thanks former task force member Aaron R. Weindling for his assistance during the earlier drafting of this standard.

The ASB voted in December 2019 to adopt this standard.

vi

ASOP No. 56—Doc. No. 195

Modeling Task Force

Dale S. Hagstrom, Chairperson Maryellen J. Coggins Stephen Mildenhall Julie H. Fried Judy K. Stromback Kenneth R. Kasner

General Committee of the ASB

Margaret Tiller Sherwood, Chairperson Ralph S. Blanchard III Susan E. Pantely Andrew M. Erman Judy K. Stromback Dale S. Hagstrom Hal Tepfer Robert S. Miccolis Christian J. Wolfe

Actuarial Standards Board

Kathleen A. Riley, Chairperson Christopher S. Carlson Darrell D. Knapp Maryellen J. Coggins Cande J. Olsen Robert M. Damler Barbara L. Snyder Mita D. Drazilov Patrick B. Woods

The Actuarial Standards Board (ASB) sets standards for appropriate actuarial practice in the United States through the development and promulgation of Actuarial Standards of Practice (ASOPs). These ASOPs describe the procedures an actuary should follow when performing actuarial services and identify what the actuary should disclose when communicating the results of those services.

vii

ASOP No. 56—Doc. No. 195

ACTUARIAL STANDARD OF PRACTICE NO. 56

MODELING

STANDARD OF PRACTICE

Section 1. Purpose, Scope, Cross References, and Effective Date

• 1.1 Purpose—This actuarial standard of practice (ASOP or standard) provides guidance to actuaries when performing actuarial services with respect to designing, developing, selecting, modifying, using, reviewing, or evaluating models .

• 1.2 Scope—This standard applies to actuaries in any practice area when performing actuarial services with respect to designing, developing, selecting, modifying, or using all types of models . For example, an actuary using a model developed by others in which the actuary is responsible for the model output is subject to this standard.

If the actuary’s actuarial services involve reviewing or evaluating models , the reviewing or evaluating actuary should be reasonably satisfied that the actuarial services were performed in accordance with this standard. The reviewing or evaluating actuary should apply the guidance in this standard to the extent practicable within the scope of the actuary’s assignment.

The guidance in this ASOP applies to the actuary when, in the actuary’s professional judgment, reliance by the intended user on the model output has a material effect for the intended user . This judgment should be made within the context of the use of the model output and the needs of the intended user , based on facts known by the actuary at the time the actuarial services are performed. For example, actuarial services performed in relation to pension plan contribution and cost projection models , insurance pricing models , predictive models , reserving models , and insurance company financial planning models may require application of the guidance in this ASOP. In assessing materiality, the actuary should be guided by ASOP No. 1, Introductory Actuarial Standard of Practice , section 2.6.

The guidance in this ASOP does not apply to the actuary when performing services with respect to individual pension benefit calculations and nondiscrimination testing, as described in section 1.2 of ASOP No. 4, Measuring Pension Obligations and Determining Pension Plan Costs or Contributions .

This standard only applies to the extent of the actuary’s responsibilities. The actuary’s responsibilities may extend to performing actuarial services related to an entire model or to only a small portion of a model .

1

ASOP No. 56—Doc. No. 195

Other ASOPs may provide guidance for actuarial services that involve models . If the actuary determines that the guidance from another ASOP conflicts with the guidance of this ASOP, the guidance of the other ASOP will govern.

If the actuary departs from the guidance set forth in this ASOP in order to comply with applicable law (statutes, regulations, and other legally binding authority), or for any other reason, the actuary should refer to section 4. If a conflict exists between this standard and applicable law, the actuary should comply with applicable law.

• 1.3 Cross References  When this ASOP refers to the provisions of other documents, the reference includes the referenced documents as they may be amended or restated in the future, and any successor to them, by whatever name called. If any amended or restated document differs materially from the originally referenced document, the actuary should consider the guidance in this ASOP to the extent it is applicable and appropriate.

• 1.4 Effective Date—This ASOP is effective for work performed on or after October 1, 2020.

Section 2. Definitions

The terms below are defined for use in this actuarial standard of practice and appear in bold throughout the ASOP.

• 2.1 Assumption—A type of explicit input to a model that is derived from data , represents possibilities based on professional judgment, or may be prescribed by law or by others. When derived from data , an assumption may be statistical, financial, economic, mathematical, or scientific in nature, and may be described as a parameter .

• 2.2 Data—Facts or information that are either direct input to a model or inform the selection of input . Data may be collected from sources such as records, experience, experiments, surveys, observations, benefit plan or policy provisions, or output from other models .

• 2.3 Governance and Controls—The application of a set of procedures and an organizational structure designed to reduce the risk that the model output is not reliably calculated or not utilized as intended.

• 2.4 Hold-out Data—A subset of data that is withheld intentionally when developing a predictive model so that the model may be validated later with data that were not used to develop the model .

• 2.5 Input— Data or assumptions used in a model to produce output .

• 2.6 Intended Purpose—The goal or question, whether generalized or specific, addressed by the model within the context of the assignment.

2

ASOP No. 56—Doc. No. 195

• 2.7 Intended User—Any person whom the actuary identifies as able to rely on the model output .

• 2.8 Model—A simplified representation of relationships among real world variables, entities, or events using statistical, financial, economic, mathematical, non-quantitative, or scientific concepts and equations. A model consists of three components: an information input component, which delivers data and assumptions to the model ; a processing component, which transforms input into output ; and a results component, which translates the output into useful business information.

• 2.9 Model Risk—The risk of adverse consequences resulting from reliance on a model that does not adequately represent that which is being modeled, or the risk of misuse or misinterpretation.

• 2.10 Model Run—The process of transforming a particular set of input to a particular set of output in a model . A model run could include the whole transformation process or part of the process, as applicable.

• 2.11 Output—The results of a model including, but not limited to, point estimates, likely or possible ranges, data or assumptions (as input for other models ), behavioral expectations, or qualitative criteria on which decisions could be made.

• 2.12 Overfitting—A situation where a model fits the data used to develop the model so closely that prediction accuracy materially decreases when the model is applied to different data .

• 2.13 Parameter—A type of statistical, financial, economic, mathematical, or scientific value that is used as input to certain types of models . Examples of parameters include expected values in probability distributions and coefficients of formula variables. Some types of models , such as predictive or statistical models , produce estimates of parameters as output , which may be used as input to other models .

Section 3. Analysis of Issues and Recommended Practices

• 3.1 Model Meeting the Intended Purpose—The actuary should understand the model’s intended purpose .

  • 3.1.1 Designing, Developing, or Modifying the Model—When the actuary designs, develops, or modifies the model, the actuary should confirm, in the actuary’s professional judgment, that the capability of the model is consistent with the intended purpose . Items the actuary should consider, if applicable, include but are not limited to the following:

    • a. the level of detail built into a model ;

3

ASOP No. 56—Doc. No. 195

• b. the dependencies recognized; and

• c. the model’s ability to identify possible volatility of output , such as volatility around expected values.

• 3.1.2 Selecting, Reviewing, or Evaluating the Model—When selecting, reviewing, or evaluating the model , the actuary should confirm that, in the actuary’s professional judgment, the model reasonably meets the intended purpose .

• 3.1.3 Using the Model—When using the model , the actuary should make reasonable efforts to confirm that the model structure, data , assumptions , governance and controls , and model testing and output validation are consistent with the intended purpose .

• 3.1.4 Model Structure—The actuary should assess whether the structure of the model (including judgments reflected in the model ) is appropriate for the intended purpose . The actuary should consider the following, as applicable, for a particular model :

  • a. which provisions and risks specific to a business segment, contract, or plan, if any, or interactions more broadly, are material and appropriate to reflect in the model ;

  • b. whether the form of the model is appropriate, such as a projection model (deterministic or stochastic), statistical model , or predictive model ;

  • c. whether the use of the model dictates a particular level of detail, for example, whether grouping inputs will produce reasonable output , or whether a certain level of detail in the output is needed to meet the intended purpose ;

  • d. whether there is a material risk of the model overfitting the data ; and

  • e. whether the model appropriately represents options, if any, that could be reasonably expected to have a material effect on the output of the model . Examples include call options on fixed income assets, policyholder surrender options, and early retirement options.

• 3.1.5 Data—The actuary should use, or confirm use of, data appropriate for the model’s intended purpose and should refer, as applicable, to ASOP No. 23, Data Quality , when selecting, reviewing, or evaluating data used in the model , either directly or as the basis for deriving, estimating, or testing assumptions used in the model .

• 3.1.6 Assumptions Used As Input—For models that use assumptions as input , the actuary should use, or confirm use of, assumptions that are appropriate given the

4

ASOP No. 56—Doc. No. 195

model’s intended purpose . The following guidance applies for models that use assumptions as input :

• a. Setting Assumptions—When setting assumptions for which the actuary is taking responsibility, the actuary should consider using the following data or information:

  1. actual experience properly modified to reflect the circumstances being modeled, to the extent actual experience is available, relevant, and sufficiently reliable;

  2. other relevant and sufficiently reliable experience, such as industry experience that is properly modified to reflect the circumstances being modeled, if actual experience is not available, relevant, or sufficiently reliable;

  3. future expectations or estimates, including those derived from market data , when available and appropriate; and

  4. other relevant sources of data or information.

• b. Range of Assumptions—The actuary may consider using a range of assumptions and, if so, whether the number of model runs analyzed reflects a set of conditions consistent with the intended purpose .

• c. Consistency—Where appropriate, the actuary should use, or confirm use of, assumptions for the model that are reasonably consistent with one another for a given model run .

If the actuary is aware of material inconsistencies among assumptions used by the actuary in the model , the actuary should disclose the inconsistencies and known reasons for the inconsistencies. In the case of assumptions prescribed by applicable law, the actuary’s disclosure may be limited to identifying the possibility of an inconsistency with other assumptions .

• d. Appropriateness of Input in Current Model Run—Where practical and appropriate, the actuary reusing an existing model should evaluate whether input unchanged from a prior model run is still appropriate for use in the current model run . For example, models used in financial reporting may offer opportunities to compare assumptions to emerging experience in the aggregate.

• e. Reasonable Model in the Aggregate—The actuary should assess the reasonability of the model output when determining whether the assumptions are reasonable in the aggregate. While assumptions might

5

ASOP No. 56—Doc. No. 195

appear to be reasonable individually, conservativism or optimism in multiple assumptions may result in unreasonable output .

• 3.2 Understanding the Model—When expressing an opinion on or communicating results of the model , the actuary should understand the following:

  • a. important aspects of the model being used, including but not limited to, basic operations, important dependencies, and major sensitivities;

  • b. known weaknesses in assumptions used as input , known weaknesses in methods or other known limitations of the model that have material implications; and

  • c. limitations of data or information, time constraints, or other practical considerations that could materially impact the model’s ability to meet its intended purpose .

• 3.3 Reliance on Data or Other Information Supplied by Others—When relying on data or other information supplied by others, the actuary should refer to ASOP No. 23 and ASOP No. 41, Actuarial Communications , for guidance.

• 3.4 Reliance on Models Developed by Others—If the actuary relies on a model designed, developed, or modified by others, such as a vendor or colleague, and the actuary has a limited ability either to obtain information about the model or to understand the underlying workings of the model , the actuary should disclose the extent of such reliance. In addition, the actuary should make a reasonable attempt to have a basic understanding of the model , including the following, as appropriate:

  • a. the designer’s or developer’s original intended purpose for the model ;

  • b. the general operation of the model ;

  • c. major sensitivities and dependencies within the model ; and

  • d. key strengths and limitations of the model .

When relying on models developed by others, the actuary should make practical efforts to comply with other applicable sections of this standard.

• 3.5 Reliance on Experts—The actuary may rely on experts in the fields of knowledge used in the development of the model . In determining the appropriate level of reliance, the actuary may consider the following:

  • a. whether the individual or individuals upon whom the actuary is relying are experts in the applicable field;

6

ASOP No. 56—Doc. No. 195

• b. the extent to which the model has been reviewed or validated by experts in the applicable field, including known material differences of opinion among experts concerning aspects of the model that could be material to the actuary’s use of the model ;

• c. whether there are industry or regulatory standards that apply to the model or to the testing or validation of the model , and whether the model has been certified as having met such standards; and

• d. whether the science underlying the expertise is likely to produce useful models for the intended purpose .

When relying on experts, the actuary should disclose the extent of such reliance.

• 3.6 Evaluation and Mitigation of Model Risk—The actuary should evaluate model risk and, if appropriate, take reasonable steps to mitigate model risk . The type and degree of model risk mitigation that is reasonable and appropriate may depend on the following:

  • a. the model’s intended purpose ;

  • b. the nature and complexity of the model ;

  • c. the operating environment and governance and controls related to the model ;

  • d. whether there have been changes to the model or its operating environment; and

  • e. the balance between the cost of the mitigation efforts and the reduction in potential model risk .

  • 3.6.1 Model Testing—For a model run or set of model runs generated at one time or over time that is to be relied upon by the intended user , the actuary should perform sufficient testing to ensure that the model reasonably represents that which is intended to be modeled. Model testing may include the following:

    • a. reconciling relevant input values to the relevant system, study, or other source of information, addressing and documenting the differences appearing in the reconciliation, if material;

    • b.

  • checking formulas, logic, and table references;

• c. running tests of variations on key assumptions used as input to test that changes in the output are consistent with expectations given the changes in the input (i.e., sensitivity testing); and

7

ASOP No. 56—Doc. No. 195

  - d. reconciling the **output** of a **model run** to prior **model runs** , given changes in **data** , **assumptions** , formulas, or other aspects of the **model** since the prior **model run** . 

• 3.6.2 Model Output Validation—The actuary should validate that the model output reasonably represents that which is being modeled. Depending on the intended purpose , model output validation may include the following:

  • a. testing, where applicable, preliminary model output against historical actual results to verify that modeled output would bear a reasonable relationship to actual results over a given time period if input to the model were set to be consistent with the conditions prevailing during such period;

  • b. evaluating whether the model applied to hold-out data produces model output that is reasonably consistent with model output developed without the hold-out data , as may be used for predictive models ;

  • c. performing statistical or analytical tests on model output to assess their reasonableness;

  • d. running tests of variations on key assumptions to test that changes in the output are consistent with the expectations given the changes in the input ; and

  • e. comparing model output to those of an alternative model (s), where appropriate.

• 3.6.3 Review by Another Professional—The actuary may consider obtaining a review by another qualified professional, depending upon the nature and complexity of the model .

• 3.6.4 Reasonable Governance and Controls—The actuary should use, or, if appropriate, may rely on others to use, reasonable governance and controls to mitigate model risk .

• 3.6.5 Mitigating Misuse and Misinterpretation—The actuary should refer to the guidance in ASOP No. 41, in particular sections 3.4.1 and 3.7, to mitigate possible misuse and misinterpretation of the model .

• 3.7 Documentation—The actuary should consider preparing and retaining documentation to support compliance with the requirements of section 3 and the disclosure requirements of section 4. If preparing documentation, the actuary should prepare such documentation in a form such that another actuary qualified in the same practice area could assess the reasonableness of the actuary’s work. The degree of such documentation should be based on the professional judgment of the actuary and may vary with the complexity and purpose

8

ASOP No. 56—Doc. No. 195

of the actuarial services. In addition, the actuary should refer to ASOP No. 41, section 3.8, for guidance related to the retention of file material other than that which is to be disclosed under section 4.

Section 4. Communications and Disclosures

• 4.1 Required Disclosures in an Actuarial Report—When issuing an actuarial report under this standard, the actuary should refer to ASOP Nos. 23 and 41. In addition, the actuary should disclose the following in such actuarial reports:

  • a. the intended purpose of the model , as discussed in section 3.1;

  • b. material inconsistencies, if any, among assumptions , and known reasons for such inconsistencies, as discussed in section 3.1.6(c);

  • c. unreasonable output resulting from the aggregation of assumptions , if material, as discussed in section 3.1.6(e);

  • d. material limitations and known weaknesses, as discussed in section 3.2;

  • e. extent of reliance on models developed by others, if any, as discussed in section 3.4; and

  • f. extent of reliance on experts, if any, as discussed in section 3.5.

• 4.2 Additional Disclosures in an Actuarial Report—The actuary should include the following, as applicable, in an actuarial report:

  • a. the disclosure in ASOP No. 41, section 4.2, if any material assumption or method was prescribed by applicable law;

  • b. the disclosure in ASOP No. 41, section 4.3, if the actuary states reliance on other sources and thereby disclaims responsibility for any material assumption or method selected by a party other than the actuary; and

  • c. the disclosure in ASOP No. 41, section 4.4, if, in the actuary’s professional judgment, the actuary has otherwise deviated materially from the guidance of this ASOP.

• 4.3 Confidential Information—Nothing in this ASOP is intended to require the actuary to disclose confidential information.

9

ASOP No. 56—Doc. No. 195

Appendix 1

Background and Current Practices

Note: This appendix is provided for informational purposes and is not part of the standard of practice.

Background

Actuaries frequently use models to analyze uncertain outcomes, with every discipline relying on a broad range of modeling applications, ranging from simple spreadsheets to complex capital models. Actuaries have used models for a variety of purposes including to help explain a system, to study the effects of different parts of a system, to predict the behavior of a system, to predict the behavior of people, to derive estimates, or to inform decisions. The importance of modeling in actuarial science has continued to increase, with results of models sometimes being reflected in financial statements.

A model is only an approximation of reality, however, and not reality itself. Therefore, even a model that is prudently developed and carefully used does not eliminate inherent uncertainty and variability, and actual results may differ, sometimes significantly, from outcomes suggested by the model.

Current Practices

Actuaries use many types of models, ranging from projection models to statistical models and predictive models. Some models evolve through a life cycle consisting of: (1) a specification phase, (2) an implementation phase, and (3) a production phase, which consists of one or more model runs. Other models evolve through a life cycle of: (1) a specification phase, (2) an iterative, assumptions estimation phase, and (3) an output evaluation, validation, and selection phase. For other models, combinations of functionally similar phases may exist.

Appropriate model governance and controls are important when using models. Examples of model governance and controls include the following:

• limitations on access to use and modify the model (that is, restricting access to model input, model programming code and calculations, and model output);

• confirmation that model output is reproducible upon rerun (if the model allows for such reproducibility);

• implementing a model change management process;

• specification, documentation, and programming standards for the model;

10

ASOP No. 56—Doc. No. 195

• procedures for secure back-up of the media storing the programming code and data;

• appropriate staff training or cross-training for continuity of use and mitigation of key-person risk;

• plans for periodic consideration of the organization’s continued ability to access and maintain the model, including data, software, staff, hardware, and any vendor relationships; and

• plans for periodic review of the assumptions, functionality, and methodology.

11

ASOP No. 56—Doc. No. 195

Appendix 2

Comments on the Fourth Exposure Draft and Responses

The fourth exposure draft titled Modeling was approved by the ASB in December 2018 with a comment deadline of May 15, 2019. Twenty-six comment letters were received, some of which were submitted on behalf of multiple commentators, such as by firms or committees. For purposes of this appendix, the term “commentator” may refer to more than one person associated with a particular comment letter. The Task Force and General Committee carefully considered all comments received, and the ASB reviewed (and modified, where appropriate) the changes proposed by the General Committee.

Summarized below are the significant issues and questions contained in the comment letters and the responses to each. Minor wording or punctuation changes that were suggested but not significant are not reflected in the appendix, although they may have been adopted.

The term “reviewers” includes the Task Force, General Committee, and the ASB. Unless otherwise noted, the section numbers and titles used below refer to those in the fourth exposure draft, which are then cross referenced with those in the final ASOP.

GENERAL COMMENTS	GENERAL COMMENTS
Comment Response	One commentator suggested that the uses of “any” when in the context of what an actuary should do or should consider, and other similar references, may be onerous to actuaries in practice, and recommended their elimination. The reviewers agree and made the change.
Comment Response	One commentator suggested retaining a definition of “simple model” conceptually similar to what was included in the third exposure, with the suggested enhancement of modifying “transparent and can be predicted” to “transparent or can be predicted” to improve its usefulness and clarity. The reviewers note the concept of“simple model”has been previously addressed and made no change.
Comment Response	One commentator suggested that the standard include a definition of and guidance for ongoing model performance monitoring. While the reviewers agree with the concept of ongoing performance monitoring within a formalized model risk management program, the reviewers disagree with the suggestion for this ASOP and therefore did not make the change.
SECTION 1. PURPOSE, SCOPE, CROSS REFERENCES, AND EFFECTIVE DATE
Section 1.1, Purpose
Comment Response	One commentator suggested that sections 1.1, Purpose, and 1.2, Scope, should include explicit reference to mitigating model risk since it is a key area of focus on the modeling process and there is an explicit section of the ASOP exposure draft dedicated to this practice. The reviewers believe the guidance is appropriate and therefore made no change.

12

ASOP No. 56—Doc. No. 195

Section 1.2, Scope	Section 1.2, Scope
Comment Response	One commentator suggested that “responsible” should be replaced by “accountable” since it implies ownership – and the use of this term is more consistent with that used in the insurance industry to indicate appropriate ownership. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator recommended the use of the words “rely” and “reliance” be clarified as the terms are rather subtle given that some users of models consider the use of a model as reliance even when it is the user’s own model. The reviewers believe the guidance is appropriate and therefore made no change.
Comment Response	One commentator suggested that the standard be applied only to financial reporting models and perhaps enterprise risk models. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that the guidance for an actuary reviewing or evaluating models is not clear as to whether it is the model itself that is being reviewed or evaluated (which is what the text seems to literally suggest), or whether it is the use of the model that is being reviewed. The reviewers clarified the guidance.
Comment Response	One commentator disagreed with the exclusion of the concept of a “simple model” from the fourth exposure draft and recommended that the scope explicitly exclude simple calculations. The reviewers disagree with the suggestion and, therefore, did not make the change. The reviewers refer the commentator to section 1.2, Scope, including the definition of “model,” when considering the applicability of the guidance in the ASOP.
Comment Response	One commentator suggested certain references to “use” might be confusing, in particular: 1) When the actuary’s “use” of a model is not for the purpose of reviewing the model itself but only for the purpose of reviewing or using the output. In this instance, the standard should explicitly state that the actuary should not be charged with applying this standard, and 2) in the second paragraph that states the reviewing or evaluating actuary should “use the guidance in this standard to the extent practicable within the scope of the actuary’s assignment” and in third paragraph that appears to use “rely” and “use” interchangeably. The reviewers agree with the potential confusion that might arise with the word “use” in the second and third paragraphs, and replaced these two references to “use” in section 1.2, Scope to improve clarity. However, the reviewers believe the guidance in the second paragraph is appropriate and therefore made no change in response to that part of the comment.
Comment Response	Two commentators suggested that the first sentence in the fifth paragraph seems unnecessary and suggested eliminating that sentence. One commentator also suggested beginning the paragraph with the current third sentence. The reviewers agree and made the change.

13

ASOP No. 56—Doc. No. 195

Comment Response	One commentator thought the example, “For example, actuarial services performed in relation to pension plan contribution and cost projection models…may require application of the guidance in this ASOP” was confusing. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 1.4, Effective Date
Comment Response	Once commentator believes that the effective date language needs to be more descriptive because as written, it leaves many questions related to when the model was run, selected, developed, or when model results were communicated. The reviewers note that ASOPs apply to the actuary performing the actuarial services, and the effective date applies to “work performed [by the actuary] on or after....” Therefore, the reviewers made no change in response to this comment.
SECTION 2. DEFINITIONS
Comment Response	One commentator suggested adding definitions for “testing,” “validation,” and “limitations.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 2.1, Assumption
Comment Response	One commentator suggested that the definition of section 2.1, Assumption, be changed to note that an assumption can be produced as output from another model. Alternatively, the definitions of data and parameter in sections 2.2 and 2.12, respectively, could be changed to remove any reference to these items being produced from other models. The reviewers agree, made changes to the definitions of “assumption,” “parameter,” “input,” and “output,”and removed references to“parameter”within section 3 of the ASOP to improve clarity.
Comment Response	One commentator question whether assumptions are always input into a model versus incorporated into the model operations or methodology. In an effort to improve clarity and in response to this comment, the reviewers revised the definition of “assumption” to “a type of explicit input…” thus differentiating between explicit and implicit assumptions.
Section 2.2, Data
Comment Response	One commentator requested examples of data that can be input to a model in the same way that examples of parameters are provided in that section since data are often refreshed with each model run while parameters and assumptions often remain unchanged from one run to the next. While the reviewers did not make the specific recommended edit, the reviewers made changes to the definitions of “assumption,” “parameter,” “input,” and “output,” and removed references to “parameter” within section 3 of the ASOP to improve clarity.
Comment Response	One commentator suggested that the drafted definition is too vague and general with respect to the kinds of data the ASOP addresses and suggested the definition be limited to quantitative or numerical data. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 2.3, Governance and Controls
Comment Response	One commentator suggested that a more descriptive definition would be “The application of a set of procedures and an organizational structure designed so that intended users can have confidence that the model output is reliably calculated and utilized as intended.” The reviewers clarified the language.

14

ASOP No. 56—Doc. No. 195

Comment Response	One commentator suggested defining “governance” and “controls” separately since they have different meaning. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 2.4, Input (now section 2.5)
Comment Response	One commentator suggested the definition of input is very broad, and that input to a model can be in the form of 1) assumptions, 2) data, or 3) parameters. While each term is defined separately later in the document, the user must glean that they are not overlapping elements of input. The reviewers agree, made changes to the definitions of “assumption,” “parameter,” “input,” and “output,”and removed references to“parameter”within section 3 of the ASOP to improve clarity.
Comment Response	One commentator suggested adding the following sentence after the current sentence: “Input may include assumptions, data, and parameters.” The reviewers agree in part, made changes to the definitions of “assumption,” “parameter,” “input,” and “output,”and removed references to“parameter”within section 3 of the ASOP to improve clarity.
Section 2.5, Intended Purpose (now section 2.6)
Comment Response	One commentator suggested clarifying whether a model can have more than one intended purpose, perhaps treating each intended purpose as a separate model, even where they have a common processing component. This approach will reinforce the need to assess the appropriateness of a combination of specific processing components, data, assumptions, parameters and output for each intended purpose. The reviewers believe the guidance is appropriate and therefore made no change.
Comment Response	One commentator understood the definition for all roles other than when the actuary is the model developer and suggested that there should be a consideration of other purposes to be efficient with modeling efforts and less siloed in approach. The reviewers disagree and therefore made no change.
Section 2.6, Intended User (now section 2.7)
Comment Response	Three commentators suggested replacing “actuarial findings” with “model’s output” (which is defined in this ASOP while “findings” are not). The reviewers agree and made the change.
Comment Response	One commentator suggested replacing the word “actuarial findings” with “output of an actuarial model.” The reviewers agree in part and replaced“actuarial findings”with“model output.”
Comment Response	One commentator noted the definition is too broad as it describes an actuary as “able” to rely, and suggested alternatives of “likely” or “expected.” The reviewers disagree and therefore made no change.
Comment Response	One commentator suggested that, while the definition is identical to that contained within ASOP No. 41, Actuarial Communications, the use of “able” and “identifies” in the definition may cause confusion, and suggested the alternative “Any person whom the actuary has indicated is permitted to rely on the actuarial findings.” The reviewers disagree and therefore made no change.
Section 2.7, Model (now section 2.8)
Comment Response	One commentator sought feedback regarding the definition of “model” in the context of several examples. The reviewers note that the ASOPs are principle-based and believe the current languagecovers these issues at the appropriate level of detail. Therefore, no change was made in response to this comment.

15

ASOP No. 56—Doc. No. 195

Comment Response	One commentator suggested adding the caveat from the background section of appendix 1 to the definition of a “model” to emphasize that a model is not bad or inaccurate just because a model did not match actual experience, namely: “A model is only an approximation of reality, not the reality itself, and the differences between the model and actual experience, by themselves, do not indicate a flawed model or noncompliance with standards.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that the definition of a “model” is very broad and recommended defining the “processing component” to enable differentiation between simple calculations and a “model.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested removing the reference to “simplified” as it seems unnecessarily restrictive. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that the definition is too broad as it could be interpreted to include any actuarial service other than individual benefit calculations and recommended that the definition should also describe what is not a model, such as nondiscrimination testing. The reviewers believe the definition of “model” is appropriate but note that section 1.2 was modified to exclude nondiscrimination testing.
Comment Response	One commentator suggested that the definition be changed to include “contractual” as a type of input and suggested adding “actuarial” to the list. In addition, the commentator suggesting adding a new definition for “system” as referenced in the definition. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested separating the “results component” from the model definition because the use of “results” in section 2.10, Output, appears to be inconsistent with the “results component” as described in this definition and the definition of output allows that such output could be used as input to other models. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested changing “to predict the behavior of a system, or to derive estimates and guide decisions” to “to predict the behavior of a system, to derive estimates of a system, or to guide decisions,” because the former could imply “guiding decisions” and “deriving estimates” should always be considered together. The reviewers note that the last sentence in the definition was removed.
Comment Response	One commentator suggested that the definition and section 1.2, Scope, were unclear, and thus it was difficult to evaluate the remainder of the exposure draft. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

16

ASOP No. 56—Doc. No. 195

Comment Response	One commentator suggested the definition was unclear as to what types of models were addressed by the ASOP, and recommended that the ASOP specifically refer to quantitative or numerical models with respect to data, parameters, input and output, and that the scope of the “models” covered by the ASOP should be limited to quantitative models (for example, estimates) or perhaps other types of models based directly on quantitative values and explicitly exclude algorithmic decision making and other forms of artificial intelligence. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 2.8, Model Risk (now section 2.9)
Comment Response	One commentator suggested that the definition include specific guidance on the use of the term, namely that “model risk” is not intended to include the likelihood that actual results of most all models will often differ, perhaps materially, from that produced by the Model’s output, and recommended that, at a minimum, the sentence from the second paragraph (if not, the entire paragraph) in the “Background” section of this ASOP be made an integral part of the ASOP: “Even a model that is prudently developed and carefully used does not eliminate inherent uncertainty and variability, and actual experience may differ, sometimes significantly, from the estimates derived from the model results,” ideally, within this definition. As an alternative, the ASOP could add an additional definition for “model outcome risk.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggesting adding the consequence of model risk to the definition, namely that “Model risk can lead to financial loss, poor business and strategic decision making, or damage to ... reputation.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested rewording for better clarity as follows: “The risk of adverse consequences resulting from reliance on a model that does not adequately represent that which is being modeled or the risk of misuse or misinterpretation.” The reviewers agree and made the change in response to this comment.
Section 2.9, Model Run (now section 2.10)
Comment Response	Two commentators sought clarification on what a model run constitutes, with one commentator recommending calling the collection of all simulations for a stochastic model as one model run to improve clarity. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggesting replacing “selection of input” with “set of input.” The reviewers agree and made the change.
Section 2.10, Output (now section 2.11)
Comment Response	One commentator suggested that the four possible uses of output (i.e., point estimates, ranges, parameters for other models, or qualitative criteria for making decisions) fail to capture the use of a model for explaining a system or predicting its behavior. The reviewers agree and added“behavioral expectations”to the definition.

17

ASOP No. 56—Doc. No. 195

Comment Response	One commentator noted that section 2.10, Output, only mentions parameters as output that might be used as input to other models, while different sections of the proposed ASOP also mention data and assumptions as possible model outputs that can be used as input to other models. The reviewers agree, made changes to the definitions of “assumption,” “parameter,” “input,” and “output,”and removed references to“parameter”within section 3 of the ASOP to improve clarity.
Comment Response	One commentator suggested eliminating “qualitative criteria on which decisions could be made,” which is vague and may include unintended application of the ASOP. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 2.11, Overfitting (now section 2.12)
Comment Response	Three commentators suggested adding “materially” to the phrase “prediction accuracy decreased” to allow for the actuary to determine whether that decrease is large enough to cause concern. The reviewers agree and made the change.
Comment Response	One commentator suggested that including “may decrease” in place of “decrease” seems more appropriate since the guidance in section 3.14 uses the words “should consider.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested including a definition of underfitting as well as adding more descriptive examples for both overfitting and underfitting. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 2.12, Parameter (now section 2.13)
Comment Response	One commentator suggested that to further distinguish parameter from data, it would be helpful to state, “Parameters often consist of product features that are used to configure a model for specific blocks of business. Unlike data, they typically remain constant from run to run, unless the model’s scope is expanded to include new products.” While the reviewers did not make the specific recommended edit, the reviewers made changes to the definitions of “assumption,” “parameter,” “input,” and “output,” and removed references to “parameter” within section 3 of the ASOP to improve clarity.
Comment Response	One commentator recommended further differentiating between a parameter used as an input to a model and that used as output from a model (for example, “input parameter” and “output parameter”). While the reviewers did not make the specific recommended edit, the reviewers made changes to the definitions of “assumption,” “parameter,” “input,” and “output,” and removed references to “parameter” within section 3 of the ASOP to improve clarity.
Comment Response	One commentator suggested adding the phrase “that is not data or assumptions” after “contractual input” in the first sentence. The reviewers removed the reference to the term “contractual” within the definition of “parameter,” and revised the definitions of“assumptions,” “input,”and“output”to improve clarity.
Comment Response	One commentator shared an analysis of the definitions and use of the terms “parameter,” “assumptions,” “input” and “output,” and stated that it is not clear how “parameters” are distinguishable from other “assumptions” or “data.” The reviewers agree, made changes to the definitions of “assumption,” “parameter,” “input,” and “output,”and removed references to“parameter”within section 3 of the ASOP to improve clarity.

18

ASOP No. 56—Doc. No. 195

Comment Response	One commentator observed that the definition of parameter appeared to be a subset of assumptions and recommended considering language to highlight that assumptions/methods may be used to develop the parameters used in the model. The reviewers agree in part, made changes to the definitions of “assumption,” “parameter,” “input,” and “output,”and removed references to“parameter”within section 3 of the ASOP to improve clarity.
Comment Response	One commentator suggested adjusting the definition to restrict it to quantitative values. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
SECTION 3. ANALYSIS OF ISSUES AND RECOMMENDED PRACTICES
Section 3.1, Model Meeting the Intended Purpose
Comment Response	One commentator noted that actuaries will often “repurpose” models for different intended purposes and suggested that the ASOP explicitly require the actuary developing, selecting, or evaluating the model to identify and document the specific purposes or ranges of parameters/inputs, etc., for which the model is valid/applicable and require actuaries to identify what aspects of the model would need to be adjusted to eliminate model limitations. The commentator also suggested that actuaries developing models should anticipate modeling changes that will develop in the near future to avoid having rigid models. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 3.1.1, Designing, Developing, or Modifying the Model
Comment Response	One commentator suggested that this section should speak directly to modeling choices. Where the design of a model includes significant modeling choices (for example, simplifications, approximations), the actuary should understand the rationale and/or justification for the choices made. Where an actuary is responsible for designing, developing, or modifying a model, the actuary should consider whether developmental testing is needed to assess the appropriateness of significant modeling choices. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator noted that the meaning of “dependencies recognized” is not clear and requires additional explanation. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator noted that it may not be clear what the actuary is looking for in terms of “consistency with the intended purpose” when discussing the volatility of the expected values and that it’s not clear what “dependencies” are, in particular whether the term is referencing the dependencies among models or consistency of the model with its data, assumptions & parameters (A&P), and methods. In addition, the commentator suggested that a definition of dependencies would be helpful. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested replacing the phrase “include but are not limited to” with “for example” since such a replacement would reduce the chance of misinterpretation of the guidance in terms of what the actuary is obliged to do. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

19

ASOP No. 56—Doc. No. 195

Section 3.1.2, Selecting, Using, Reviewing, or Evaluating the Model (now titled, Selecting, Reviewing, or Evaluating the Model).****Note: Changes to old section 3.1.2 have been incorporated into new section 3.1.3, Using the Model, as referenced below.	Section 3.1.2, Selecting, Using, Reviewing, or Evaluating the Model (now titled, Selecting, Reviewing, or Evaluating the Model).****Note: Changes to old section 3.1.2 have been incorporated into new section 3.1.3, Using the Model, as referenced below.
Comment Response	One commentator noted that the initial input as well as revisions to input need to be consistent with the intended purpose, and therefore recommended removing the words “any revisions to.” The reviewers agree and made the change, which appears in new section 3.1.3.
Comment Response	One commentator noted general agreement, with the exception of “governance and controls,” which in many situations will be set at a firm-wide level and are not available for an actuary’s review (for instance, when an actuary uses its firm’s actuarial valuation software). Further, although the commentator agrees that governance and controls may affect the actuary’s ability to rely on the model, the commentator does not believe these factors would affect the model’s inherent consistency with its intended purpose, and suggested the ASOP should contain a separate section describing what an actuary should consider with respect to governance and controls for models. The reviewers believe the guidance, which now appears in new section 3.1.3, is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator noted confusion with the use of “output are consistent with the intended purpose,” and that the use of “consistent” might result in confusion between sections 3.1.1 and 3.1.2. Further, the commentator suggested the word “validation” should be replaced with “testing” given that the term “validation” is a very particular word for many companies and usually corresponds to Independent Model Validation. The reviewers believe the guidance, which now appears in new section 3.1.3, is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested replacing “confirm the model reasonably meets the intended purpose ...” with “review that the model is reasonable with respect to meeting the intended purpose ...” In addition, the commentator suggested replacing “to ensure that any revisions to the input and ... are consistent with the intended purpose.” with “to consider whether the revisions to the input and ... are consistent with the intended purpose.” The reviewers clarified the guidance.
Comment Response	One commentator suggested replacing the word “ensure” with “validate” and sought an example for what “the standard require(s) with respect to the determination of reasonability.” The reviewers clarified the guidance and replaced the word “ensure” with “use or confirm” in new section 3.1.3.
Section 3.1.3, Understanding the Model (now section 3.2)
Comment Response	One commentator suggested replacing “results of the model,” with “output” as defined in section 2, requested clarification of “methods” in paragraph b, and suggested removing “time constraints” in paragraph c. The reviewers believe the guidance is appropriate and therefore made no change in response to these comments.
Comment Response	One commentator asked whether the actuary should also understand the appropriate use of the model. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator did not think this paragraph should be limited to when the actuary is expressing an opinion on or communicating results of the model and suggested “rewording would be helpful here.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

20

ASOP No. 56—Doc. No. 195

Comment Response	One commentator expressed uncertainty regarding the meaning of “dependencies,” and questioned whether “methods” meant the model “methodology” or whether it meant the methods used to develop the A&P. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested replacing section 3.1.3 with the following: “When providing actuarial services which depend significantly on the use of one or more models, the actuary should understand the important aspects of each model being used, such as: a. basic operation of the model, significant dependencies and sensitivities among variables or parameters, input and output, in the model; b. significant known limitations with respect to assumptions and parameters used as input, with respect to the data, information or methods used to build, calibrate, test or validate the model, or with respect to other considerations known to pose material implications when using the model or interpreting model output; and c. significant limitations with respect to a material impact affecting the ability of the model to meet its intended purpose due to other practical considerations, such as data issues, incomplete information, time constraints, etc.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 3.1.4, Model Structure
Comment Response	One commentator recommended removing the examples in 3.1.4(e), suggesting that they are not “useful or necessary.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that this section should clarify when the actuary should make this assessment, such as when designing, developing, modifying, selecting, using, reviewing, or evaluating a model, or only when doing some of those actions. In addition, the commentator requested further clarification on the meaning of “judgments reflected in the model” and recommended the removal of “the structure of” from the stem as it would not change the guidance and could prevent confusion/misinterpretation. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator questioned why only overfitting is considered, and suggested consideration of parsimony, identifiability, goodness of fit, theoretical consistency and predictive power given that overfitting is just one of many types of error that would result in deteriorating a model’s predictive power. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested including definitions for “projection model,” “statistical model,” and “predictive model.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested replacing the current statement “whether the model is overfitting the data” with “whether the model is overfitting or underfitting the data” to fully capture the bias/variance tradeoff instead of focusing solely on overfitting. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

21

ASOP No. 56—Doc. No. 195

Comment Response	One commentator suggested using “structure” instead of “form” for consistency with the title of 3.1.4, Model Structure. The reviewers disagree and therefore made no change.
Comment Response	One commentator suggested replacing should “consider” in section 3.1.4 with “evaluate and document,” and suggested adding wording that requires actuary to indicate how, if at all, modeling of these provisions, risks and interactions are simplified and therefore appropriate only in certain situations. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested adding the word “product” to the list in section 3.1.4(a), adding “or type” after “whether the form” to better reflect the reference to projection, statistical, predictive models, and whether “model requirements” may be necessary in section 3.1.4(c). The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested rewording of section 3.1.4, subsections a, d, e as follows: “(a) whether there are specific provisions and risks reflected in the model which are material and appropriate to the use of the model, for example, differences by business segment, contract or plan; (d) whether there is a significant and material risk of overfitting the model with the available data; (e) whether the model appropriately reflects the existence of significant options or features, which may apply, that could be reasonably expected to have a material effect on the output of the model. Examples include call options on fixed income assets, policyholder surrender options, and early retirement options.” The reviewers clarified the language regarding overfitting the model but made no change in response to the other comments.
Section 3.1.5, Data
Comment Response	One commentator suggested that the actuary should consider what transformations of input data and assumptions, if any, are required and how these affect results. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 3.1.6, Assumptions and Parameters Used As Input (now section 3.1.6, Assumptions Used As Input)
Comment Response	One commentator believes that it is “unnecessary, confusing and burdensome to include assumptions setting guidance in this standard, given the Assumptions ASOP currently under development, and given the many other ASOPs that provide assumption setting guidance for certain activities.” The reviewers believe the guidance is appropriate and therefore made no change related to this comment. This ASOP may not reference another ASOP that continues to be within the exposure process.
Comment Response	One commentator suggested adding “As” to the beginning of the stem of section 3.1.6, to read, “As for models that use assumptions and parameters as input....” In addition, the commentator noted that assumption setting and parameterization of assumptions should be mentioned separately for clarity as they are different activities and imply different risks. While the reviewers did not make the specific recommended edit, the reviewers made changes to the definitions of “assumption,” “parameter,” “input,” and “output,” and removed references to “parameter” within section 3 of the ASOP to improve clarity.
Comment Response	One commentator suggested the addition of an example of a model that does not use assumptions or parameters as input. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

22

ASOP No. 56—Doc. No. 195

Section 3.1.6(a), Setting Assumptions and Parameters (now section 3.1.6[a], Setting Assumptions)	Section 3.1.6(a), Setting Assumptions and Parameters (now section 3.1.6[a], Setting Assumptions)
Comment Response	One commentator stated that it should be a criterion that the actuary document assumptions appropriately or ensure that assumptions provided by others are documented as such. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested referring to ASOP No. 25,Credibility Procedures, when discussing using actual experience to the extent it is “relevant and sufficiently reliable” within section 3.1.6(a)(1). The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested adding a fifth line item to section 3.1.6(a), namely “prescribed assumptions set by law” and “prescribed assumptions set by another party” (as used in ASOP No. 27,Selection of Economic Assumptions for Measuring Pension Obligations, and ASOP No. 35,Selection of Demographic and Other Noneconomic Assumptions for Measuring Pension Obligations) (for example, accounting assumptions), and assumptions developed with the opinion of experts. In addition, the commentator does not believe that the actuary should be required to assess whether assumptions that include prescribed assumptions set by law or prescribed assumptions set by another party are reasonable in the aggregate. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested changing the title of section 3.1.6(a) from “Setting Assumptions and Parameters” to “Setting Assumptions or Parameters” because the former could imply both are required, and adding reasonableness of individual assumptions or parameters that could have a material impact on model results to section 3.1.6(a) since reasonableness in aggregate is mentioned in 3.1.6(f). The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggesting rewording section 3.1.6(a)(1) to be “actual experience adjusted to current conditions where applicable, to the extent that adjustments to the data are considered to be available, relevant, and sufficiently reliable;” and requested a definition of “market data.” While the reviewers did not make the specific changes suggested, the reviewers replaced “It” with “actual experience”in section 3.1.6(a), Setting Assumptions, to improve clarity.
Section 3.1.6(b), Margins
Comment Response	Several comments were received on the guidance or necessity of section 3.1.6(b), Margins. In response, the reviewers removed section 3.1.6(b), Margins.
Section 3.1.6(c), Range of Assumptions and Parameters (now Section 3.1.6[b], Range of Assumptions)
Comment Response	One commentator suggested that it is not clear what is meant by a range of assumptions and parameters in section 3.1.6(c) and offered a number of alternative of the meaning of the phrase. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator questioned why the number of model runs was relevant to the range of assumptions and parameters. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

23

ASOP No. 56—Doc. No. 195

Section 3.1.6(d), Consistency (now section 3.1.6[c], Consistency)	Section 3.1.6(d), Consistency (now section 3.1.6[c], Consistency)
Comment Response	One commentator suggested changing the phrase “…possibility of an inconsistency…” to “…potential of an inconsistency…” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that just requiring the actuary to “use or confirm use” is very weak guidance, and that the standard should use "not unreasonably inconsistent" in order to indicate that consistency in this context is subject to considerable judgment. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 3.1.6(e), Appropriateness of Input in Current Model Run (now section 3.1.6[d], Appropriateness of Input in Current Model Run)
Comment Response	One commentator stated agreement with 3.1.6(e), and suggested the addition, perhaps in a separate paragraph, that the model itself (not just the input) should be evaluated. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested clarifying the following “… reusing an existing model…” given that the term “reusing” can also be interpreted as using an existing model for a different purpose while the intention here seems to be around using a model with updated data. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 3.1.6(f) Reasonable Model in the Aggregate (now section 3.1.6[e] Reasonable Model in the Aggregate)
Comment Response	One commentator suggested that it would be helpful to provide an example of a situation where assumptions which are reasonable individually can produce output which is unreasonable in the aggregate, and recommended adding guidance around appropriate potential actions if the actuary determines this to be the case. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator noted that the determination on the reasonability of a model in the aggregate as well as the assumptions and parameters in the aggregate would typically involve examining the reasonability of the output of the model in making such a determination, and suggested articulating the importance of considering the reasonability of the output in making the determination of the reasonability of the model in the aggregate as well as the reasonability of the parameters and assumptions in the aggregate. The reviewers agree and added“the reasonability of the model output when determining”after“assess.”
Comment Response	One commentator suggested rewording section 3.1.6(f) as follows: "The actuary should assess whether the assumptions and parameters are reasonable in the aggregate. The actuary should consider those assumptions and parameters which might appear to be reasonable individually, but would produce unreasonable output, due to conservatism or optimism in multiple assumptions and parameters." The reviewers agree and made changes similar to those suggested to improve clarity.
Section 3.2, Reliance on Data or Other Information Supplied by Others (now section 3.3, Reliance on Data or Other Information Supplied by Others)
Comment Response	One commentator suggested adding the title of ASOP No. 23 consistent with the title of ASOP No. 41. The reviewers note that the ASOP follows an approved style guide. Since the title of ASOP No. 23, Data Quality, had been previously mentioned, no further reference is required for subsequent mentions.

24

ASOP No. 56—Doc. No. 195

Section 3.3, Reliance on Models Developed by Others (now section 3.4, Reliance on Models Developed by Others)	Section 3.3, Reliance on Models Developed by Others (now section 3.4, Reliance on Models Developed by Others)
Comment Response	One commentator suggested that the actuary also consider the experience and qualifications of the colleague/vendor. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that to the extent the actuary relies on testing performed by others, the actuary should also make a reasonable attempt to understand testing that has been performed on the model, i.e., implementation testing as well as any developmental testing. In addition, the commentator suggested that actuary who relies on a model built by a vendor or other developer is still responsible for ensuring the model is appropriate given its intended purpose and that results of any ongoing performance monitoring processes should be added to the list items to examine and understand. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that this section would lead to a tremendous amount of additional, unnecessary work, and potential litigation risk if the work is not performed, such as when relying upon centralized valuation systems implemented and tested by others. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested removing the last sentence in the section as it is somewhat ambiguous and could leave open to interpretation which sections of the standard are applicable, and that the detailed sub-bullets 3.3(a)-(d) seem sufficient. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator noted that it isn’t clear whether the intent is that the actuary should disclose reliance if they can do neither, or if they can do one but not the other, and that it is not clear whether “a limited ability … to understand the underlying workings of the model” would include a situation where the actuary cannot review programming but can understand what the model is intended to produce and can verify reasonableness and recommended clarification. The reviewers agree with the suggestion that the actuary may have a limited ability to either “obtain information about the model or to understand the underlying workings of the model” or both. The reviewers added “either” to improve clarity. Otherwise, the reviewers believe the guidance is appropriate and made no further change.
Comment Response	One commentator recommended that a new sentence be added after the listing, “The actuary should continually evaluate model results in light of emerging experience to determine that the model is still appropriate for its intended purpose.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator objected to permitting actuaries to rely upon models which they do not fully understand and feels this violates Precept 1 of the_Code of Professional Conduct_and diminishes our profession. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

25

ASOP No. 56—Doc. No. 195

Section 3.4, Reliance on Experts (now section 3.5, Reliance on Experts)	Section 3.4, Reliance on Experts (now section 3.5, Reliance on Experts)
Comment Response	One commentator expressed no significant concerns with section 3.4, however noted that it will become cumbersome, confusing, and misleading in certain circumstances when the expert is employed by the same firm as the actuary. As a result, the commentator recommended that the requirement to disclose the extent of any reliance be limited to situations where the experts were not employed by the actuarial firm issuing the report. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested removing the last sentence, “The actuary should disclose the extent of any such reliance,” because section 4.1(f) already lists the disclosure requirement for 3.4. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 3.5, Mitigation of Model Risk (now section 3.6, Evaluation and Mitigation of Model Risk)
Comment Response	One commentator recommended including a statement that model materiality is an important consideration in actions the actuary should take to mitigate model risk. The more material the impacts of a model can have on the company financial statements, capital positions, or management action, the more actions the actuary should take to mitigate the model risk. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator stated that the actuary should use judgment when assessing mitigation efforts as compared to model risk, and that the level of model risk mitigation should be commensurate with the perceived or actual level of risk associated with the use of the model. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator believes that “evaluate” implies a quantitative process and recommended replacing “evaluate” with a term such as “understand.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested changing the title of section 3.5 from “Mitigation of Model Risk” to “Evaluation and Mitigation of Model Risk” given the guidance. The reviewers agree and made the change.
Comment Response	One commentator suggested changing 3.5(d) to read “whether there have been any changes to the model or its operating environment” for consistency. The reviewers agree and made the change.
Comment Response	One commentator recommended the inclusion of guidance related to when and how often the actuary should an actuary evaluate model risk. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested replacing 3.5(d) with the following: “(d) whether there have been significant changes to the model or to the underlying environment, conditions, experience, or process for which the model was designed; and” While the reviewers did not make the specific changes suggested, the reviewers replaced “modeling” with“operating”environment to improve clarity.

26

ASOP No. 56—Doc. No. 195

Section 3.5.1, Model Testing (now section 3.6.1, Model Testing)	Section 3.5.1, Model Testing (now section 3.6.1, Model Testing)
Comment Response	One commentator suggested that section 3.5.1, Model Testing, should include reference to sensitivity testing given that it is an important part of model testing. The reviewers agree and added “running tests of variation on key assumptions used as input to test that changes in the output are consistent with expectations given the changes in the input (sensitivity testing).”
Comment Response	One commentator suggested that it should be clearer that “reconciling,” means that the values are input correctly in to the model or modeling software, and not just that the input data before it is loaded in to the model reconciles to the source data given that if someone reconciles that initial data before it is loaded in to a model reconciles with the admin system, but then loads it in to the model incorrectly, it is a source of model risk. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that section 3.5.1(b) deserves more attention as this is often the most time- consuming element of model testing and recommended stating that the actuary should consider what the major modeling methodology choices and simplifications are, as well as determine the best way to appropriately test formulas. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested adding in a new section 3.5.1(c): “Performing sample runs of individual model points to validate application of model logic and inputs” and shifting the existing 3.5.1(c) to 3.5.1(d). The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator sought clarification on how the actuary's responsibility for testing the model would differ between a “model run” and a “set of model runs generated at one time or over time.” In addition, the commentator suggested moving “data” to appear before “input,” and changing the definition of “model” to reference “formula” instead of “processing component” given that the term is more intuitive. The reviewers agreed with moving the reference to “data” to be before “assumptions” but did not make other changes in response to this comment.
Comment Response	One commentator suggested renaming these sections 3.5.1 and 3.5.2 to “model integrity testing” and “model output validation.” The reviewers agree that section 3.5.2, Model Validation, should be renamed to Model Output Validation, but did not change the title of section 3.5.1.
Comment Response	One commentator sought clarification on the determination of materiality in section 3.5.1(a), and on the difference between testing and validation. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator noted that sections 3.5.1 (a)-(c) could be considered model controls and governance, and not necessarily model testing. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

27

ASOP No. 56—Doc. No. 195

Section 3.5.2, Model Validation (now section 3.6.2, Model Output Validation)	Section 3.5.2, Model Validation (now section 3.6.2, Model Output Validation)
Comment Response	One commentator sought clarification on the term “Model Validation,” and how the use of term in the ASOP differs from the use of that same term under SR 11-7: Guidance on Model Risk Management. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that section 3.5.2 should include and reference the concept of an “effective challenge,” and that the intensity and effort of the challenge should be commensurate with the risk and materiality of the model. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested adding an additional item under 3.5.2 related to predictive models, namely, “For predictive models, testing should include running the developed model against a hold-out dataset, not used to develop the model, to verify that modeled output would bear a reasonable relationship to actual results from the hold-out data.” In addition, the commentator suggested adding a definition of “hold-out data” such as: “Hold-out data – typically a random subset of the data being modeled. Hold-out data is not used to create the model itself, but rather, used to validate that the model that was built is truly predictive when applied to a previously unseen set of data.” The reviewers agree that changes were appropriate and modified the language in this section and added a definition of“hold-out data.”
Comment Response	One commentator suggested changing “The actuary should take appropriate steps to validate” to “The actuary should validate” for greater clarity. The reviewers agree and made the change.
Comment Response	One commentator suggested that section 3.5.2 be called Model Testing, given that Validation has a specific connotation to many companies that is not meant by what is being described. The reviewers modified the title of section 3.5.2 from Model Validation to Model Output Validation.
Section 3.5.3, Review by Another Professional (now section 3.6.3, Review by Another Professional)
Comment Response	One commentator recommended striking section 3.5.3 since actuaries can always consider having another professional review their work and the section provides no guidance and is not needed. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator questioned when it would be appropriate to not obtain such a review and suggested that the word “may” be replaced by “should” or removing the sentence altogether. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested replacing section 3.5.3 with the following: The actuary may consider obtaining a review by a second, qualified professional. Use of another review would increase depending upon the nature and complexity of the model as well as with the materiality of the intended use(s).” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Section 3.5.5, Mitigating Misuse and Misinterpretation (now section 3.6.5, Mitigating Misuse and Misinterpretation)
Comment Response	One commentator suggested that section 3.5.5 is already handled in the stem of 3.5 and recommended that this section be removed. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

28

ASOP No. 56—Doc. No. 195

Comment Response	One commentator noted the reference in section 3.5.5 to sections 3.4.1 in ASOP No. 41 but noted there is no section 3.4.1 in ASOP No. 41. The reviewers note that section 3.4.1 in ASOP No. 41 is titled“Uncertainty or Risk.”
Comment Response	One commentator suggested mentioning the headings/titles of the section in other ASOPs in addition to the section numbers when they are being used as reference in case that the section numbers got changed in another ASOP for any reason. The reviewers note the standard follows an approved style guide and made no change in response to this comment.
Section 3.6, Documentation (now section 3.7, Documentation)
Comment Response	One commentator suggested that the section should be more specific about what to document, with documentation best practices including the documentation of inputs, calculations – including key methodology choices (including simplifications and approximations), outputs, intended purpose, use limitations, and ongoing performance monitoring processes, model testing (including any developmental testing) and validation. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	Three commentators suggested strengthening the guidance by replacing “should consider” with “should.” The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested that the provision that the documentation could allow that another actuary qualified in the same practice area “assume the assignment if necessary” could be onerous in many cases and recommended that the ASOP should not expand upon general documentation requirements as the provision in the draft ASOP - that “another actuary qualified in the same practice area could assess the reasonableness of the actuary’s work”- is sufficient. The reviewers agree and deleted“or could assume the assignment if necessary.”
SECTION 4. COMMUNICATIONS AND DISCLOSURES
Section 4.1, Required Disclosures in an Actuarial Report
Comment Response	One commentator recommended changing the section name to “Disclosures in an Actuarial Report” since the use of “required” in the title is confusing given the guidance that the actuary “should disclose,” and recommended adding any unreasonable, unexplained variances from recent ongoing performance monitoring processes (addressed in a recommended new section 3.5.6) should be added to the list of items that should be disclosed. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator suggested replacing 4.1(d) with “d. unreasonable output resulting from the aggregation of assumptions and parameters used as input, if material, as discussed in section 3.1.6(f).” The reviewers agree with the concept and modified the language accordingly.
Comment Response	One commentator recommended changing “material limitations” to “material limitations, important aspects and weaknesses” to ensure disclosures cover all related items discussed in section 3.1.3. The reviewers agree in part and added“and known weaknesses”after“material limitations.”
Comment Response	One commentator suggested adding a clarification as to whether the “experts” in section 4.1(f) refer to outside experts or both outside and in-house experts. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

29

ASOP No. 56—Doc. No. 195

Comment Response	One commentator noted that not all items in section 3.3 are covered by the disclosures in section 4.1, namely key methods and A&P and model testing (sensitivities). The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator recommended that it be made clear that the ASOP does not require an actuarial report with respect to the models used by the actuary. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.
Comment Response	One commentator proposed removing section 4.2 as section 4.1 already requires compliance with the disclosure standards of ASOP No. 41. The reviewers believe the guidance is appropriate and therefore made no change in response to this comment.

30