ENH: Introducing local sensitivity analysis #575

Lucas-Prates · 2024-03-11T23:07:32Z

Pull request type

Other (please describe): This PR is a draft to implement sensitivity analysis in RocketPy. It is a work in progress and needs validation.

Checklist

Lint (black rocketpy/ tests/) has passed locally

Current behavior

The Sensitivity Analysis notebook teaches the users how to perform the simulations, plot the distribution
of some flight variables (e.g. apogee), and computes the prediction ellipses for the landing point.

New behavior

Our goal is to take sensitivity analysis even further. Briefly, we attempt to answer the following question: Which parameters would reduce the variability of the variable of interest (e.g. apogee) the most if we measured them with greater precision?

To that end, a bit of theory is developed, check the technical document. What was developed resembles the work of [1], a core reference in sensitivity analysis for engineering. His approach is a global sensitivity analysis with a full model containing interaction terms. Our first implementation considers a local sensitivity analysis using only first-order terms.

A quick and dirty test of the functionality of the SensitivityModel class is provided the "sensitivity_model_usage" notebook. This notebook is currently giving weird results! The linear approximations for the variables are, for some reason I still have to figure out, not good enough. This was not happening at previous experimentations that suggested that this approached worked. I have to look carefully at what is happening, but I did not want to delay the PR.

The concepts are discussed in-depth in the "sensitivity_analysis_parameter_importance" notebook (the notebook was not updated to the new SensitivityModel yet!)

Breaking change

No

Additional information

Technical Document

[1] Sobol, Ilya M. "Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates." Mathematics and computers in simulation

codecov · 2024-03-11T23:22:44Z

Codecov Report

Attention: Patch coverage is 7.72358% with 227 lines in your changes missing coverage. Please review.

Project coverage is 72.11%. Comparing base (2a74fa4) to head (8bbb7c4).
Report is 12 commits behind head on develop.

Files	Patch %	Lines
rocketpy/sensitivity/sensivity_model.py	8.16%	180 Missing ⚠️
rocketpy/tools.py	4.08%	47 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #575      +/-   ##
===========================================
- Coverage    73.69%   72.11%   -1.59%     
===========================================
  Files           70       72       +2     
  Lines        10304    10550     +246     
===========================================
+ Hits          7594     7608      +14     
- Misses        2710     2942     +232

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Gui-FernandesBR · 2024-05-15T13:17:28Z

Tests are not passing

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/runner/work/RocketPy/RocketPy/rocketpy/__init__.py", line 2, in <module>
    from .environment import Environment, EnvironmentAnalysis
  File "/home/runner/work/RocketPy/RocketPy/rocketpy/environment/__init__.py", line 1, in <module>
    from .environment import Environment
  File "/home/runner/work/RocketPy/RocketPy/rocketpy/environment/environment.py", line [16](https://github.com/RocketPy-Team/RocketPy/actions/runs/9096367936/job/25001616031?pr=575#step:6:17), in <module>
    from ..tools import exponential_backoff
  File "/home/runner/work/RocketPy/RocketPy/rocketpy/tools.py", line 556, in <module>
    parameters_list: list[str],
TypeError: 'type' object is not subscriptable
Error: Process completed with exit code 1.

Could you fix it before our review, please? That would help us. @Lucas-Prates

Lucas-Prates · 2024-05-15T17:07:02Z

Could you fix it before our review, please? That would help us. @Lucas-Prates

Sure, I will fix it briefly. This simplified type hinting started at python 3.9. I will make sure the tests pass this time. :P

Gui-FernandesBR · 2024-05-15T18:30:07Z

Please be aware of #444, we are not supporting type hinting or annotations yet.

Gui-FernandesBR · 2024-05-25T17:19:35Z

setup.py

Why adding a setup.py file?

We are using the pyproject.toml file now. We no longer support the setup.py

Indeed, I had this file from a previous pull request and thought the project still used setup.py, sorry.

rocketpy/tools.py

rocketpy/sensitivity/sensivity_model.py

MateusStano · 2024-05-27T16:09:58Z

rocketpy/sensitivity/sensivity_model.py

+    def set_parameters_nominal(
+        self,
+        parameters_nominal_mean,
+        parameters_nominal_sd,
+    ):
+        """Set parameters nominal mean and standard deviation
+
+        Parameters
+        ----------
+        parameters_nominal_mean : np.array
+            An array contaning the nominal mean for parameters in the
+            order specified in parameters names at initialization
+        parameters_nominal_sd : np.array
+            An array contaning the nominal standard deviation for
+            parameters in the order specified in parameters names at
+            initialization
+        """


So do you have to set the mean and sd simultaneously?

Also, do you have to set mean and sd of all parameters? Setting for just some of them does not work?

Another thing, to run a Monte Carlo sim, the mean and sd is already given in the Monte Carlo class right? So it would be natural to get them from there automatically

It would be great to recover the mean and sd used in the simulation! I tried to find where it was stored, but I might have missed it. Do you know where it is stored?

Indeed, you don't have to provide both. But I thought that, if you had one of them, you probably had the other.

MateusStano · 2024-05-27T16:12:50Z

rocketpy/sensitivity/sensivity_model.py

+        if parameters_matrix.shape[1] != self.n_parameters:
+            raise ValueError(
+                "Number of columns (parameters) does not match number of parameters passed at initialization."
+            )


Hasn't this already been checked if there is a parameters_matrix ?

You are absolutely right, this check and the one in _estimate_target_nominal are redundant.

Gui-FernandesBR

This is just a partial review. I could not run your codes to validate it is working, but I suggested a few changes that might improve the quality (i.e. readability) of the code.

rocketpy/tools.py

Gui-FernandesBR · 2024-05-30T19:48:30Z

rocketpy/tools.py

+    for i in range(n_parameters):
+        parameter = parameters_list[i]
+        parameters_matrix[:, i] = parameters_samples[parameter]
+
+    for i in range(n_variables):
+        target_variable = target_variables_list[i]
+        target_variables_matrix[:, i] = target_variables_samples[target_variable]


Using enumerate is a more pythonic and readable solution.

Suggested change

for i in range(n_parameters):

parameter = parameters_list[i]

parameters_matrix[:, i] = parameters_samples[parameter]

for i in range(n_variables):

target_variable = target_variables_list[i]

target_variables_matrix[:, i] = target_variables_samples[target_variable]

for i, parameter in enumerate(parameters_list):

parameters_matrix[:, i] = parameters_samples[parameter]

for i, target_variable in enumerate(target_variables_list):

target_variables_matrix[:, i] = target_variables_samples[target_variable]

Clear and useful, replaced most of the ranges in code with enumerate. Thanks!

rocketpy/tools.py

Gui-FernandesBR · 2024-05-30T20:35:08Z

rocketpy/tools.py

+    # Auxiliary function that unnests dictionary
+    def unnest_dict(x):
+        new_dict = {}
+        for key, value in x.items():
+            # the nested dictionary is inside a list
+            if isinstance(x[key], list):
+                # sometimes the object inside the list is another list
+                # we must skip these cases
+                if isinstance(value[0], dict):
+                    inner_dict = unnest_dict(value[0])
+                    inner_dict = {
+                        key + "_" + inner_key: inner_value
+                        for inner_key, inner_value in inner_dict.items()
+                    }
+                    new_dict.update(inner_dict)
+            else:
+                new_dict.update({key: value})
+
+        return new_dict


I would define this function outside the load_monte_carlo_data() function, this would allow us to re-use this function in other contexts.

Also, the term flatten is usually used to describe this kid of "parse a nested dictionary" operation. Maybe it would be a good alternative of name.

flatten_dict is definitely a better name for it! With respect to defining it outside the load_monte_carlo_data, there is a detail in the implementation that is specific to the way MonteCarlo simulation data is saved. I have not defined it outside of the function in this first review round because of this, but I might think of a way to make it general and work with the MonteCarlo data in the future.

rocketpy/__init__.py

rocketpy/sensitivity/sensivity_model.py

…t errors

…stall to setup.py

…les, and folders

…imulation's.

… weird results)

Lucas-Prates requested a review from a team as a code owner March 11, 2024 23:07

github-actions bot assigned Lucas-Prates Mar 11, 2024

Gui-FernandesBR marked this pull request as draft April 4, 2024 04:33

Lucas-Prates force-pushed the enh/sensitivity_analysis branch from 58b1cdf to 1c7f84d Compare May 14, 2024 21:45

Lucas-Prates changed the base branch from develop to enh/class_dispersion May 14, 2024 21:46

Lucas-Prates added Enhancement New feature or request, including adjustments in current codes Monte Carlo Monte Carlo and related contents labels May 14, 2024

Lucas-Prates changed the title ~~ENH: (DRAFT) Introducing Parameter Importance in Sensitivity Analysis~~ ENH: Introducing local sensitivity analysis May 14, 2024

Lucas-Prates requested review from Gui-FernandesBR and phmbressan May 14, 2024 22:03

Gui-FernandesBR marked this pull request as ready for review May 15, 2024 13:14

Gui-FernandesBR added this to the Release v1.X.0 milestone May 15, 2024

Gui-FernandesBR linked an issue May 15, 2024 that may be closed by this pull request

Sensitivity Analysis on Monte Carlo Simulations #200

Open

Base automatically changed from enh/class_dispersion to develop May 21, 2024 22:52

Gui-FernandesBR requested a review from MateusStano May 25, 2024 17:17

Gui-FernandesBR reviewed May 25, 2024

View reviewed changes

MateusStano requested changes May 27, 2024

View reviewed changes

Gui-FernandesBR requested changes May 30, 2024

View reviewed changes

Lucas-Prates added 8 commits June 24, 2024 22:55

ENH: introducing ImportanceModel class for parameter importance analysis

2720ecb

MNT: adding imports and renaming analysis folder

06185c9

ENH: implementing plot to ImportanceModel and fixing estimation/impor…

cf87515

…t errors

ENH: implementing summary method to ImportanceModel

2e87f4a

MNT: adding optional requirements for ImportanceModel

68bda07

MNT: using optional import tools and adding sensitivity dependency in…

9884b10

…stall to setup.py

MNT: renaming the term 'importance' to 'sensitivity' in variables, fi…

e3d2acc

…les, and folders

MNT: completing renaming from 'importance' to 'sensitivity'

bc45eab

Lucas-Prates and others added 9 commits June 24, 2024 22:55

MNT: Improving doc and input validation.

a0ffb0e

MNT: fixing plot and input validation in SensitivityModel.

36b7505

ENH: implementing function in tools to extract data from MonteCarlo s…

4fcc23d

…imulation's.

DOC: providing a notebook for quick testing of SensitivityModel (with…

2a0067a

… weird results)

MNT: adding json dependency to tools.py

0208195

Fix code style issues with Black

7047a24

MNT: removing type hints for consistency with codebase (#444)

9adaa8d

MNT: applying review suggestions to sensitivity analysis.

e1151b7

Fix code style issues with Black

8bbb7c4

Gui-FernandesBR force-pushed the enh/sensitivity_analysis branch from 171ad30 to 8bbb7c4 Compare June 25, 2024 01:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Introducing local sensitivity analysis #575

ENH: Introducing local sensitivity analysis #575

Lucas-Prates commented Mar 11, 2024 •

edited

Loading

codecov bot commented Mar 11, 2024 •

edited

Loading

Gui-FernandesBR commented May 15, 2024

Lucas-Prates commented May 15, 2024 •

edited

Loading

Gui-FernandesBR commented May 15, 2024

Gui-FernandesBR May 25, 2024

Lucas-Prates Jun 10, 2024

MateusStano May 27, 2024

Lucas-Prates Jun 10, 2024

Lucas-Prates Jun 10, 2024 •

edited

Loading

MateusStano May 27, 2024

Lucas-Prates Jun 10, 2024

Gui-FernandesBR left a comment

Gui-FernandesBR May 30, 2024

Lucas-Prates Jun 10, 2024 •

edited

Loading

Gui-FernandesBR May 30, 2024

Lucas-Prates Jun 11, 2024

ENH: Introducing local sensitivity analysis #575

Are you sure you want to change the base?

ENH: Introducing local sensitivity analysis #575

Conversation

Lucas-Prates commented Mar 11, 2024 • edited Loading

Pull request type

Checklist

Current behavior

New behavior

Breaking change

Additional information

codecov bot commented Mar 11, 2024 • edited Loading

Codecov Report

Gui-FernandesBR commented May 15, 2024

Lucas-Prates commented May 15, 2024 • edited Loading

Gui-FernandesBR commented May 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lucas-Prates Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gui-FernandesBR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lucas-Prates Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lucas-Prates commented Mar 11, 2024 •

edited

Loading

codecov bot commented Mar 11, 2024 •

edited

Loading

Lucas-Prates commented May 15, 2024 •

edited

Loading

Lucas-Prates Jun 10, 2024 •

edited

Loading

Lucas-Prates Jun 10, 2024 •

edited

Loading